Interview questions about Work/life balance

I am in the process of looking for a new job and my biggest priority is work life balance. At my current job I have essentially 0 time for personal hobbies, making/meeting friends, life in general, etc during the work week. This is not sustainable and I am deeply burned out here after just a short time.

How can I bring up this topic during the interview process? I know that on-call is a reality of this job and I'm fine with that, but at my current job we have a particularly brutal on-call, long working hours, and regular crunch time which is just too oppressive and completely unsustainable.

How can I trust peoples answers to my questions that I bring up? It has been my experience that companies will very easily lie or tell half-truths about these questions?

https://redd.it/13jk9l5
@r_devops

Any advice to integrate my knowledge of DevOps?

I've been a Sofware developer for a year and a half in a DevOps setting and a few months ago I started to train more in areas of Ops.

I'm beyond confused, I thought most Software Dev was to code, test and deploy and that's it. I was fine with using Git, pushing to Bitbucket and tested using unittests and Gherkin BDDs. But now I'm struggling to learn how all of these concepts fit in this methodology like Chef Cookbooks, recipes, SCM, Jenkins CI/CD Pipelines, Docker and Containers, Kubernetes Pods, Cloud Services (AWS, GCP, AZR), Bash Scripting/Bootstrapping. And every video I watched is like it's in a different language (Not a coding one lol)

I'm trying to understand the process of DevOps, why do we use all of these tools and what's the full process like from the beginning to the end of the cycle. I seem to have an idea of a few of them but it's very vague and I don't know if it's realistic.

If you have any advices on books, content, tutorials I can check out please let me know! Also, I would really appreciate any help with further questions I have in this area.

https://redd.it/13jlbwc
@r_devops

What does it take to be a true DevOps?

Besides fancy tools such as k8s, Jenkins, ansible etc... What does it take to be a DevOps Engineer, one of the top

I think DevOps is a complex role cause combines different IT skills and is way more beyond knowing "x" or "y" technology

What do you think?

https://redd.it/13jn3vf
@r_devops

Simple cost-effective way to deploy multiple docker containers?

Read a bunch of threads on this but they were all 2+ years old. Thought maybe there are new or better ways to do things now.

I have multiple side projects that I host on an EC2 instance (t4g.medium) using a process manager and nginx as a reverse proxy. Works incredibly well and is simple and cheap. I use either lets-encrypt or cloudflare for SSL.

As my number of projects grow, I find myself using newer tech, but it's a hassle to update things, say your language runtime or security updates to the OS.

So now I want to move over to docker containers but keep things simple and cheap and add on CI/CD. Some of my considerations:

- Optimize for cost
- Add in CI/CD (mostly just to move my secrets into the cloud so if my EC2 instance or laptop dies, it's not an issue)
- Keep RDS connections low (I tried to go serverless, but I'd have to pay for an RDS Proxy)
- Keep complexity low, have as few components as possible (looking at you k8s)
- Remain on AWS, ok with lock-in
- Scalability is not important, I'm ok with a single container for a monolithic service
- Hopefully reduce some maintenance
- Try to keep to the t4g.medium instance

I've looked around at previous threads and options are:
1. Swap the process manager with docker on the EC2 instance, call it a day: cheap and effective, but if you need to update the server, all containers go down, also not sure how easy this is going to be to set up CI/CD
2. ECS + EC2: I hear you have to use ELB, which ads ~$18/mo, or hack it together with Service Discovery or Service Connect
3. ECS + Fargate: Probably the best option but has a cold start and some of my older apps have built-in cron jobs, so the service needs to always be up
4. Self-managed K8s on the EC2: really don't want to go this route, too much of heavy-lift and also I don't think it's recommended to attempt a single-node configuration
5. Don't bother with any infra and use AWS App Runner
6. Suck it up and go to Vercel

Ideally, I'd be able to go from container to live deployment in half an hour.

What's everyone else doing?

https://redd.it/13j85pu
@r_devops

Building logging-as-a-service with ClickHouse

I work as an engineer at a startup focused on application monitoring, and recently we introduced a logging product powered by ClickHouse. For those interested, here's a brief overview of what we learned during the process.
To begin with, a significant amount of time was dedicated to designing the architecture for this product, particularly the schema of our logging table. We initially used the OTEL specification as a starting point to expedite the design, but we still had to experiment with column definitions. To optimize query performance, we made several adjustments such as modifying the precision of our timestamp column, introducing indices to our attributes map, and configuring the primary key in a way that facilitated cursor pagination (further details are available in the post below). Implementing support for multi-tenancy posed an interesting challenge as each of our customers had unique data retention requirements.
Overall, it has been an enjoyable journey, and we have found ClickHouse to be exceptionally fast (previously, we used OpenSearch). We hope this will be beneficial for future startups building their applications from scratch with ClickHouse; here's a full blog link:
Link to post: https://www.highlight.io/blog/how-we-built-logging-with-clickhouse

https://redd.it/13j78w1
@r_devops

So, I just had an incident today

and it cost the company an unrecoverable amount of USD 2,742,111.75.

Spoiler: it’s due to a sneaky bug 🐛

https://redd.it/13j9uha
@r_devops

Udacity's nanodegree reviews

I bought Udacity's SRE nano-course to upskill and get better for interviews. It promised 10 hours weekly would take you four months to complete.


I literally completed it in a day, their statements are a joke. Also worse they try to get you to sign up for four months for a single-class. They do not have a subscription model where you can take different courses or nanos whatever they call them. Luckily they offer a 7 day refund period


In terms of the content, it mainly consists of very short videos and reading some modules, and completing some simple quizzes. There was only one hands on assignment -- and the final project very much resembles this assignment that is even too easy for a junior engineer. The instructions/solutions for one lab was wrong and outdated -- had to go to forums to see that they dont update their content. Final project was literally setting up Prometheus/Grafana on a K8s cluster and monitoring four metrics.

One of there other Cloud Engineer courses was reputable, and they have other programs too https://www.udacity.com/course/cloud-native-application-architecture-nanodegree--nd064 but it seems sus after my encounter. Maybe their other nanos like ML/robotics is good but the quality of this course was a falsly advertised.

https://redd.it/13jrxqw
@r_devops

Best Metrics for Scaling Django REST API with KEDA?

Hey! Exploring KEDA for scaling Django API services. Is the request count from Prometheus NGINX Ingress:

round(sum(irate(nginx_ingress_controller_requests{ingress="api",exported_service="timelog"}[1m])) by (exported_service), 0.001)

a reliable metric for KEDA?

https://redd.it/13jrghc
@r_devops

Modeling EC2 on-demand vs reserved instance pricing

Managing cloud costs can be quite complex. You've got to predict the future while understanding how pricing structures impact all possible load patterns. In this post I discuss a how re-framing on-demand instances as a "premium" instead of reserved instances as a "discount" can shift your thinking. There's a break even point at around 15 hours per day when on-demand and reserved instance pricing meet (across almost all instance types). I walk through how to calculate costs and model costs in a formal way. This can help you save the most money when auto-scaling.

https://redd.it/13iuxop
@r_devops

Introducing Terrateam Self-Hosted

https://github.com/terrateamio/terrateam

https://terrateam.io/blog/terrateam-self-hosted

I'm excited to announce the self-hosted version of Terrateam designed for Terraform users that need to meet strict security and compliance requirements or just prefer to host their own deployment.

Really looking forward to folks kicking the tires, providing feedback, and submitting feature requests.

https://redd.it/13jvaxr
@r_devops

Venting - CI/CD requirements

This is just for general venting, I don't necessarily need any advice. I work for a company that just acquires other companies, it's their whole gimmick. That being said, when an acquisition happens it's always a mad house getting them integrated. The product I work on was acquired a few years ago and that integration just never happened. I like to joke that it's the forgotten child. What happened was the entire devops/infrastructure team quit immediately after the acquisition and the week before I started.

So I stepped into this absolute shit show of an environment and spent near 2 years patching holes and migrating to new infrastructure. Recently our build/deploy pipelines in Jenkins shit the bed more or less permanently. It grew out of control largely because of neglect on my part which was due to me being bogged down with other requirements.

I proposed and more or less was given hand wave approval to move to gitlab as the fix and future solution since our code is already there, as well as a huge cost saving, to empower the dev team to build their own pipelines and all that jazz. So, I start working on this and now I'm about halfway done. Our build and deploys are in gitlab and it's time to start setting up our automated tests. As I'm doing this it's now time to get financial approval to move to this.

All in all we'll be saving about 30k a year to ditch jenkins for gitlab. Builds, deploys, and testing (I've already set up a couple) are significantly improved in the respect time to complete and reliability (jenkins builds would lose connection to the host node intermittently among other things). It seems like an easy win.

Email gets sent off for approval from my boss to upper management and then a shit show ensues. My company largely uses BitBucket and a different Jenkins setup. Another of our devops teams now want to own the builds and also want to move from gitlab to bitbucket. I tell them this is a terrible idea as we have many client commitments through the end of the year and this will pose various issues and blockers while they get this product integrated. But the other team insists that this is the way it has to be.

So fine, I did my part with discussing and arguing over a few weeks and ultimately lost the argument. The other team used the bitbucket importer as a test run on one of our nastier repos and it goes horribly. We lost previous PRs and comments and whatnot. We have various public repos for reasons and the internal system/security standard is more or less a hell no with public content. That should have been a show stopper but I lost that argument as well. So they decide "fine, you can stay in gitlab for now as your SCM but builds have to be set up in the other jenkins". I stressed that the load we'll put on their small-ish minimal set up will cause problems, but again I lost the argument. So they set up the web hooks and whatnot then go "you gotta convert these libraries of scripts into declarative jenkinsfiles." Again, stupid busy idea that just prolongs the inevitable and wastes time as we're blocked on a few repos.

So I start rewriting these jenkinsfiles and whatnot, sure enough the nasty repo starts beating the shit out of their jenkins. Now other products are blocked due to jenkins crashing. So we submit our tickets that its broke and the other team starts working on fixing it. They increase the server resources and all that but it still isn't enough and jenkins keeps crashing. Its now a mad house on their team because they were busy enough, I can't do much because I don't have access to that infrastructure and I'm twiddling my thumbs and working on a couple of other projects.

I'm like 90% sure they'll give up in a week or two but by that time we'll be 3 weeks behind schedule on commitments and sprints that we were booked through the year on. I'm a bit cynical and I love morbid humor. So, I've done nothing but have a laugh at the whole dumpster fire.

I've obviously glossed over a bit of detail but I think the gist is there.

TL;DR: I need to migrate our

pipelines to something else, gitlab being the target. Halfway through, another team won an argument on not doing that in favor of other options. Despite evidence showing it was going to end terribly, they won the argument. And its gone terribly, so I'm having a morbid laugh at the situation.

https://redd.it/13jvxzq
@r_devops

Odigos v0.1.5 - Managing OpenTelemetry using Kubernetes labels

https://keyval.dev/labels-support/

https://redd.it/13jxid4
@r_devops

Repo for small scripts - What's the best practice

We use Azure Devops as a source control repository. The primary language we use is C#. Occasionally, some devops engineers write small automation scripts in python or bash. e.g. Generating a list of stale branches, or delete large number of files from S3 bucket.


What is the best practice to store such scripts? I am thinking of creating one dedicated repo just to store such scripts. This will provide all source control benefits.


Couple of downsides I can think of are as following but I don't think these are major issues.
1) This repo will grow over time and engineers will need to pull all of it before contributing their own scripts
2) Engineers will need to be more careful about not pushing any secrets in those scripts

https://redd.it/13jyvue
@r_devops

Is Backstage a good solution for the needs of our project?

I'm not sure if this is the right subreddit for this question, but here it goes:

Our clients usually demand a Minimum viable product or Proof of concept before commiting to a project.

Right now we do this MVP from scratch. That means that we set up a dev, test, and prod environment, we setup the right CI/CD workflows, create necessary documentation and so on for each proposal to the client.

A lot of the time the programming languages, frameworks and tools are different so it has been difficult to reuse basically anything.
On top of that, these MVPs are created by different teams inside the organization, making the MVP an isolated project that no one has acces to besides the development team.
And seeing that the developer's experiences ranges from seniors to fresh graduates, the whole thing becomes a mess (Or in other words, it doesnt have the consistency or quality that we are aiming for).

Our department has been given the task to create a solution that automates this process.

The requirements are:

1- With "minimum work" we need to be able to deliver a blank project (for example a webpage that displays hello world after calling a backend API) + tests.

2- This blank project must have all the documentation needed for the developers to start working efficiently and it has to be easily accessible.

3- Needs to have all the CI/CD workflow working from the start. It would be a plus if it can be customizable (could use different tools depending on the project needs or clients request).

4- Optional: have terraform code ready to be used.

Another member of the team suggested that we build our own solution and I'm mostly against it (Mainly because of time and money constrains aswell as lack of seniors in our team).
I have been researching Backstage and it looks like its a good solution for our needs, but if I'm being completely honest, I'm having a hard time understanding it.

I want to ask if someone that used or knows of Backstage knows if its the right tool.
Also, I'm open to any other suggestions since I'm a little bit lost with all the tools there are.

Sorry for the long post and thank you in advance!

https://redd.it/13k0h4n
@r_devops

Didn’t get hired because interview was too good

Been studying my ass off and i used GPT to generate interview questions and answers i might be asked during the interview to practice. Unfortunately, I practiced a bit too much and they gave the offer to their second choice because my interview was perfect. Any advice on what i should do to avoid this outcome again?

https://redd.it/13k6pvf
@r_devops

Looks like GitHub is responding to the chronic downtime they have been having

https://github.blog/2023-05-16-addressing-githubs-recent-availability-issues/

https://redd.it/13k7v8i
@r_devops

Advanced End-to-End DevOps Pipeline for a Java Web Application: A Step-by-Step Guide

Hi Everyone,

I've created this Project which simulates a real world CICD pipeline for deploying a Java weba application on Kubernetes cluster on AWS.

https://mandeepsingh10.hashnode.dev/advanced-end-to-end-cicd-pipeline-for-a-java-web-application-a-step-by-step-guide#heading-references

https://redd.it/13k6nsn
@r_devops

Infrastructure As Code - Trying to setup an automation around a very messy tech stack

As the title stated, our tech stack is unique and rough around the edges. I want to see how can I make the best out of it.We currently have:

1. Setting up requests in Service-Now (For Hardware - Kubernetes Clusters)
2. Trigger Pipelines (via Jenkins) for creating namespaces, deploying ISTIO & Nginx
3. Requesting Certificates (Internal & third party vendor cert requests) & uploading them.
4. Deploying OpenTelemetry Agents (elk, splunk... etc etc etc)
5. Configure ISTIO Secrets, Confit-Gateways

​

I know I can't leverage a single IaC tool (like Terraform or Ansible) to set these up. I want to get different perspectives here in the group to get more ideas on the topic.

https://redd.it/13k9wy7
@r_devops

Open-source IAM Access Visualizer

Hey folks!

Recently created an IAM access visualizer that displays access relationships between AWS identities and resources.
It’s part of an open source cloud security platform that we maintain.

Some potential use cases we wanted to address:

Which IAM roles can become effective admin?
Which IAM roles can read data on your sensitive S3 bucket?
What's the blast radius of an EC2 instance compromise?
What IAM privilege escalations exist in your environment?


Would love your feedback on if something like this is helpful for your cloud IAM workflows!


Click around the Sandbox Environment
Check out our Loom Demo
Check out the Github Repo

https://redd.it/13k8qao
@r_devops

Create Service Now requests via Ansible - Possibility

I am currently working on updating our configuration management system and want to see this possibility of creating Service-Now requests via Ansible.

Are there api's available from Service-Now for us to automate request creations?

​

Cheers!!!

https://redd.it/13kd8xk
@r_devops