Reddit DevOps
277 subscribers
69 photos
32.2K links
Reddit DevOps. #devops
Thanks @reddit2telegram and @r_channels
Download Telegram
Are you team A or team B?
https://redd.it/1vg0i1u
@r_devops
Uni vs non Uni degree

For context, I'm a DevOps engineer who never went to uni, started in SysAdmin, have bit more of an old-school philosophy and my first hiring manager was a CCIE who went to trade school like me and accidentally fell into I.T (like me).

I thought a discussion might be worthwhile, as people always ask about getting into a non entry lvl DevOps position. And I guess people don't see most of us are I.t guys who have seen prod burn from the sidelines before we ended up here.

I managed to get here through some pretty hard work and solving irl problems, and made my promotions into DevOps by recognition of my work and the fact it probably already was DevOps before I wore the hat.

I never chose the DevOps life to be honest, it chose me. I was a happy SysAdmin, I am a very disgruntled DevOps engineer but I make more money and deal with more interesting problems.

What's everyone's thoughts, you think uni is required or not?

(If your a manager, would you hire someone like me, who had years of SysAdmin but no official DevOps title/experience, or are you choosing the CS grad?)

I got nothing against CS grads, but damn, they like to ask 50 questions, like to suggest why everything isn't a binary, and have a tendency to think of a rocket ship solution when the daily boat just needs a hole patched to stop it sinking.

My hypothesis comes from maintaining 2 different 20+ year old products for context ha

https://redd.it/1vg0igb
@r_devops
Why is context switching between ide and logs still a problem?

Something that keeps coming up during our incident response is just how much time we lose jumping between the ide and whatever tool holds the relevant logs, traces, or metrics. Typical flow: you are in the ide looking at a failing code path, you hit unexpected behavior and the next 20 minutes is alt-tabbing between your editor, log search, a distributed tracing ui, metrics dashboards, feature flag console and deploy history. You copy a trace id from logs over to the tracing tool then you copy a user id back into a sql query then you try to map all of that back to the exact function and commit you are staring at in the ide. We've got what most people would call a modern observability stack: distributed tracing, structured logs, dashboards, decent tagging and reasonably instrumented services. the problem isn't that the telemetry doesn't exist, it's that none of it really lives where developers spend their time writing and reviewing code. During incidents, people end up doing their own ad‑hoc integration work: copy from log search, paste into the ide, grep locally, jump back to the metrics dashboard, repeat. The pain points i keep seeing during production debugging are pretty consistent. there's no single place that shows this line of code, these commits, these deploys and these recent errors and traces in one view. Most observability tools are optimized for operators staring at dashboards, not developers trying to understand how a specific code path behaves in production. even when telemetry is tagged correctly, you still have to remember which query or dashboard to open and how to line it up with what you're debugging in the ide and during a live incident, that context‑switching overhead turns directly into mttr and oncall fatigue. What's interesting is that we keep buying more observability tooling but the core developer workflow is still: ide here, production reality over there and your brain plus clipboard as the glue connecting the two. How have you cut down on context switching between the ide and your logs, traces and metrics during debugging and incident response, whether that's pulling production context directly into the ide, pushing more code context into your observability tools or standardizing on a single pane for incident work?

https://redd.it/1vg5rfq
@r_devops
How do you prove a deployment wasn't the cause?

Hi all, one thing thats been bugging me lately is how often the most recent deployment becomes the default suspect.

Service starts acting up at 2:15, someone notices there was a deploy at 2:07, and from that point on everyone's looking at the release. Then an hour later it turns out the actual problem was an upstream dependency having a bad few minutes

We've started pulling deployment events into incident reviews because it gives us a better timeline, but its still surprisingly easy to get anchored on something changed instead of what changed first.

For teams with a lot of deployments everyday, how are you separating coincidence from causation? Do you have a workflow that's worked well, or is this just part of the job?

https://redd.it/1vg69zp
@r_devops
How do you manage multiple environments when Dev and Prod use different infrastructure?

I currently manage 2 different environments: a dev server running in ec2 and an EKS environment for production server. Problem is that their setup is different, which adds extra management and makes it harder to test prod changes before deployment.

I can spawn a UAT EKS for load testing and preparing for prod but it would be just too expensive. I already raised the cost concerns with EKS that this would be an expensive and unnecessary setup but the clients wanted it so I did it. Now they're complaining with cost.

I'm just trying to find the best way to manage the current architecture without increasing costs too much.

How would you handle this?

https://redd.it/1vg6nuu
@r_devops
Confused about how different environments factor into building MLOps systems?

Assume you are building an MLOps platform which has pipelines for the entire model lifecycle written with code and storage components. Something I am struggling to understand is how CI/CD factors into the MLOps system itself that brings the model through the entire lifecycle when we have multiple deployment environments.

The MLOps system takes a model through the entire lifecycle (get data, preprocess, train, validate, promote, deploy, monitor) in reproducible and automated workflows, but these workflows in the MLOps system need to be tested and validated with CI/CD in different deployment environments.

Are the pre-production environments (dev, test, uat...) meant only for ensuring that the MLOps system (pipelines, artifact storage, monitoring) works - where finally, and only, in the production environment that has passed all the tests and checks, does the model go through the entire lifecycle from dataset curation to deployment and monitoring, and each environment has isolated model/artifact registries (and feature stores) for testing that the system works.

Or, does the model meant for production go through each step in the model lifecycle together with the MLOps system as it moves through different environments until it finally reached production where both the "MLOps System" and the "Model" are production-ready and deployed to interact with real users.

It's a little confusing for me.

https://redd.it/1vgab26
@r_devops
Does being in Devops mean no work life balance?

I've got about 6 years of overall experience, with the last 1.5-2 years specifically in DevOps/infra, based in India. Lately it feels like there's no real boundary on my day. I've been working well past standard office hours, like basically every single day. AND I MEAN LIKE ALMOST 12 HOURS

A few things I'm trying to figure out:

\\- Is this just what DevOps is?

\\- Is this a "you get better at this and it calms down" thing, or does it stay this way as you grow?

\\- Is this specific to my company, or to working in India, or is this the norm across the field globally? Would switching companies actually fix it?

Would love to hear from people a few years further in, does it get better, or is this just what we signed up for?

https://redd.it/1vgdzsj
@r_devops
What does AI agentic workflows hype\adoption remind you about ?

I am almost 50, and was working as an SW engineer, an early DevOps adapter (when it was not just another tool to keep siloing system administrator from "stupid" SW).

Hype\\adoption of "agentic workflows" for SRE\\pro active monitoring\\self healing\\<put your magic here> really reminds me of those days when AWS released lambdas.

Everything was lambdas, all applications must be put to lambdas, a bit later - "serverless functions" appeared as a poor man self hosted alternative to aws lambdas, etc.

2026, lambdas are still here. CloudFlare looks like reviving the concept with their workers. ¡Viva la serverless!

But, back to the beginning, don't you feel the same vibes currently, while reading about "agentic workflows" ?

PS: no issues with AI, it's inevitable. Just to share some feelings.

PPS: blocked bots: zeke780

https://redd.it/1vgha5g
@r_devops
I built an open-source tool to deploy and tear down real OpenShift clusters on AWS (UPI, Terraform + Ansible, MIT)

Overview of the GUI

I wanted a real OpenShift lab on AWS to practise on, and I specifically wanted UPI — user-provisioned infrastructure — because it's the install method where nothing is hidden. You build the VPC, the load balancers and the DNS records, and the installer only takes over once that's all correctly in place. IPI hides most of that inside the installer binary; ROSA is managed.

The problem with UPI is that doing it by hand is slow and easy to get wrong, and I couldn't find anything that automated the whole lifecycle for AWS. So I wrote one.

**What it does**

`cluster.yaml` is the only file you edit. Everything else — the Terraform variables, the Ansible variables, `install-config.yaml` — is generated from it.

```

ocplab deploy # terraform apply, wait for bootstrap, approve CSRs, finalize

ocplab verify # ClusterVersion, node readiness, ClusterOperators

ocplab cost # what's deployed right now, priced live

ocplab repair # recreate a worker AWS reclaimed, approve its CSRs

ocplab power off # graceful shutdown, as an alternative to destroying

ocplab destroy # ordered teardown, including what the cluster made itself

```

Some things it handles that took me a while to get right:

\- **Version pinning.** One field pins the installer, the `oc` client and the RHCOS AMI together, cached locally.

\- **Spot instances** for compute and bootstrap, with a "minimal" profile that uses them. Not for the control plane — UPI has no ControlPlaneMachineSet, so a reclaimed master isn't replaced automatically.

\- **Cost visibility.** `ocplab cost` reads live AWS state and prices it, Spot at Spot rates, and it's power-state aware.

\- **A budget safety net.** AWS Budget with alerts, an automatic lockdown action, and a daily killswitch Lambda — managed outside Terraform so it survives a failed teardown.

\- **Optional capability trimming**, which cuts the container images pulled per deploy by about a third.

\- **Teardown that actually completes.** The ingress operator creates a load balancer, a security group and DNS records outside Terraform; the teardown removes them in the right order first.

**The web UI**

There's also a local browser UI (`ocplab web start`) if you'd rather not watch a 40-minute deploy in a terminal: a status dashboard, a `cluster.yaml` editor that validates before saving, and live output for every operation — including Terraform's resource-by-resource log and the installer's own log. It binds 127.0.0.1 only and runs the same commands as subprocesses, so there's no second implementation to keep in sync.

**Scope**

It's built for labs and PoCs, not production. Single AZ, single NAT gateway, no HA design beyond etcd's own quorum. It costs roughly $0.83–1.06/hour while it's up, so the intended flow is create, test, destroy the same day.

Tested end to end against OpenShift 4.22.6 and 4.22.7. MIT licensed.

https://github.com/LuixyToledo97/openshift-upi-aws

Happy to answer questions, and issues are welcome if you try it and something doesn't work.

https://redd.it/1vgjgkx
@r_devops
how to create N integration environments for integration heavy apps
https://redd.it/1vgflsh
@r_devops
First day finally receiving accesses as a Jr and It really fucking sucked.

Today I finally got a ticket and I was quite shocked to realize that I understand next to nothing about their systems even after reading a couple docs that explained the architectural walk through

I don't know why they pick X ports, I don't know how the services interact together and I don't know how anything is connected or how to implement monitoring. I am on probation, 2 months and afraid I might not pass it. It's remote so there is no way to have a senior around and the team is fairly small despite the size of the org which is prominent and well established.

The seniors openly dislike hand holding and I don't know what to do, it's frustrating and they are responsible for assessing me that the end of this whole thing. At the same time afraid that I might ask too much and get written down as downright incompetent.

How on earth can I survive here when everything is so complex and connected? Shit I never coded an API or whole app just simple things and I genuinely don't know what it takes for an app to communicate with all its components.

Took me a fucking whole day to write notes on the architecture and spit out a document with questions that seemed semi decent and I understood perhaps only 45% of it

I have no Dev exp , am a new grad and a couple of months ago I was going through an awful time and working a hotel job and aiming for sysadmin. anyway no idea why they chose me.

Three people vouched for me and that also fucking sucks because if I don't do well their judgement will get called out inside the org.

I don't want to quit but I genuinely am concerned that I might not make the cut and the guilt is killing me

https://redd.it/1vgo89x
@r_devops
can anyone help me with my problem ?

I am looking for to be a SRE. I am just a student in 10th class. I finished networking fundamentals and leaning Linux. But i got a problem , I can't stop using windows. I am a tech enthusiast so i spend lot of time to master windows. I use many tools and software which save lot of my time and can do my tech research efficiently. ya, Linux have many alternative but Linux softwares are not suitable for my tech research. I want to master linux like windows but i can't stay in Linux after sometimes i switch dual boot 1st priority to windows. As a senior of mine. Can anyone help me, please.

I really need help . also which programming language should i learn? Is that will be good to learn golang or python first then switch to Linux .

https://redd.it/1vgq5y2
@r_devops
How is your SecOps team handling Claude Code / Copilot access for proprietary repos?

Our security team flat-out refuses to let cloud AI agents scan whole proprietary repos or run freely on dev machines, but the productivity gap is getting hard to ignore.

We’ve been playing with a middle ground: mapping repo trees locally first, pulling only specific context/signatures into the prompt, and making every diff require manual write approval on disk.

Are you guys using enterprise SaaS zero-retention SLAs, running local models, or putting proxy/mapping layers in front of web models? What's actually working in practice?

https://redd.it/1vgqksj
@r_devops
Training Recommendations

As part of the Security team I work closely with our DevOps engineers. What are some good resources to get some hands on experience in a lab environment to learn the foundations of DevOps? With a focus on AWS. TIA.

https://redd.it/1vgr34k
@r_devops
Do I need to learn now how to write python or shell scripting?

Hello Everyone! I started as an intern as a DevOps Trainee and Now I am a full time here , and most of the shell scripts or python automation scripts I need is generated by Claude/ GPT . So I wanted a headsup as to how much should i know to write them by myself because when i get them done from AI I feel that why I can't write such big automation scripts point being considered as a beginner I cannot write but still that self doubt comes. Also Why i use AI because company wants that and want their tasks to finish fast so there's no room to write on my own . I want to switch next year around May -Jun so I need help with that as well that what more should I study or keeping on learning to get a good paying job.

My current capabilities include intermediate or beginner level AWS , Azure Exp., Docker , Linux, Kubernetes and Terraform , Jenkins . Please I need genuine help!

https://redd.it/1vgspo4
@r_devops
Anyone still using Jenkins?

I mean as title says, I've used Jenkins couple of years ago. I can still fire it up now just to test, but on current alternative tools/stacks available specially in enterprise, do you still use Jenkins? Why keep it? why not change it?

https://redd.it/1vgzwr7
@r_devops
What actually breaks your focus during the day?

I've been struggling to stay focused during my work days lately , either interrupted by emergencies, unwanted conversations, multiple not organized micro-tasks. I want to know about your experiences .


Not looking for a specific answer, just want to hear what it's actually like and how you deal with it.

https://redd.it/1vh0rsr
@r_devops
Migrating From Gitlab to Azure DevOps

We're an on prem shop and our manager wants to migrate all DevOps teams from Enterprise Gitlab self managed to ADO on prem.

We don't know what to expect with this move, anyone has experience with ADO on prem ? what kind of challenges would we face if our entire Gitlab infra (platform + runners) are hosted on K8s.

https://redd.it/1vh40yh
@r_devops
Why not do your own opensource infra?

I think this is more of a rant post, or despair.

I am a linux sysadmin by trade, and doing a lot of linux work with scripting here and there, have setup a lot of self hosted stuff.

Now at my organisation I am setting up Loki with self hosted S3 options. This is my second logging infra, the first one was based on Elasticsearch stack.

I have deployed and maintained a lot of other FOSS stuff as well. Deploying stuff is all I know, difficult and complex systems.

However, very often I see posts here saying something along the lines of:

Do not deploy your own logging infra. Just do EKS. It is cheaper than paying someone to do it for you self hosted.

Look, I wish I was a good software developer and write fantastic code. But no matter how many times I try learn programming I just can't, either I run out of time or get demotivated.

Setting up and running good FOSS infra is all I know and at this rate, I will be out of job if everyone would just go to datadog and AWS.

All this devops stuff like monitoring, prometheus, logs, OTEL, kubernetes, it is all interesting and I can do it, all except writing the actual product.

I should pick up another backup trade I guess.

https://redd.it/1vh7t76
@r_devops
It's 2030 and the marketing dudes at a CICD company accidentally get access to Mythos 6.7 and ask for help making their competitors look bad

https://preview.redd.it/39x5bnkzmthh1.png?width=1490&format=png&auto=webp&s=ba5505b1b89f0078b07cef661836f425ef6f4726



https://redd.it/1vhh5kk
@r_devops