What's the most time you've lost to a bug that wasn't actually a bug?

I spent way too long trying to fix something recently before realizing nothing was actually wrong.

It turned out to be a configuration mistake that I had made myself.

It got me wondering how much time people in ML and software spend debugging their own setups rather than actual code.

What's the dumbest "bug" you've ever chased?

https://redd.it/1vf8x4l
@r_devops

Does anyone have information on hardening publically exposed k8s LB nodes? Loadbalancers in AWS are really expensive.

Just exploring the space and price comparisons

ALBs and NLBs are expensive. especially in AW

So some ingress only nodes with proper hardening seems cheaper, but curious about battle scars and stories people wanna share.

My idea is simply that, hardening a fleet of ingress only nodes in a public subnet, pointing all queries at these nodes and letting the proxy and ingress rules handle the traffic from there

https://redd.it/1vfi0gg
@r_devops

How much attention is harness engineering getting?

AI model quality is converging, or at least changing often enough that chasing the newest thing doesn't seem like a good strategy. How much emphasis are teams putting on building a solid harness into which new models can fit?

Our clients are generally in high compliance industries, so there is thought put into the harness, but what about smaller teams or ones that aren't required into a compliance framework?

https://redd.it/1vfkoi4
@r_devops

Hetzner Cloud vs Turkish VPS for a production mobile backend?

Hi everyone,



I'm building a mobile application primarily for users in Turkey.



My stack:

\- NestJS

\- PostgreSQL

\- Redis

\- Docker

\- Coolify



The app is still in development and I'm planning the production infrastructure.



I'm considering two options:



1. Hetzner Cloud (Germany)

2. A VPS/Dedicated server located in Turkey



The backend is a standard REST API (no gaming or real-time voice/video). Most requests are authentication, CRUD operations, QR code lookups, notifications, and media metadata.



Latency from Turkey to Germany seems to be around 25-40 ms.



Would you prioritize:

\- Better hardware and infrastructure (Hetzner)

\- Lower latency with a Turkish provider



Has anyone run production applications for Turkish users from Hetzner? I'd love to hear real-world experiences.

https://redd.it/1vfkjvd
@r_devops

KodeKloud Challenges are worth it?

I am thinking on start the 100 days of DevOps challenge in KodeKloud just to improve my devops skills and fill in some gaps that I might have.

Does this challenge worth it? Specially for the time that needs to be invested.

https://redd.it/1vfqoqe
@r_devops

Self-hosted enterprise app builders with strong RBAC and compliance features

We’ve been evaluating self-hosted low-code and internal application builder platforms mainly for compliance-heavy requirements like fine-grained RBAC, audit logging, and enterprise access control, and while tools like Retool and ToolJet are still commonly used in this space, they now feel more oriented toward managed or hybrid deployments rather than purely self-hosted governance-first setups, and Appian is still an option but feels closer to a full enterprise BPM suite than a lightweight internal tool builder, so we’re now looking for more modern self-hosted alternatives that can be deployed via Docker or Kubernetes, support strong role-based permissioning, integrate with SSO/SAML or LDAP, provide detailed audit trails, and avoid heavy vendor lock-in since we’re planning to migrate around 300 applications, so it would be useful to hear what stacks are actually working reliably in production environments today.

https://redd.it/1vfxku2
@r_devops

Are you team A or team B?
https://redd.it/1vg0i1u
@r_devops

Uni vs non Uni degree

For context, I'm a DevOps engineer who never went to uni, started in SysAdmin, have bit more of an old-school philosophy and my first hiring manager was a CCIE who went to trade school like me and accidentally fell into I.T (like me).

I thought a discussion might be worthwhile, as people always ask about getting into a non entry lvl DevOps position. And I guess people don't see most of us are I.t guys who have seen prod burn from the sidelines before we ended up here.

I managed to get here through some pretty hard work and solving irl problems, and made my promotions into DevOps by recognition of my work and the fact it probably already was DevOps before I wore the hat.

I never chose the DevOps life to be honest, it chose me. I was a happy SysAdmin, I am a very disgruntled DevOps engineer but I make more money and deal with more interesting problems.

What's everyone's thoughts, you think uni is required or not?

(If your a manager, would you hire someone like me, who had years of SysAdmin but no official DevOps title/experience, or are you choosing the CS grad?)

I got nothing against CS grads, but damn, they like to ask 50 questions, like to suggest why everything isn't a binary, and have a tendency to think of a rocket ship solution when the daily boat just needs a hole patched to stop it sinking.

My hypothesis comes from maintaining 2 different 20+ year old products for context ha

https://redd.it/1vg0igb
@r_devops

Why is context switching between ide and logs still a problem?

Something that keeps coming up during our incident response is just how much time we lose jumping between the ide and whatever tool holds the relevant logs, traces, or metrics. Typical flow: you are in the ide looking at a failing code path, you hit unexpected behavior and the next 20 minutes is alt-tabbing between your editor, log search, a distributed tracing ui, metrics dashboards, feature flag console and deploy history. You copy a trace id from logs over to the tracing tool then you copy a user id back into a sql query then you try to map all of that back to the exact function and commit you are staring at in the ide. We've got what most people would call a modern observability stack: distributed tracing, structured logs, dashboards, decent tagging and reasonably instrumented services. the problem isn't that the telemetry doesn't exist, it's that none of it really lives where developers spend their time writing and reviewing code. During incidents, people end up doing their own ad‑hoc integration work: copy from log search, paste into the ide, grep locally, jump back to the metrics dashboard, repeat. The pain points i keep seeing during production debugging are pretty consistent. there's no single place that shows this line of code, these commits, these deploys and these recent errors and traces in one view. Most observability tools are optimized for operators staring at dashboards, not developers trying to understand how a specific code path behaves in production. even when telemetry is tagged correctly, you still have to remember which query or dashboard to open and how to line it up with what you're debugging in the ide and during a live incident, that context‑switching overhead turns directly into mttr and oncall fatigue. What's interesting is that we keep buying more observability tooling but the core developer workflow is still: ide here, production reality over there and your brain plus clipboard as the glue connecting the two. How have you cut down on context switching between the ide and your logs, traces and metrics during debugging and incident response, whether that's pulling production context directly into the ide, pushing more code context into your observability tools or standardizing on a single pane for incident work?

https://redd.it/1vg5rfq
@r_devops

How do you prove a deployment wasn't the cause?

Hi all, one thing thats been bugging me lately is how often the most recent deployment becomes the default suspect.

Service starts acting up at 2:15, someone notices there was a deploy at 2:07, and from that point on everyone's looking at the release. Then an hour later it turns out the actual problem was an upstream dependency having a bad few minutes

We've started pulling deployment events into incident reviews because it gives us a better timeline, but its still surprisingly easy to get anchored on something changed instead of what changed first.

For teams with a lot of deployments everyday, how are you separating coincidence from causation? Do you have a workflow that's worked well, or is this just part of the job?

https://redd.it/1vg69zp
@r_devops

How do you manage multiple environments when Dev and Prod use different infrastructure?

I currently manage 2 different environments: a dev server running in ec2 and an EKS environment for production server. Problem is that their setup is different, which adds extra management and makes it harder to test prod changes before deployment.

I can spawn a UAT EKS for load testing and preparing for prod but it would be just too expensive. I already raised the cost concerns with EKS that this would be an expensive and unnecessary setup but the clients wanted it so I did it. Now they're complaining with cost.

I'm just trying to find the best way to manage the current architecture without increasing costs too much.

How would you handle this?

https://redd.it/1vg6nuu
@r_devops

Confused about how different environments factor into building MLOps systems?

Assume you are building an MLOps platform which has pipelines for the entire model lifecycle written with code and storage components. Something I am struggling to understand is how CI/CD factors into the MLOps system itself that brings the model through the entire lifecycle when we have multiple deployment environments.

The MLOps system takes a model through the entire lifecycle (get data, preprocess, train, validate, promote, deploy, monitor) in reproducible and automated workflows, but these workflows in the MLOps system need to be tested and validated with CI/CD in different deployment environments.

Are the pre-production environments (dev, test, uat...) meant only for ensuring that the MLOps system (pipelines, artifact storage, monitoring) works - where finally, and only, in the production environment that has passed all the tests and checks, does the model go through the entire lifecycle from dataset curation to deployment and monitoring, and each environment has isolated model/artifact registries (and feature stores) for testing that the system works.

Or, does the model meant for production go through each step in the model lifecycle together with the MLOps system as it moves through different environments until it finally reached production where both the "MLOps System" and the "Model" are production-ready and deployed to interact with real users.

It's a little confusing for me.

https://redd.it/1vgab26
@r_devops

Does being in Devops mean no work life balance?

I've got about 6 years of overall experience, with the last 1.5-2 years specifically in DevOps/infra, based in India. Lately it feels like there's no real boundary on my day. I've been working well past standard office hours, like basically every single day. AND I MEAN LIKE ALMOST 12 HOURS

A few things I'm trying to figure out:

\\- Is this just what DevOps is?

\\- Is this a "you get better at this and it calms down" thing, or does it stay this way as you grow?

\\- Is this specific to my company, or to working in India, or is this the norm across the field globally? Would switching companies actually fix it?

Would love to hear from people a few years further in, does it get better, or is this just what we signed up for?

https://redd.it/1vgdzsj
@r_devops

What does AI agentic workflows hype\adoption remind you about ?

I am almost 50, and was working as an SW engineer, an early DevOps adapter (when it was not just another tool to keep siloing system administrator from "stupid" SW).

Hype\\adoption of "agentic workflows" for SRE\\pro active monitoring\\self healing\\<put your magic here> really reminds me of those days when AWS released lambdas.

Everything was lambdas, all applications must be put to lambdas, a bit later - "serverless functions" appeared as a poor man self hosted alternative to aws lambdas, etc.

2026, lambdas are still here. CloudFlare looks like reviving the concept with their workers. ¡Viva la serverless!

But, back to the beginning, don't you feel the same vibes currently, while reading about "agentic workflows" ?

PS: no issues with AI, it's inevitable. Just to share some feelings.

PPS: blocked bots: zeke780

https://redd.it/1vgha5g
@r_devops

I built an open-source tool to deploy and tear down real OpenShift clusters on AWS (UPI, Terraform + Ansible, MIT)

Overview of the GUI

I wanted a real OpenShift lab on AWS to practise on, and I specifically wanted UPI — user-provisioned infrastructure — because it's the install method where nothing is hidden. You build the VPC, the load balancers and the DNS records, and the installer only takes over once that's all correctly in place. IPI hides most of that inside the installer binary; ROSA is managed.

The problem with UPI is that doing it by hand is slow and easy to get wrong, and I couldn't find anything that automated the whole lifecycle for AWS. So I wrote one.

**What it does**

`cluster.yaml` is the only file you edit. Everything else — the Terraform variables, the Ansible variables, `install-config.yaml` — is generated from it.

```

ocplab deploy # terraform apply, wait for bootstrap, approve CSRs, finalize

ocplab verify # ClusterVersion, node readiness, ClusterOperators

ocplab cost # what's deployed right now, priced live

ocplab repair # recreate a worker AWS reclaimed, approve its CSRs

ocplab power off # graceful shutdown, as an alternative to destroying

ocplab destroy # ordered teardown, including what the cluster made itself

```

Some things it handles that took me a while to get right:

\- **Version pinning.** One field pins the installer, the `oc` client and the RHCOS AMI together, cached locally.

\- **Spot instances** for compute and bootstrap, with a "minimal" profile that uses them. Not for the control plane — UPI has no ControlPlaneMachineSet, so a reclaimed master isn't replaced automatically.

\- **Cost visibility.** `ocplab cost` reads live AWS state and prices it, Spot at Spot rates, and it's power-state aware.

\- **A budget safety net.** AWS Budget with alerts, an automatic lockdown action, and a daily killswitch Lambda — managed outside Terraform so it survives a failed teardown.

\- **Optional capability trimming**, which cuts the container images pulled per deploy by about a third.

\- **Teardown that actually completes.** The ingress operator creates a load balancer, a security group and DNS records outside Terraform; the teardown removes them in the right order first.

**The web UI**

There's also a local browser UI (`ocplab web start`) if you'd rather not watch a 40-minute deploy in a terminal: a status dashboard, a `cluster.yaml` editor that validates before saving, and live output for every operation — including Terraform's resource-by-resource log and the installer's own log. It binds 127.0.0.1 only and runs the same commands as subprocesses, so there's no second implementation to keep in sync.

**Scope**

It's built for labs and PoCs, not production. Single AZ, single NAT gateway, no HA design beyond etcd's own quorum. It costs roughly $0.83–1.06/hour while it's up, so the intended flow is create, test, destroy the same day.

Tested end to end against OpenShift 4.22.6 and 4.22.7. MIT licensed.

https://github.com/LuixyToledo97/openshift-upi-aws

Happy to answer questions, and issues are welcome if you try it and something doesn't work.

https://redd.it/1vgjgkx
@r_devops

how to create N integration environments for integration heavy apps
https://redd.it/1vgflsh
@r_devops

First day finally receiving accesses as a Jr and It really fucking sucked.

Today I finally got a ticket and I was quite shocked to realize that I understand next to nothing about their systems even after reading a couple docs that explained the architectural walk through

I don't know why they pick X ports, I don't know how the services interact together and I don't know how anything is connected or how to implement monitoring. I am on probation, 2 months and afraid I might not pass it. It's remote so there is no way to have a senior around and the team is fairly small despite the size of the org which is prominent and well established.

The seniors openly dislike hand holding and I don't know what to do, it's frustrating and they are responsible for assessing me that the end of this whole thing. At the same time afraid that I might ask too much and get written down as downright incompetent.

How on earth can I survive here when everything is so complex and connected? Shit I never coded an API or whole app just simple things and I genuinely don't know what it takes for an app to communicate with all its components.

Took me a fucking whole day to write notes on the architecture and spit out a document with questions that seemed semi decent and I understood perhaps only 45% of it

I have no Dev exp , am a new grad and a couple of months ago I was going through an awful time and working a hotel job and aiming for sysadmin. anyway no idea why they chose me.

Three people vouched for me and that also fucking sucks because if I don't do well their judgement will get called out inside the org.

I don't want to quit but I genuinely am concerned that I might not make the cut and the guilt is killing me

https://redd.it/1vgo89x
@r_devops

can anyone help me with my problem ?

I am looking for to be a SRE. I am just a student in 10th class. I finished networking fundamentals and leaning Linux. But i got a problem , I can't stop using windows. I am a tech enthusiast so i spend lot of time to master windows. I use many tools and software which save lot of my time and can do my tech research efficiently. ya, Linux have many alternative but Linux softwares are not suitable for my tech research. I want to master linux like windows but i can't stay in Linux after sometimes i switch dual boot 1st priority to windows. As a senior of mine. Can anyone help me, please.

I really need help . also which programming language should i learn? Is that will be good to learn golang or python first then switch to Linux .

https://redd.it/1vgq5y2
@r_devops

How is your SecOps team handling Claude Code / Copilot access for proprietary repos?

Our security team flat-out refuses to let cloud AI agents scan whole proprietary repos or run freely on dev machines, but the productivity gap is getting hard to ignore.

We’ve been playing with a middle ground: mapping repo trees locally first, pulling only specific context/signatures into the prompt, and making every diff require manual write approval on disk.

Are you guys using enterprise SaaS zero-retention SLAs, running local models, or putting proxy/mapping layers in front of web models? What's actually working in practice?

https://redd.it/1vgqksj
@r_devops

Training Recommendations

As part of the Security team I work closely with our DevOps engineers. What are some good resources to get some hands on experience in a lab environment to learn the foundations of DevOps? With a focus on AWS. TIA.

https://redd.it/1vgr34k
@r_devops