How you guys are avoiding tying to any specific GPU vendor for AI workloads?

With more and more teams using LLMs and inference services wondering how everyone is making decisions about their infrastructure like are you choosing one major cloud provider, using Kubernetes across multiple providers or trying platforms like RunPod, CoreWeave, Yotta Labs, Lambda to keep things flexible?

more interested in whether anyone has found a real way to move workloads between different GPU providers without having to maintain completely separate deployment processes.

https://redd.it/1vawhik
@r_devops

How do experienced teams answer "What changed?" during a production incident?

I've been curious how different teams approach this in practice.

Imagine you're on call and something in production starts behaving differently. It's not immediately obvious whether it's a deployment, infrastructure change, configuration change, scaling event, cloud service issue, or something else.

Once the alert fires, how do you actually answer questions like:

What changed?

When did it change?

Was the change intentional?

What's the fastest way to confirm the root cause?

I'm less interested in the tools themselves (Terraform, Kubernetes, CloudTrail, Grafana, etc.) and more interested in the actual investigation workflow that experienced engineers follow.

For example:

Where do you usually look first?

Which sources of truth do you trust the most?

What part of the investigation tends to consume the most time?

Are there still manual steps that you wish were easier?

I'm trying to understand how this is handled in real production environments, especially at companies running on AWS/Kubernetes.

Thanks in advance—I’m hoping to learn from people who've been through enough incidents to know what actually works.

https://redd.it/1vatqr6
@r_devops

How many of you went multi-cloud, or switched cloud providers entirely, for a new job?

I have spent most of my career in AWS, but more and more, I am seeing companies asking for skill sets with multiple cloud providers. I feel like the skill set is transferable; if you know how to spin up EKS then spinning up a GKE cluster shouldn't be that much different. It's still Kubernetes under the hood, still installing helm charts, still using cluster autoscaler to provision new nodes, still installing Argo CD via Helm along with Prometheus, Grafana, Loki, etc. All that changes are the resources used and modules on the Terraform/cloud provider level. Am I correct in thinking this or am I way off base here?

https://redd.it/1vazyx8
@r_devops

Formalising methods whilst keeping it informal as a small business mentality

I was employed by a business because they're growing. I need some advice, because part of making changes in company processes is changing ways of thinking. I've helped introduce new frameworks but they seem to be reluctant to change their processes, which makes it difficult to do my job.

I don't want to overstep the mark or looking like I'm overruling/ignoring them

For example, my manager and I have different ideas about approaches to developing a solution. Most of our work where we have different ideas is writing data integrations. My manager is more knowledgeable about the data and underlying system, but I have more technical expertise at the scale we are working, given the new systems/architecture.

The problem is that we're a small team with work from home policy, so catch up meetings get cancelled and put off. The specifications are usually email threads, which are fine for quick fixes or small tasks, whereas I like a central written document/project plan that outlines the requirements and approach, including potential problems or limitations from the outset. Those questions may become something in an email thread, but unless there is a document/project plan then things get lost.

Generally I like standardized approaches at least to get people on the same page and understand accountability. I accept we can't know everything up front and it's easy to go off on a tangent; that's the real world. But without a base to bring everything together and focus priorities, that leads to failure or problems further down the line.

https://redd.it/1vb0wfs
@r_devops

Share your tips and tricks

0 3 * * * /usr/local/bin/send_alerts.sh

https://redd.it/1vb530a
@r_devops

How do you stop thinking about work after your workday ends?

I’ve realized that even after I finish work, my brain keeps thinking about it.
I replay conversations, think about problems I didn’t solve, plan tomorrow’s tasks, or imagine different scenarios. Sometimes it feels like I’m still working hours after I’ve logged off.
I’m curious: how do you mentally switch off?
Do you have any routines, hobbies, or habits that help you leave work at work? Or did it simply get easier with experience?
I’d love to hear what actually works for you.

https://redd.it/1vbb735
@r_devops

I'm lost after 3 years in DevOps. What would you do?

Hi everyone,

I'm looking for advice from people who have been in this field longer than I have.

I'm an Arts graduate who got into DevOps about 3.3 years ago. I've switched 3 companies and recently joined my 4th company, a large GCC. Along the way I've worked with AWS, Kubernetes, Terraform, CI/CD, automation, and earned AWS & CKA certifications.

The problem is that I'm no longer sure what I should be aiming for.

My current role has an exhausting work culture, the learning isn't what I expected, and it has made me rethink my long-term direction. I don't want to keep switching jobs without a clear goal.

If you were in my position today, what would you optimize for over the next 3–5 years?

1. Become a top-tier Platform Engineer/SRE?

2. Move towards AI/LLM engineering?

3. Solutions Architect

4. Security?

5. Target remote international jobs?

6. Prepare to move abroad?

I'd really appreciate hearing from engineers with 8–15+ years of experience. If you were starting over with my experience today, what roadmap would you follow and why?

https://redd.it/1vbottm
@r_devops

Another Friday, another Bitbucket outage.

Who doesn't love Friday releases? Atlassian status page is green as usual.

https://redd.it/1vbpxj4
@r_devops

Derivations to Deployments: Practical Nix in Production

Great talk about how Antithesis uses Nix in production by John Murray. Thought some of you would enjoy. https://youtu.be/WaPOtisNu1k

I've been interested in Nix for a while but have not yet taken the plunge. Do any of you use it work? What has your experience been with it?

https://redd.it/1vbpliq
@r_devops

How do i get my confidence back?!

Hey! Quick background on me — graduated in 2020 from a tier 2 college, landed a third-party role at a Tier1 investment bank as an SRE, and ended up staying 5 years (got converted to full-time after the first 2). Got laid off in May 2025, honestly wasn’t mad about it since I wanted out anyway.
Most of the work i did was on there in house tools and i felt there was no learning curve there

That same month I interviewed for a “DevOps Engineer - Migration Specialist” role and got it. Felt good, got comfortable again. But once I actually joined, I realized it’s basically an L2 support role. Pay was decent though, so I told myself I’d stick around a year, upskill on the side, then jump ship.

Except… I didn’t. The role’s been so low on mental engagement that I just coasted. Now it’s actually time to switch, and I’m stuck in this weird spot — feels like I need to relearn everything from scratch, but every time I try, it’s like “wait, I already know this?” Hard to tell if I’m actually behind or just psyching myself out.

So — what’s a realistic (and ideally not soul-crushing) way to get back on track?
Please give me suggestions for fast-track courses and small projects to build production skills again.

Ps:- Used Ai to fix my crude version of this post

https://redd.it/1vbufg6
@r_devops

What are the top automated root cause analysis tools today?

We had a major production incident a few weeks ago that really exposed how manual our root cause analysis process still is. Incident was pretty standard on paper: error rate spike across a couple of microservices, latency blowup on one critical api, customers hitting timeouts. We have the usual observability stack, distributed tracing, logs, metrics, dashboards, plus tickets and slack channels but getting to "what broke first, and why" was mostly humans correlating tabs manually. The incident response workflow looked like this: jump into the oncall war room, pull up dashboards, zoom in on the time window and pivot between error logs, traces, deploy history, feature flag changes and infra events. someone manually builds an incident timeline, a deploy goes out, error rate spikes shortly after, latency climbs, autoscaling kicks in, db connections start flapping. then another person tries to connect that narrative back to the specific service, function, or config change that caused it, which is the actual root cause analysis part nobody's automated yet. We do have some ai-powered anomaly detection and basic rca signals turned on but in practice they mostly narrow the blast radius instead of giving a confident, explainable root cause. it still feels like humans doing the hard work, with the tooling just providing hints rather than a real diagnosis. Looking for automated root cause analysis tools that can take telemetry, logs, metrics, traces, infra events, together with deploy and feature flag changes and output a credible "this change in this service is most likely the cause, here's why." also interested in platforms where the rca output is good enough to drop straight into a postmortem instead of needing another hour of digging and anything beyond the usual big observability platforms, especially newer ai-native rca tools that sit on top of existing observability instead of trying to replace it, that's noticeably reduced incident triage time or mttr without just adding another noisy dashboard. if you're running any of these in production, what do you still consider "top" for automated root cause analysis today?

https://redd.it/1vbyfzq
@r_devops

Built an open-source Agentic DevSecOps pipeline with SonarQube, Trivy, Claude/Google ADK, policy gates, and container publishing

# I’ve been working on an open-source reference implementation of an agentic DevSecOps container-release pipeline.

The goal was to keep security decisions deterministic while using an AI agent only to explain findings, prioritize remediation, and make scanner output easier for engineering teams to act on.

The pipeline includes:

zSonarQube source-code and quality analysis
Trivy Dockerfile and Kubernetes misconfiguration scanning
Local container-image build
Trivy vulnerability and embedded-secret scanning
Deterministic release-policy evaluation
Claude Sonnet 5 or Google ADK advisory triage
HTML, PDF, Markdown, JSON, and SARIF reports
Email delivery through Resend
Discord notifications
Protected human approval
Docker Hub publishing only after authorization

For the demonstration, the workflow processed an intentionally insecure training fixture and found:

13 SonarQube findings
25 configuration misconfigurations
347 container-image vulnerabilities
372 total Trivy findings
74 policy-blocking findings

The workflow continued scanning long enough to collect complete evidence, but the deterministic policy kept the release BLOCKED. Human approval and image publishing were skipped.

The Claude advisory successfully analyzed a bounded subset of the scanner evidence. It prioritized remediation and explained possible attack paths, but it could not approve, reject, waive, or override the deterministic policy.

The design principle is:

>

The pipeline generates three primary reports:

1. Code and configuration security report
2. Container vulnerability and secret report
3. Consolidated release-security report

Repository:
https://github.com/DevOpsAIguru123/awesome-agentic-devops/tree/main/agents/container-image-release-advisor

Demonstration run:
https://github.com/DevOpsAIguru123/awesome-agentic-devops/actions/runs/30590931717

This is an open-source reference implementation rather than a universal production template. Production adoption would require organization-specific policies, isolated builds, artifact signing, provenance, registry controls, and continuous monitoring.

I’d appreciate feedback on:

Whether AI advisory analysis belongs inside the release pipeline or should run asynchronously
How you deliver security findings without overwhelming developers
Which additional supply-chain controls you would add
Whether a deliberately blocked security workflow should appear red or use a successful “blocked as designed” conclusion

https://redd.it/1vccfq8
@r_devops

Designing a Linux patch management system with Aptly, AWX, PostgreSQL and Grafana

https://preview.redd.it/bhyfdt5t9pgh1.png?width=1920&format=png&auto=webp&s=e13f6a15dac958b9f4dbcf651a8ef954cf940098

A production server rebooted after an unattended kernel update, which made me rethink how Linux patching should work in production.

Rather than just disabling unattended-upgrades, I designed a patch management system around controlled promotion and governance.

The approach uses:

Aptly for internal package mirrors and immutable snapshots
PostgreSQL to enforce promotion gates and maintain an audit trail
Grafana for fleet health and CVE visibility
Two-phase rollback to revert both the repository and client packages

I wrote a detailed article covering the architecture, design decisions, and trade-offs:

BLOG: 2ssk.medium.com/patch-management

REPO: 2SSK/patchops

I'm curious how others handle patch governance.

Do you use internal mirrors, Landscape, Satellite, Foreman, Custom tooling, or something else? What has worked well (or poorly) in your environment?

[AI has been used in drafting this post\]

https://redd.it/1vcelwk
@r_devops

I counted every tool our team touches just to ship one deploy. We hit 14. What's yours?

I did a dumb exercise last week: sat down and wrote out every tool that touches a single deploy on my team, start to finish. Not “tools we have licenses for”, just the ones actually in the critical path of getting one change into production.

Landed on 14.

Source control. The CI runner. A separate container registry. An image scanner that runs after the registry push. A secrets manager. The Kubernetes dashboard we check for rollout status. A log aggregator. A separate APM/tracing tool, because the logs don't show latency well on their own. An alerting tool that is, confusingly, different from the paging tool. A DNS/cert dashboard. A cost/billing dashboard nobody opens until the invoice is a surprise. The ticketing system for the actual incident. And a chatops bot that glues some of this together, badly.

None of these were wrong choices individually. Each one was probably the right call in isolation, at the time it got added. But nobody ever sat down and asked whether this thing talks to the other 13. It's less a stack and more a pile that happens to work most days.

What's bugging me isn't the number itself. It's that almost nobody on the team could tell you the number without doing this exercise. We just live inside it. New hires take weeks just to learn where to look when something breaks, and that's before they've learned what any individual tool actually does.

So, genuinely curious: if you did this exercise on your own team, what would your number be? And more interesting to me: which of your 10+ tools do you think is actually load-bearing, versus which one is just... there, inherited from a decision three engineers ago that nobody wants to be the one to rip out?

Not fishing for “just consolidate everything into one platform” as the answer. I don't think that's automatically true either. Sometimes the sprawl is a symptom of real, unavoidable complexity, not laziness. More curious whether other teams' numbers look like mine, and whether anyone's actually fixed this in a way that stuck.

https://redd.it/1vcdrvq
@r_devops

How deep do you go when monitoring production traffic?

Hi everyone,

I'm interested in how different teams approach this.

Once an application is in production, do you actively monitor incoming traffic patterns, or do you mostly rely on infrastructure and application metrics?

For example, if a service suddenly starts using 80% CPU, is it common practice to investigate:

* Top client IPs
* User-Agents
* Most requested endpoints
* Bots vs. real users
* Traffic anomalies or spikes

Or is that level of traffic analysis only common in larger organizations?

If you do monitor it, what does your observability stack look like? Nginx or Traefik access logs, IIS logs, Loki, ELK, OpenSearch, Splunk, Datadog, or something else?

I'm trying to understand what's considered standard practice in production environments.

https://redd.it/1vcw5jg
@r_devops

I tested an IaC tool with real deploys on AWS, Azure and GCP. 20 scenarios each. Here is what only real clouds teach you
https://redd.it/1vcxs6l
@r_devops

Where do infra/platform engineers actually hang out online these days?

Altho I’m an active lurker here, I’ve recently started using X to try and connect with people working in infra, platform engineering, and devops, but I’m finding it surprisingly difficult to discover that side of the community.

But my feed is still mostly AI hype and general tech content, while I’m more interested in more informal discussions around k8s, ai & cloud infra, observability, networking, platform engineering, etc.

For those of you who are active outside of Reddit, where do you usually discover good infra content or interact with other engineers?

https://redd.it/1vdh4yt
@r_devops

Idempotency in IaC is just an equality check
https://www.spamsbykarim.com/blog/idempotency-in-iac

https://redd.it/1vdhrjb
@r_devops

CS student here... what actually happens after code reaches production?

Hi everyone,

I'm a CS student, and I realized something recently.

In college, we're taught how to write code, debug it, maybe deploy it once, and then we move on to the next assignment. But I keep reading incident reports from companies like GitHub and Cloudflare, and it feels like the real engineering work actually starts after deployment.

I'm curious because I honestly have no idea what that world looks like.

When something breaks in production at 2 AM, what actually happens?

Do you immediately know what's wrong, or is it more like putting together pieces of a puzzle? What's usually the most frustrating part of the whole process?

Also, what's one thing about running software in production that you wish every CS student knew before joining the industry?

I'm not building anything or looking for textbook answers—I just want to understand what the job is really like beyond what we learn in college.

I'd love to hear your stories, even if it's just one incident that taught you something you'll never forget.

https://redd.it/1vdmb5t
@r_devops

the cloudfront vpc origins outage caught me half-migrated, and the parallel workstreams were the real problem

the cloudfront outage on the 16th is still bugging me. the fleet that manages connections to private vpc origins couldn't load updated config, traced back to one frankfurt AZ, and it served 5xx on vpc origin traffic for about three and a half hours. hugging face, tailscale and ubiquiti were among the ones down.

bad timing, because i was in the middle of moving four services off public alb origins onto vpc origins. to hit a compliance date i ran the migrations in parallel instead of one clean cutover at a time.

the outage wasn't the hard part. all four workstreams were editing the same terraform repo and the same shared network module. one branch tightened a security group, another assumed the old rule was still there, and a third rebased on both and quietly reverted half of it. i spent more time untangling my own branches than dealing with aws.

what kept it sane was isolating each migration so the changes didn't collide. i run each one as its own task in verdent, parallel lines of work in the same codebase without branches stepping on each other. at least i had a clean diff per service to reason about.

the ugly part: the agent's config for one service moved the origin to vpc origins but left the alb's public security group open to 0.0.0.0/0. on paper we were private, in reality the load balancer was still reachable from the internet, which is the whole reason we were migrating. i only caught it because the diff on the shared module looked off.

what i still haven't solved is sequencing. running four in parallel meant that during the actual outage, two services were on vpc origins and degraded while two were still public and fine, and my runbook assumed one uniform state. my only fix is a per-service origin flag and a single cutover window, which just trades the messy middle for a concentrated blast radius.

https://redd.it/1vdovj6
@r_devops

I'm so tired of pushing 10 times just to fix one GitHub Actions error

Every time I touch a workflow file it's the same loop, small change, push, wait for the runner, watch it fail on some tiny thing, tweak, push again. Ten commits later it finally goes green. No real way to test it before it runs, so I'm just push-and-praying. Do you hit the same thing? Or is there some other part of CI/Actions that drives you nuts? What you'll guys actually dealing with?

https://redd.it/1vdqoy3
@r_devops