Idempotency in IaC is just an equality check
https://www.spamsbykarim.com/blog/idempotency-in-iac

https://redd.it/1vdhrjb
@r_devops

CS student here... what actually happens after code reaches production?

Hi everyone,

I'm a CS student, and I realized something recently.

In college, we're taught how to write code, debug it, maybe deploy it once, and then we move on to the next assignment. But I keep reading incident reports from companies like GitHub and Cloudflare, and it feels like the real engineering work actually starts after deployment.

I'm curious because I honestly have no idea what that world looks like.

When something breaks in production at 2 AM, what actually happens?

Do you immediately know what's wrong, or is it more like putting together pieces of a puzzle? What's usually the most frustrating part of the whole process?

Also, what's one thing about running software in production that you wish every CS student knew before joining the industry?

I'm not building anything or looking for textbook answers—I just want to understand what the job is really like beyond what we learn in college.

I'd love to hear your stories, even if it's just one incident that taught you something you'll never forget.

https://redd.it/1vdmb5t
@r_devops

the cloudfront vpc origins outage caught me half-migrated, and the parallel workstreams were the real problem

the cloudfront outage on the 16th is still bugging me. the fleet that manages connections to private vpc origins couldn't load updated config, traced back to one frankfurt AZ, and it served 5xx on vpc origin traffic for about three and a half hours. hugging face, tailscale and ubiquiti were among the ones down.

bad timing, because i was in the middle of moving four services off public alb origins onto vpc origins. to hit a compliance date i ran the migrations in parallel instead of one clean cutover at a time.

the outage wasn't the hard part. all four workstreams were editing the same terraform repo and the same shared network module. one branch tightened a security group, another assumed the old rule was still there, and a third rebased on both and quietly reverted half of it. i spent more time untangling my own branches than dealing with aws.

what kept it sane was isolating each migration so the changes didn't collide. i run each one as its own task in verdent, parallel lines of work in the same codebase without branches stepping on each other. at least i had a clean diff per service to reason about.

the ugly part: the agent's config for one service moved the origin to vpc origins but left the alb's public security group open to 0.0.0.0/0. on paper we were private, in reality the load balancer was still reachable from the internet, which is the whole reason we were migrating. i only caught it because the diff on the shared module looked off.

what i still haven't solved is sequencing. running four in parallel meant that during the actual outage, two services were on vpc origins and degraded while two were still public and fine, and my runbook assumed one uniform state. my only fix is a per-service origin flag and a single cutover window, which just trades the messy middle for a concentrated blast radius.

https://redd.it/1vdovj6
@r_devops

I'm so tired of pushing 10 times just to fix one GitHub Actions error

Every time I touch a workflow file it's the same loop, small change, push, wait for the runner, watch it fail on some tiny thing, tweak, push again. Ten commits later it finally goes green. No real way to test it before it runs, so I'm just push-and-praying. Do you hit the same thing? Or is there some other part of CI/Actions that drives you nuts? What you'll guys actually dealing with?

https://redd.it/1vdqoy3
@r_devops

Why did you choose to be a DevOps?

What motivated you to choose DevOps as a career? Was there a specific reason or did you somehow end up on this path by accident?

https://redd.it/1vdu0p1
@r_devops

DevOps roadmap?

I'm currently pursuing a Master's in Cloud Computing, but I feel like my courses are very generalized and don't go deep into the practical skills I'm looking for.

My goal is to build a career in DevOps, but I'm honestly feeling overwhelmed and confused about where to start.

I have a few questions for those already working in DevOps or who have successfully made the transition:

Where can I learn DevOps for free? (Courses, documentation, YouTube channels, hands-on labs, etc.)
What are the best platforms to practice real-world DevOps skills?
How many hours should I dedicate each day to learning and practicing?
Realistically, how long does it take to become job-ready?
At what stage should I start applying for entry-level DevOps roles or internships?

I'm not looking for shortcuts. I just want a clear, realistic roadmap from people who've been through this journey. Any genuine advice, resources, or lessons you wish you had known when starting would be greatly appreciated.

Thanks in advance!

https://redd.it/1vdsxx8
@r_devops

AI code reviews under 1c after GPT5.6 Luna price drop

Luna's 80% price cut landed this week so I ran the numbers on my PR review bot.

A real review of a \~180-line PR: about 2,300 tokens in, 590 visible out, which works out around $0.001

The dashboard total for my whole billing window, 15 requests including three full reviews, is 6,244 tokens and $0.00.

Screenshots and links to example code review PR: https://codylabs.uk/blog/ai-code-review-cost/

https://preview.redd.it/3kl6us18bzgh1.png?width=2340&format=png&auto=webp&s=d85a0f590b9dc7f7d4ffa7300c77c128505c8361




https://redd.it/1vdks4s
@r_devops

Is ~$18/mo just the price for a small app that needs real database backups?

Built a small internal billing tool for my own business. One user (me), barely any traffic. I keep landing around $18/month for hosting, which is more than I wanted, and I'm trying to figure out if I'm missing something obvious or if that's just what it costs?

Not a professional developer so apologies if I use any terms wrong.

It's a Node app with a React frontend and a Postgres database, runs in Docker.

What I actually need:
\- Postgres where I can restore to a specific point in time, not just last night's backup. It tracks invoices and payments against Stripe, so if I restore a 24 hour old copy my records won't match what Stripe already charged people. That seems like a bad situation to be in.
\- Somewhere to store generated PDF receipts that doesn't get wiped. These are the actual documents I send customers and I can't just regenerate them later. Seems like a lot of the cheap container hosts have disks that disappear every deploy.
\- A cron job that runs once a day at a set time in my timezone. It's the billing run, so if it drifts to the wrong side of midnight it processes the wrong day.
\- Under $20/month.

What I don't need: high availability, scaling, or speed. It's one person clicking around a few times a day. If it was down for a few hours nothing bad would happen.

What I've already looked at:
\- Fly.io's managed Postgres is $38/mo on its own, way over budget
\- Railway's Postgres backups looked like snapshots only, no point in time restore
\- Supabase Pro is $25 and point in time restore is a $100/mo add-on
\- Cloud Run, App Runner, DO App Platform — filesystem doesn't stick around, kills the PDF requirement
\- Cheap VPS running everything myself is like $8, but then I'm responsible for making sure backups actually work, and honestly I'm not sure I'd notice if they quietly stopped

Best I've come up with is Fly for the app (cheap if the machine sleeps when idle) plus DigitalOcean managed Postgres at $15, which includes 7 day point in time restore.

Is there something cheaper that still lets me actually restore the database? Or is $15$20 ish just what managed Postgres costs and I should stop looking?

https://redd.it/1ve2jsa
@r_devops

Weekly Self Promotion Thread

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

https://redd.it/1ve5h6c
@r_devops

Need Career Advice - Is it worth continuing to chase DevOps?

Hi everyone,

I'm an MCA graduate (2025) from a Tier 3 college in Maharashtra(Pune).

I had worked 1 yrs into Infrastructure Support before enrolling for masters. During my MCA, I also did 2 DevOps internships, and my goal has always been to build a career in DevOps.

Since February, I've been applying for DevOps roles almost every day, tailoring my resume, working on my interview prep, and trying to improve my skills. Unfortunately, I haven't had much success so far, and it's honestly starting to feel a bit discouraging.

My current skills include:

\\- Linux
\\- Networking
\\- AWS & Azure
\\- Terraform
\\- Docker
\\- Kubernetes
\\- Jenkins
\\- GitHub Actions
\\- Prometheus & Grafana

At this point, I feel like I have two choices:

\\- Keep searching for a full-time DevOps role.
\\- Reach out to my previous employer and continue working in Infrastructure Support.

What I'm really worried about is making the wrong decision.

\\- If I keep searching and still don't get a DevOps job after a few more months/years, what would you recommend?

\\- If I go back to Infrastructure Support, will it become much harder to switch into DevOps later?

\\- Has anyone here made the transition from Infrastructure Support to DevOps after a year or two? If yes, what helped you make that move?

I'd really appreciate hearing from people who've been in a similar situation or anyone currently working in DevOps.

Also if anyone has any opportunities for DevOps,SRE,Linux,cloud support or even NOC please help me out as I just want a start my career.

Thanks!

https://redd.it/1ve75i1
@r_devops

If we already have Grafana, Datadog, Splunk, PagerDuty, etc., why are production incidents still so hard?

I've been trying to understand what actually happens during production incidents, and something doesn't add up in my head.

Let's say I'm on call and I get paged at 2 AM because "checkout is broken."

We already have tools like Grafana, Datadog, Splunk, OpenTelemetry, PagerDuty, Kubernetes dashboards, cloud monitoring, CI/CD history, Git, runbooks... basically a ton of observability and operational tools.

So where does the time actually go?

Is it because the information is spread across too many places? Is it understanding what changed? Figuring out the blast radius? Knowing which team or dependency is actually responsible? Or is there something I'm completely missing?

Reading incident postmortems, it feels like engineers already have lots of data, but they still spend a long time piecing together what actually happened before they can confidently act.

For those of you who've been on call, what makes that first 15–30 minutes difficult, even with all these tools available?

I'm genuinely trying to understand what I'm missing as a student because from the outside it feels like we already have a tool for everything, yet companies still spend hours resolving some incidents.

https://redd.it/1ve6uem
@r_devops

Where do you store code for one off tasks that might be useful later on?

I've been a developer for a few decades now, almost always in a DevOps kind of role, and every once in a while I have stumbled over this issue and never found a solution that sits well with me.

As a DevOps guy, I both write and maintain the code, as well as being partly responsible for the operations and the data of the system. In our case the core of the system is a CMS, but it could be a DMS, CRM, IAM etc etc.

And on occation we have a need to do some bulk operation on the data, in a way that has not been done before, and can't be done easily using some UI. If there will be a repeated need for that kind of bulk job then we will likely integrate it fully into the code, and have a user friendly interface for it.

But what about more one off kind of tasks? Tasks that require coding, but that likely won't be needed again in the foreseeable future.

The last time it happened to me, this was roughly the task at hand:

* Identify all objects in the system with properties X, Y and (Z1 or Z2) within section S1.
* Move them all from section S1 to section S2.
* For each object, update property X to a calculated new value.

It is possible to do this manually in the regular GUI for the CMS. But that's not feasable when there are several hundreds of objects matching the criteria. Especially not when it's fairly trivial to write code that does the bulk job.

But after the job has been run (first in dev/test/stage, and then in production), what should happen with that code?

Just deleting it feels wrong, since it can be useful for some developer in the future.

But letting it live among the regular code of the project seems odd to. Not only am I not sure where exactly in the code base it should live, but in what form? An own class with a main method that does this? Or in a method that has no code pointing to it? Or commented out?

Or it could live in some Wiki, I guess. But unless the people involved in the project uses the wiki often, code snippets there are likely to be forgotten about the next time some bulk job needs to be done. I have seen various documentation projects shrivel up and die after a few months because people don't work in them regularly and then tend to forget they even exist.

How do you guys handle this? In my case, the code usually ended up in a text file on my own computer, since it has involving project with me as the main developer and the guy who did all these kind of tasks. But it doesn't seem very future proof.

https://redd.it/1ve9bqv
@r_devops

Need a DevOps Roadmap for a Complete Beginner (2026)

Hi everyone,

I'm a complete beginner and I want to learn DevOps from scratch in 2026.

My goal is to become job-ready as a DevOps Engineer, not just collect certificates. I'm willing to spend around 4–6 hours a day learning.

I have basic knowledge of:

Linux (very basic)
Networking (basic)
Python (basics)
Git (beginner)

I need guidance on:

What should I learn first?
Which tools should I focus on? (Linux, Git, Docker, Kubernetes, Jenkins, Terraform, AWS, Ansible, etc.)
Which free YouTube channels or courses do you genuinely recommend?
What projects should I build to make my resume stand out?
What mistakes do beginners usually make that I should avoid?
If you were starting from zero today, what roadmap would you follow?

I'm looking for practical advice from people who are already working in DevOps.

Thanks in advance!

https://redd.it/1ve9pnr
@r_devops

DevOps not a real job.

Good morning everyone, to be honest I was on vacation for a couple of weeks and all the time the thought that the career I had for the last decade has been not real hit me hard. The news hit me during my last post on this subreddit when I posted about DevOps always being vague.

The post had a lot of positive replies and a lot of things that opened my eyes and made me think but also something that caused a complete identity crisis and it was this "DevOps is culture and not a role."

I have mixed feelings about this because maybe the term was coined as a set of best practices and culture rather than a role and maybe it is all the things that devs don't want to do dumped into a role.

Honestly I've worked in startups and I've met very little devs that have enough competence to be DevOps and I've worked at hyper growth companies at scale and trust me if you don't have a team or at least one person for each DevOps position, SRE, DevEx, Infra the company might as well just shut down

https://redd.it/1vef6rh
@r_devops

from your cellphone to server how network works

When I type "google.com", first my browser checks its DNS cache, then the OS cache, and if it still doesn't find the IP, it queries the configured DNS server. Once it gets the IP address, it reaches the Load Balancer, performs the TCP three-way handshake, and then establishes a TLS connection.

Once the secure connection is established, the browser sends an encrypted HTTP request to the Load Balancer. The Load Balancer forwards the request to a suitable backend server based on its routing rules.

Now let's talk about what happens within the server. The request is processed, the service does whatever work is required, maybe querying a database, calling another service, or executing some business logic, and then sends the response back.

But how did the browser send the request in the first place?

Your home router uses DHCP to assign your device a private IP address from its pool. Your ISP provides your router with a public IP address. When your browser opens a connection, the operating system assigns it a random ephemeral source port. The router then performs NAT, more specifically PAT, by mapping your device's private IP and source port to its public IP and another source port. It keeps this mapping in its NAT table so that when the response comes back from the server, it knows exactly which device and which browser connection the data belongs to.

Once the response reaches your browser, the browser decrypts the HTTPS data using the established TLS session and renders the webpage. HTTPS works over TLS, so both the request and the response are encrypted while in transit. When the connection is no longer needed, the TCP connection is closed, which also ends the TLS session, although modern browsers often keep the connection alive for a while so they can reuse it for subsequent requests.

https://redd.it/1veoivy
@r_devops

Book recommendations for on premises deployment

So I am new to the DevOps field and my job is to deploy an app on premises. Now the problem is that the material I have read, such as:

"Fundamentals of DevOps and Software Delivery" by Yevgeniy Brikman

"Road to Kubernetes" by Justin Mitchel

focuses mainly on cloud based deployment, which I do not want to use. To clarify, I already have the necessary servers, but for privacy reasons, I would prefer not to disclose too much about the setup. Does anyone have a beginner friendly book recommendation that explains how to deploy an application on premises?

I also need to use Kubernetes and Docker, and I already have learning material for both, so the recommended books do not need to explain them in great depth. Thank you for you time!

https://redd.it/1veqes4
@r_devops

If I cannot start a devops job as a fresher, what can I start to learn to get job?

Hey all,

I read and heard that devops is not a starter job that I can learn number of skills and start job searching. There is no junior or entry level devops jobs.

I can understand it. If it is true, what can I learn and search for jobs? Shall I go for AWS? Linux? The only one skill will get me a job?

I am sorry I am very new to IT. I want to change my career.

https://redd.it/1veo5ve
@r_devops

Genuinely asking: is "move fast and break things" dead, or did it just get rebranded?

Every team I talk to these days will tell you, unprompted, that they've moved past that mentality. More deliberate now, more careful, more "quality-first." Nobody wants to be associated with the phrase anymore, it's become a bit of a punchline.

But when I look at what actually happens day to day, ship first, find out what broke from a user report or a support ticket, patch it after, it looks pretty similar to what it always looked like. The vocabulary changed. The retros got more polished. There's a Slack channel for "learnings" now instead of just quietly fixing it and moving on. The underlying loop, though, ship, discover, patch, doesn't look that different from five years ago on most teams I've seen.

Maybe that's fine. Maybe the loop was never really the problem, and the actual improvement is somewhere else, better rollback speed, better observability so the discover step is faster, smaller blast radius when something does break. That would be real progress even if the core behavior looks the same from a distance.

Or maybe I'm just seeing survivorship bias, the teams that talk publicly about being careful and deliberate are a specific, vocal subset, and they're not representative of what's actually happening at most places.

Genuinely asking, not trying to make a point here: what's concretely different about how your team ships now versus a few years ago? Not how you'd describe the philosophy, what's actually different in practice, deploy frequency, review process, what happens in the fifteen minutes after something goes wrong.

https://redd.it/1ves8iv
@r_devops

Interview tips and suggestions for platform engineer

hey everyone,

I have an upcoming interview for a Platform Engineer role. I’ve been grinding scenarios around IaC (Terraform/Terragrunt), CI/CD pipelines, Kubernetes, and SRE concepts (SLIs/SLOs, Error Budgets) .

i need tips and guidance from senior engineer . it will be a quick chat

https://redd.it/1vf5auy
@r_devops

I keep seeing the same issues on WAF configs in my audits

Been doing security audits for a while (I’m a secops engineer) and the same patterns keep showing up. Regardless of the vendor, so these are the same whether it’s Akamai, Cloudflare, AWS WAF.

1. Origin is directly reachable! Traffic bypasses the CDN/WAF (check cert transparency logs for leaked origin hostnames)

2. They've got bypass rules that were meant to be temporary but never got removed

3. Cache key too broad (cache poisoning risk) or too narrow (kills hit ratio, looks like a DDoS)

4. WAF rule sets are treated as “once and done”. Rules deployed once and never tuned, so there are a lot of silent false positives on real traffic.

5. TLS/cert management with no clear owner, resulting in expired certs, weak ciphers left on etc.



Wrote up a longer breakdown with a checklist if anyone wants to run through their own setup, happy to share.



https://redd.it/1vf70m0
@r_devops

found out my LLM feature's cost problem from an invoice, not a dashboard, and that felt very wrong

shipped an LLM feature, felt fine at launch, moved on to other work. a while later the bill showed up noticeably higher than expected and that was the first real signal anything had changed. no alert, no dashboard flag, just a number at the end of the month that made me go "wait, what happened."

went back and actually set up proper tracing and token/cost monitoring per request instead of just trusting it'd be fine, and found a specific workflow was making way more calls than i thought due to a retry loop that wasn't being logged anywhere visible. it had been quietly running up cost for weeks with zero visibility until the invoice.

also added latency budgets and caching for repeated queries after this, which cut cost noticeably on top of the retry fix.

feels like a pretty basic devops instinct (you monitor what you ship, you don't wait for the bill) that somehow gets skipped constantly once "AI feature" is involved, like people ship LLM stuff with less observability discipline than they'd ever accept for a normal service. anyone else's team caught something similar the hard way before actually building proper monitoring in from the start

https://redd.it/1vfa6tu
@r_devops