If we already have Grafana, Datadog, Splunk, PagerDuty, etc., why are production incidents still so hard?

I've been trying to understand what actually happens during production incidents, and something doesn't add up in my head.

Let's say I'm on call and I get paged at 2 AM because "checkout is broken."

We already have tools like Grafana, Datadog, Splunk, OpenTelemetry, PagerDuty, Kubernetes dashboards, cloud monitoring, CI/CD history, Git, runbooks... basically a ton of observability and operational tools.

So where does the time actually go?

Is it because the information is spread across too many places? Is it understanding what changed? Figuring out the blast radius? Knowing which team or dependency is actually responsible? Or is there something I'm completely missing?

Reading incident postmortems, it feels like engineers already have lots of data, but they still spend a long time piecing together what actually happened before they can confidently act.

For those of you who've been on call, what makes that first 15–30 minutes difficult, even with all these tools available?

I'm genuinely trying to understand what I'm missing as a student because from the outside it feels like we already have a tool for everything, yet companies still spend hours resolving some incidents.

https://redd.it/1ve6uem
@r_devops

Where do you store code for one off tasks that might be useful later on?

I've been a developer for a few decades now, almost always in a DevOps kind of role, and every once in a while I have stumbled over this issue and never found a solution that sits well with me.

As a DevOps guy, I both write and maintain the code, as well as being partly responsible for the operations and the data of the system. In our case the core of the system is a CMS, but it could be a DMS, CRM, IAM etc etc.

And on occation we have a need to do some bulk operation on the data, in a way that has not been done before, and can't be done easily using some UI. If there will be a repeated need for that kind of bulk job then we will likely integrate it fully into the code, and have a user friendly interface for it.

But what about more one off kind of tasks? Tasks that require coding, but that likely won't be needed again in the foreseeable future.

The last time it happened to me, this was roughly the task at hand:

* Identify all objects in the system with properties X, Y and (Z1 or Z2) within section S1.
* Move them all from section S1 to section S2.
* For each object, update property X to a calculated new value.

It is possible to do this manually in the regular GUI for the CMS. But that's not feasable when there are several hundreds of objects matching the criteria. Especially not when it's fairly trivial to write code that does the bulk job.

But after the job has been run (first in dev/test/stage, and then in production), what should happen with that code?

Just deleting it feels wrong, since it can be useful for some developer in the future.

But letting it live among the regular code of the project seems odd to. Not only am I not sure where exactly in the code base it should live, but in what form? An own class with a main method that does this? Or in a method that has no code pointing to it? Or commented out?

Or it could live in some Wiki, I guess. But unless the people involved in the project uses the wiki often, code snippets there are likely to be forgotten about the next time some bulk job needs to be done. I have seen various documentation projects shrivel up and die after a few months because people don't work in them regularly and then tend to forget they even exist.

How do you guys handle this? In my case, the code usually ended up in a text file on my own computer, since it has involving project with me as the main developer and the guy who did all these kind of tasks. But it doesn't seem very future proof.

https://redd.it/1ve9bqv
@r_devops

Need a DevOps Roadmap for a Complete Beginner (2026)

Hi everyone,

I'm a complete beginner and I want to learn DevOps from scratch in 2026.

My goal is to become job-ready as a DevOps Engineer, not just collect certificates. I'm willing to spend around 4–6 hours a day learning.

I have basic knowledge of:

Linux (very basic)
Networking (basic)
Python (basics)
Git (beginner)

I need guidance on:

What should I learn first?
Which tools should I focus on? (Linux, Git, Docker, Kubernetes, Jenkins, Terraform, AWS, Ansible, etc.)
Which free YouTube channels or courses do you genuinely recommend?
What projects should I build to make my resume stand out?
What mistakes do beginners usually make that I should avoid?
If you were starting from zero today, what roadmap would you follow?

I'm looking for practical advice from people who are already working in DevOps.

Thanks in advance!

https://redd.it/1ve9pnr
@r_devops

DevOps not a real job.

Good morning everyone, to be honest I was on vacation for a couple of weeks and all the time the thought that the career I had for the last decade has been not real hit me hard. The news hit me during my last post on this subreddit when I posted about DevOps always being vague.

The post had a lot of positive replies and a lot of things that opened my eyes and made me think but also something that caused a complete identity crisis and it was this "DevOps is culture and not a role."

I have mixed feelings about this because maybe the term was coined as a set of best practices and culture rather than a role and maybe it is all the things that devs don't want to do dumped into a role.

Honestly I've worked in startups and I've met very little devs that have enough competence to be DevOps and I've worked at hyper growth companies at scale and trust me if you don't have a team or at least one person for each DevOps position, SRE, DevEx, Infra the company might as well just shut down

https://redd.it/1vef6rh
@r_devops

from your cellphone to server how network works

When I type "google.com", first my browser checks its DNS cache, then the OS cache, and if it still doesn't find the IP, it queries the configured DNS server. Once it gets the IP address, it reaches the Load Balancer, performs the TCP three-way handshake, and then establishes a TLS connection.

Once the secure connection is established, the browser sends an encrypted HTTP request to the Load Balancer. The Load Balancer forwards the request to a suitable backend server based on its routing rules.

Now let's talk about what happens within the server. The request is processed, the service does whatever work is required, maybe querying a database, calling another service, or executing some business logic, and then sends the response back.

But how did the browser send the request in the first place?

Your home router uses DHCP to assign your device a private IP address from its pool. Your ISP provides your router with a public IP address. When your browser opens a connection, the operating system assigns it a random ephemeral source port. The router then performs NAT, more specifically PAT, by mapping your device's private IP and source port to its public IP and another source port. It keeps this mapping in its NAT table so that when the response comes back from the server, it knows exactly which device and which browser connection the data belongs to.

Once the response reaches your browser, the browser decrypts the HTTPS data using the established TLS session and renders the webpage. HTTPS works over TLS, so both the request and the response are encrypted while in transit. When the connection is no longer needed, the TCP connection is closed, which also ends the TLS session, although modern browsers often keep the connection alive for a while so they can reuse it for subsequent requests.

https://redd.it/1veoivy
@r_devops

Book recommendations for on premises deployment

So I am new to the DevOps field and my job is to deploy an app on premises. Now the problem is that the material I have read, such as:

"Fundamentals of DevOps and Software Delivery" by Yevgeniy Brikman

"Road to Kubernetes" by Justin Mitchel

focuses mainly on cloud based deployment, which I do not want to use. To clarify, I already have the necessary servers, but for privacy reasons, I would prefer not to disclose too much about the setup. Does anyone have a beginner friendly book recommendation that explains how to deploy an application on premises?

I also need to use Kubernetes and Docker, and I already have learning material for both, so the recommended books do not need to explain them in great depth. Thank you for you time!

https://redd.it/1veqes4
@r_devops

If I cannot start a devops job as a fresher, what can I start to learn to get job?

Hey all,

I read and heard that devops is not a starter job that I can learn number of skills and start job searching. There is no junior or entry level devops jobs.

I can understand it. If it is true, what can I learn and search for jobs? Shall I go for AWS? Linux? The only one skill will get me a job?

I am sorry I am very new to IT. I want to change my career.

https://redd.it/1veo5ve
@r_devops

Genuinely asking: is "move fast and break things" dead, or did it just get rebranded?

Every team I talk to these days will tell you, unprompted, that they've moved past that mentality. More deliberate now, more careful, more "quality-first." Nobody wants to be associated with the phrase anymore, it's become a bit of a punchline.

But when I look at what actually happens day to day, ship first, find out what broke from a user report or a support ticket, patch it after, it looks pretty similar to what it always looked like. The vocabulary changed. The retros got more polished. There's a Slack channel for "learnings" now instead of just quietly fixing it and moving on. The underlying loop, though, ship, discover, patch, doesn't look that different from five years ago on most teams I've seen.

Maybe that's fine. Maybe the loop was never really the problem, and the actual improvement is somewhere else, better rollback speed, better observability so the discover step is faster, smaller blast radius when something does break. That would be real progress even if the core behavior looks the same from a distance.

Or maybe I'm just seeing survivorship bias, the teams that talk publicly about being careful and deliberate are a specific, vocal subset, and they're not representative of what's actually happening at most places.

Genuinely asking, not trying to make a point here: what's concretely different about how your team ships now versus a few years ago? Not how you'd describe the philosophy, what's actually different in practice, deploy frequency, review process, what happens in the fifteen minutes after something goes wrong.

https://redd.it/1ves8iv
@r_devops

Interview tips and suggestions for platform engineer

hey everyone,

I have an upcoming interview for a Platform Engineer role. I’ve been grinding scenarios around IaC (Terraform/Terragrunt), CI/CD pipelines, Kubernetes, and SRE concepts (SLIs/SLOs, Error Budgets) .

i need tips and guidance from senior engineer . it will be a quick chat

https://redd.it/1vf5auy
@r_devops

I keep seeing the same issues on WAF configs in my audits

Been doing security audits for a while (I’m a secops engineer) and the same patterns keep showing up. Regardless of the vendor, so these are the same whether it’s Akamai, Cloudflare, AWS WAF.

1. Origin is directly reachable! Traffic bypasses the CDN/WAF (check cert transparency logs for leaked origin hostnames)

2. They've got bypass rules that were meant to be temporary but never got removed

3. Cache key too broad (cache poisoning risk) or too narrow (kills hit ratio, looks like a DDoS)

4. WAF rule sets are treated as “once and done”. Rules deployed once and never tuned, so there are a lot of silent false positives on real traffic.

5. TLS/cert management with no clear owner, resulting in expired certs, weak ciphers left on etc.



Wrote up a longer breakdown with a checklist if anyone wants to run through their own setup, happy to share.



https://redd.it/1vf70m0
@r_devops

found out my LLM feature's cost problem from an invoice, not a dashboard, and that felt very wrong

shipped an LLM feature, felt fine at launch, moved on to other work. a while later the bill showed up noticeably higher than expected and that was the first real signal anything had changed. no alert, no dashboard flag, just a number at the end of the month that made me go "wait, what happened."

went back and actually set up proper tracing and token/cost monitoring per request instead of just trusting it'd be fine, and found a specific workflow was making way more calls than i thought due to a retry loop that wasn't being logged anywhere visible. it had been quietly running up cost for weeks with zero visibility until the invoice.

also added latency budgets and caching for repeated queries after this, which cut cost noticeably on top of the retry fix.

feels like a pretty basic devops instinct (you monitor what you ship, you don't wait for the bill) that somehow gets skipped constantly once "AI feature" is involved, like people ship LLM stuff with less observability discipline than they'd ever accept for a normal service. anyone else's team caught something similar the hard way before actually building proper monitoring in from the start

https://redd.it/1vfa6tu
@r_devops

Question about trader status under the EU Digital Services Act

Hello,
I’m trying to understand whether I should classify myself as a trader under the EU Digital Services Act, and I would really appreciate input from developers who have dealt with this. I’ve received conflicting answers from AI tools, so I’m not confident relying on those alone.
Apple requires developers to choose one of the following:

“I’m a trader under the DSA”

“I’m not a trader under the DSA or I don’t plan to distribute in the EU”

I haven’t published any apps yet, but I’m preparing to release my first independently developed apps on the App Store and Google Play.
App development is currently a hobby alongside my main job, and I’m not operating through a registered company.
The apps will be free to download but include ads. One app will also offer a paid option to remove ads, and I may add in-app purchases in the future.
I’ve noticed that some similar apps with ads and in-app purchases are published by developers who don’t appear to be marked as traders, though I understand that may not reflect their actual status.
Given this, would you consider someone in my situation to be a trader under the DSA, or could it reasonably be treated as non-trader activity since it’s currently a hobby?
I also have a few related questions:
Does the same approach apply when selecting trader status on Google Play?
What typically happens if you declare yourself a non-trader but the platform later disagrees? Are you usually allowed to update your status, or could your account be affected?
Is it true that apps from non-trader accounts may be removed in the EU if the classification is later found to be incorrect?
Is trader status set per app or per developer account on Apple and Google?
I understand this isn’t legal advice. I’m mainly interested in how other independent developers have interpreted and handled this in practice.
Thank you very much for your help.

https://redd.it/1vfbzdw
@r_devops

Question about trader status under the EU Digital Services Act

Hello,
I’m trying to understand whether I should classify myself as a trader under the EU Digital Services Act, and I would really appreciate input from developers who have dealt with this. I’ve received conflicting answers from AI tools, so I’m not confident relying on those alone.
Apple requires developers to choose one of the following:
“I’m a trader under the DSA”
“I’m not a trader under the DSA or I don’t plan to distribute in the EU”
I haven’t published any apps yet, but I’m preparing to release my first independently developed apps on the App Store and Google Play.
App development is currently a hobby alongside my main job, and I’m not operating through a registered company.
The apps will be free to download but include ads. One app will also offer a paid option to remove ads, and I may add in-app purchases in the future.
I’ve noticed that some similar apps with ads and in-app purchases are published by developers who don’t appear to be marked as traders, though I understand that may not reflect their actual status.
Given this, would you consider someone in my situation to be a trader under the DSA, or could it reasonably be treated as non-trader activity since it’s currently a hobby?
I also have a few related questions:
Does the same approach apply when selecting trader status on Google Play?
What typically happens if you declare yourself a non-trader but the platform later disagrees? Are you usually allowed to update your status, or could your account be affected?
Is it true that apps from non-trader accounts may be removed in the EU if the classification is later found to be incorrect?
Is trader status set per app or per developer account on Apple and Google?
I understand this isn’t legal advice. I’m mainly interested in how other independent developers have interpreted and handled this in practice.
Thank you very much for your help.

https://redd.it/1vfc04y
@r_devops

What's the most time you've lost to a bug that wasn't actually a bug?

I spent way too long trying to fix something recently before realizing nothing was actually wrong.

It turned out to be a configuration mistake that I had made myself.

It got me wondering how much time people in ML and software spend debugging their own setups rather than actual code.

What's the dumbest "bug" you've ever chased?

https://redd.it/1vf8x4l
@r_devops

Does anyone have information on hardening publically exposed k8s LB nodes? Loadbalancers in AWS are really expensive.

Just exploring the space and price comparisons

ALBs and NLBs are expensive. especially in AW

So some ingress only nodes with proper hardening seems cheaper, but curious about battle scars and stories people wanna share.

My idea is simply that, hardening a fleet of ingress only nodes in a public subnet, pointing all queries at these nodes and letting the proxy and ingress rules handle the traffic from there

https://redd.it/1vfi0gg
@r_devops

How much attention is harness engineering getting?

AI model quality is converging, or at least changing often enough that chasing the newest thing doesn't seem like a good strategy. How much emphasis are teams putting on building a solid harness into which new models can fit?

Our clients are generally in high compliance industries, so there is thought put into the harness, but what about smaller teams or ones that aren't required into a compliance framework?

https://redd.it/1vfkoi4
@r_devops

Hetzner Cloud vs Turkish VPS for a production mobile backend?

Hi everyone,



I'm building a mobile application primarily for users in Turkey.



My stack:

\- NestJS

\- PostgreSQL

\- Redis

\- Docker

\- Coolify



The app is still in development and I'm planning the production infrastructure.



I'm considering two options:



1. Hetzner Cloud (Germany)

2. A VPS/Dedicated server located in Turkey



The backend is a standard REST API (no gaming or real-time voice/video). Most requests are authentication, CRUD operations, QR code lookups, notifications, and media metadata.



Latency from Turkey to Germany seems to be around 25-40 ms.



Would you prioritize:

\- Better hardware and infrastructure (Hetzner)

\- Lower latency with a Turkish provider



Has anyone run production applications for Turkish users from Hetzner? I'd love to hear real-world experiences.

https://redd.it/1vfkjvd
@r_devops

KodeKloud Challenges are worth it?

I am thinking on start the 100 days of DevOps challenge in KodeKloud just to improve my devops skills and fill in some gaps that I might have.

Does this challenge worth it? Specially for the time that needs to be invested.

https://redd.it/1vfqoqe
@r_devops

Self-hosted enterprise app builders with strong RBAC and compliance features

We’ve been evaluating self-hosted low-code and internal application builder platforms mainly for compliance-heavy requirements like fine-grained RBAC, audit logging, and enterprise access control, and while tools like Retool and ToolJet are still commonly used in this space, they now feel more oriented toward managed or hybrid deployments rather than purely self-hosted governance-first setups, and Appian is still an option but feels closer to a full enterprise BPM suite than a lightweight internal tool builder, so we’re now looking for more modern self-hosted alternatives that can be deployed via Docker or Kubernetes, support strong role-based permissioning, integrate with SSO/SAML or LDAP, provide detailed audit trails, and avoid heavy vendor lock-in since we’re planning to migrate around 300 applications, so it would be useful to hear what stacks are actually working reliably in production environments today.

https://redd.it/1vfxku2
@r_devops

Are you team A or team B?
https://redd.it/1vg0i1u
@r_devops