Server monitoring (cpu/ram/disk) recommendations please

Hello everyone, so I am currently using NewRelic for half of my setup and grafana (self-hosted) + prometheus for the other half, however I'm looking to move out of grafana and probably use a cloud service, however NR is kinda expensive for my needs; I need to monitor CPU usage, RAM and DISK + any syslog error (as extra), any recommendations/ideas?

https://redd.it/1uwoo76
@r_devops

How do I approach dev ops problems with lack of experience?

I'm a college student with a very narrow knowledge of C/C++, data structures/computer architecture and the more theoretical side of Computer Science. I'm interning at a small sized company this summer and my software engineering role has turned into more of a dev ops role. I may enjoy it, but it's been frustrating to be dropped in a world I (and actually my bosses don't have much experience either) where nothing is familiar. Are there any recommendations for a crash course about development pipelines/ infra that is recommended? My dm's are also open if I could talk through my struggles with someone experienced.

https://redd.it/1uwugip
@r_devops

What enterprise backup solution are you using for Azure VMs and Blob Storage?

We're taking another look at our Azure backup strategy and are curious how other teams are handling it

Azure Backup does a good job for many use cases, but we're also thinking about scenarios where we'd want copies of our data outside the same Azure environment. That could mean another tenant, another cloud, or a dedicated backup platform to improve recovery options.

Our environment is primarily Azure VMs, Azure Blob Storage, Azure SQ databases, and a few terabytes of data overall.

For those running production workloads, what approach has worked best for you?

Did you stick with Azure-native services, maintain backups in a separate tenant, use a third-party platform, or take a different approach altogether?

Looking back, were there any restore or disaster recovery lessons that changed how you designed your backup strategy?

https://redd.it/1uwx5nc
@r_devops

People who bought a cloud cost optimization tool: did the savings number they promised actually show up on your bill?

Disclosure up front: I'm a founder of a startup in this exact space, so read everything below with that bias in mind. Not naming or pitching the product, that's not what this is.

Here's why I'm asking. Before a deal, every vendor in this category (mine included) shows you a big number. "We found 40% waste." And I've started noticing the games that can hide inside that number. Identified savings quietly presented as if they were realized savings. Savings from work your own team did getting counted in the tool's total. One-time cleanups counted every month forever. Baselines cherry-picked from your most wasteful week.

So I'm curious about the other side of the table. For those of you who actually bought one of these tools:

Did the promised number materialize on the actual invoice? How did you verify it, or could you even? Did anyone here catch a vendor inflating? And for those where it worked out, what did the vendor do differently that made you trust the math?

Trying to figure out what proof would actually convince a skeptical platform team, because "trust our dashboard" clearly isn't it.

https://redd.it/1uwyi5o
@r_devops

Google Maps for Clusters
https://redd.it/1ux1leq
@r_devops

I'm tired of bolting RBAC and Audit logs onto our custom ops scripts. How do you safely execute these?

We have the standard graveyard of custom bash, Python and even Groovy scripts for atomic ops tasks (draining nodes, killing stuck pods, db failovers, etc.).

While standard "community playbooks" are useful for the boilerplate stuff, we inevitably end up needing to run our own custom logic out of our own repos too.

But the actual friction is the execution boundary. Trying to enforce "who is allowed to run this script in prod at 3am" (RBAC), requiring an approval before execution, and getting a clean, immutable audit log of the outputs is driving me crazy. Handing devs raw kubeconfig/SSH access is a minefield.

Because you basically have to own the runtime to actually enforce this governance, how did you solve it? Did you write a custom internal CLI/runner that handles the auth + audit and then invokes the script? Or did you end up wiring something heavy (like AAP or Temporal) purely to get that secure execution boundary?

https://redd.it/1ux21mx
@r_devops

The compliance push finally made me look at self-hosting LLMs seriously

Been self hosting most of my stack for years. LLMs were always the one thing i kept on a closed api, not because i liked it, just because every time i looked at the open alternatives they were noticeably worse. privacy is a nice idea until you are explaining to a paying user why the output got dumber.

What actually forced my hand was a client deal last year. their legal team wanted to know exactly where AI-processed data physically sat, which country, which server. we had the dpa, zero retention agreements, all of it. did not matter. they kept coming back with more questions and the whole thing dragged for weeks, and at some point i had to sit with the fact that my closed api was the one thing breaking an otherwise clean self-hosted setup. that was the moment i actually got serious about finding something else.

Saw something about glm-5.2 being open weight and apparently landing close to opus on coding benchmarks. have not tested that myself, maybe someone here has. if it is even close to true then the quality excuse i have been leaning on for two years might not hold anymore. that was always the real reason, not the ops work. infra i can figure out.

Still have not done anything yet. model is massive and i am genuinely unsure what the hardware requirement looks like in practice. also thinking about prompt injection, if users can feed it arbitrary input that is a real surface area to worry about and i have not thought through all of it.

But this is the first time a self hosted option has not felt like a step down going in. that feeling is new and i am not totally sure what to do with it yet.

https://redd.it/1ux3tlq
@r_devops

Our CI/CD secrets are scattered across GitHub, Jenkins, .env files (and a few more). How can I get to runtime injection (relatively) peacefully?

We spent the last year shipping as fast as we could and the bill came due on secrets management. Right now they're everywhere: hardcoded in a few GitHub repos, sitting in .env files, baked into Jenkins credentials, and on at least three devs' laptops that I know of. It works until it doesn't, and I'd rather fix it before it becomes an incident instead of after.

The goal is runtime injection so nothing sensitive lives in the repo or the CI config at all, but I don't have six months to stand up a whole platform. I'm trying to find the pragmatic middle path between "keep living like this" and "boil the ocean".

A few things I'm weighing: IT already runs Passwork for human credentials and it has an API and CLI, so one option is just consolidating machine secrets there too rather than introducing yet another system. The other direction is a dedicated secrets store built for the pipeline. Underneath all of this is the identity question of do I go OIDC federation so the runner authenticates without a long-lived token, or accept a bootstrap secret somewhere and just minimize the blast radius?

https://redd.it/1ux5x2v
@r_devops

Managing DB credentials for k8s services

Hey all,

Trying to figure how people actually manage DB credentials for apps at scale.

Our current setup works, but kinda fragile:

1. Liquibase runs DDLs using shared creds pulled from Parameter Store.
2. A custom Jenkins shared lib provisions dedicated per app creds at the SQL level and drops them into Secrets Manager. Apps pull from there and connect.

The pain - no visibility into what uses what and it's forward only, nothing cleans up when service is decommissioned, stale SQL users and secrets everywhere.

We're fully on AWS, so RDS + EKS and some Redshift and DocumentDB.

Where I've landed so far and where I'd love a sanity check:

* Vault (or OpenBao) for credentials lifecycle
* A separate git repo owning the durable roles (one for DDL, one for app access) plus the Vault config, so grants live in one reviewed place instead of scattered across app repos. DDLs for apps would still live in their respective repos managed via Liquibase.
* Terraform postgres/mysql providers for the grants, not sure about Redshift or DocumentsDB, afaik there is no official provider for either.

Never ran Vault before - how hard is the initial lift realistically?

How to handle redshift and mongo grants declaratively?

I've considered IAM auth before, forgot why we gave up, should I re-visit?

Vault vs OpenBao vs something else?

I guess there is no golden solution, but want to hear what's actually held up in production.

Thanks.

https://redd.it/1ux8asg
@r_devops

OpenTelemetry Agent Skills
https://ollygarden.com/blog/teach-your-coding-agent-opentelemetry

https://redd.it/1uxe03k
@r_devops

Our biggest deployment scare had nothing to do with the deployment itself

A while back we were preparing for what should have been a routine release. The code had been reviewed CI/CD looked good monitoring was ready rollback plans were in place everything you'd expect. Someone on our infrastructure team had been testing Kibu for another project around the same time which led to a conversation about release approvals. It made us realize the closest we came to a problem wasn't technical at all it was confusion over who had actually given the final go ahead. Nothing bad happened in the end but it was a reminder that even well automated deployments still depend on people communicating clearly. Has anyone else had a release where the biggest risk turned out to be process rather than code?

https://redd.it/1uxgl1m
@r_devops

What's the most 'temporary' thing in your stack that's now load-bearing in prod?

Every place I've worked has had at least one. Mine right now is a \~40-line bash script someone wrote 'just for the migration weekend' about three years ago. It's still the only thing that reconciles two systems that were supposed to be fully merged by that Q2. Nobody wants to own it, everyone's a little afraid to touch it, and it has exactly zero tests.

I'm curious what everyone else is quietly sitting on: the cron job with no owner, the one instance nobody can confidently identify, the 'staging' service that's actually taking prod traffic, the manual runbook step that's really the whole system.

And the part I actually want to learn from: did you ever successfully retire one of these, or do they just accumulate? If you killed one, what finally made it possible - a rewrite, an outage, a new hire with no fear, or just budget to do it properly?

https://redd.it/1uxha98
@r_devops

How would you investigate random production downtime when there are almost no useful logs?

Hi everyone,

I'm looking for advice from people who have experience troubleshooting production systems. I'm less interested in the exact fix and more interested in how you would investigate a problem like this.

Environment

\- Windows Server + IIS

\- ASP.NET Core MVC + Web APIs

\- Angular frontend

\- SQL Server Web Edition on a dedicated server (8 GB RAM)

\- Elasticsearch cluster (3 nodes) on separate servers

\- Separate monitoring/tools server

\- Around 8 million products in Elasticsearch

\- Traffic goes directly to IIS (no reverse proxy, CDN, WAF, or load balancer). We also don't control the domain.

The problem

Several times a day, the website becomes unavailable for about 1–2 minutes and then recovers by itself.

Both Pingdom and Uptime Kuma report:

«Socket timeout, unable to connect to server»

Example:

2026-07-09 12:06:43

Socket timeout, unable to connect to server

Confirmed from San Jose and Frankfurt

The issue is completely random. Sometimes it happens during busy hours, sometimes when traffic is low.

What we've already checked

\- DNS resolution is fast.

\- The hosting provider reports no network or infrastructure problems.

\- Windows stays online.

\- IIS logs don't show anything useful.

\- ASP.NET Core logs don't show failed requests.

\- SQL connection pool exhaustion was a problem in the past, but after introducing caching those alerts disappeared.

\- SQL now appears healthy, but the outages continue.

I also know the application has technical debt (blocking calls, synchronous code, etc.), but before changing the application I'd like to understand whether I'm looking at the right layer.

My current investigation plan

I'm planning to:

\- Deploy OpenTelemetry (not deployed yet)

\- Collect runtime metrics (ThreadPool, GC, active requests, request duration)

\- Enable distributed tracing

\- Investigate HTTPERR logs

\- Monitor HTTP.sys and IIS request queues

\- Add Windows Performance Counters to Grafana

\- Correlate Windows, IIS, SQL Server, Elasticsearch, and application metrics when the next outage happens

My questions

If you were the on-call engineer for this production environment:

\- What would be the first things you would monitor?

\- How would you narrow down whether the problem is in the network, Windows, HTTP.sys, IIS, ASP.NET Core, SQL Server, or Elasticsearch?

\- Which metrics or dashboards have helped you the most with intermittent outages like this?

\- Have you ever seen socket timeouts where the application and IIS logs contained almost no useful information?

\- What tools would you add before waiting for the next outage?

\- Is there anything obvious that I'm missing?

I'd love to hear how experienced DevOps/SRE engineers approach this kind of investigation. I'm trying to build a proper troubleshooting process instead of guessing every time an incident happens.

Thanks!

https://redd.it/1uxfdnd
@r_devops

How would you define this role?

I need help defining a role we're looking to hire for.

Our current job postings mention DevOps / Platform Engineer.

The issue is that this is attracting a lot of candidates whose primary experience is as a developer, with some basic cloud experience.

However, this isn't what we're looking for. We're looking for an engineer whose primary experience is in Ops / Sysadmin areas, but uses modern dev tools to manage it, such as IaC, Kubernetes, Ansible, etc.

We have a range of projects on our to do list, and there's some Ops / Sysadmin, cloud infra, SRE, and platform engineering. So we're really looking for someone who has some experience with all of those.

How would you define this role?

(Please don't DM me asking to apply for the position).

https://redd.it/1uxrmsy
@r_devops

Is a yes/no prompt enough before an agent deletes local project files?

For teams that are experimenting with coding agents on developer machines, how are you treating destructive local commands?

I do not mean normal test commands or read-only inspection. I mean things like deleting build folders, clearing caches, resetting generated files, or running a migration helper that changes local state.

I once stopped an agent before it cleaned a generated folder because I realized I had not checked whether the folder was fully reproducible.

With normal automation, I would want a rollback story: current git state, affected path, reason for the command, and maybe a snapshot before it runs. A plain yes/no approval prompt feels too easy to click through after the fifth prompt of the day.

If you were allowing this in a team, would you block by command pattern, require a clean git checkpoint, log every destructive action, or keep these agents read-only by default?

https://redd.it/1uxs2me
@r_devops

Has anyone here actually found consulting work, a remote DevOps job, or long-term clients through Reddit?

I'm curious about the community's real experiences.

Over the past few months, I've seen many posts offering remote work, consulting opportunities, startups looking for DevOps engineers, or requests to "DM me" for more details.

However, I've also noticed that many posters don't share a company name, website, GitHub organization, or LinkedIn profile before moving the conversation to private messages. In many cases, the discussion simply ends without any real opportunity.

For those who've been around this community:

Have you actually landed a real DevOps job, consulting engagement, or long-term client through Reddit?
Which subreddits or communities have worked best for you?
What are the signs that a post is legitimate versus someone just collecting contacts or exploring ideas?
Do you ask for company details before investing your time?

I'm genuinely trying to learn where Reddit fits into a professional job search and whether others have had success here.

I'd appreciate hearing your experiences good or bad.

https://redd.it/1uxwrw0
@r_devops

My CTO forced a microservices migration for our tiny user base and it's a dumpster fire.

I need to vent, and maybe get some perspective to see if I'm just being a bad engineer or if this is actually as crazy as it feels.

We have a relatively small product. A standard Python/Java backend, a Postgres database, serving maybe 10k daily active users. It was running perfectly fine. Fast, easy to debug, simple deployments.

Then our CTO went to some tech conference, came back, and decided we urgently need to be "enterprise scale" and "cloud-native." For the last four months, we've been tearing apart our perfectly good monolith.

It's been absolute hell. We now have 14 different microservices for a platform that barely needs one. The sheer amount of orchestration required just to get a simple user login working and keep these services talking to each other is completely eating up our sprint capacity. We are spending 80% of our time writing YAML files and debugging obscure networking issues instead of actually shipping features.

Yesterday it took me three hours to trace a failed transaction because a message got silently dropped somewhere between three different containers.

Is this normal? Is this just what modern distributed systems engineering is now, or is this pure resume-driven development from leadership? Honestly thinking about jumping ship. How would you guys handle this?

https://redd.it/1uxxfpv
@r_devops

Built a free, KMS-backed alternative to ACM Private CA for IAM Roles Anywhere

ACM Private CA is $400/month minimum before you've issued a single certificate. Roles Anywhere itself is free, but it needs a CA to trust, and that's basically the only paved-road option AWS gives you for one.


So I built my own. Two ways to run it:


A laptop-based version where the CA private key lives on your machine fine for messing around or a small POC, but you're trusting your laptop with the whole thing.


A KMS-backed version where the key never leaves AWS at all, issuance goes through a Lambda, and there's a public API endpoint (API key auth) so someone with zero AWS credentials can request their own certificate. Full audit trail in DynamoDB every cert issued, renewed, revoked, with timestamps and reasons.


Revocation is one call and it's actually enforced within seconds it publishes the CRL straight to Roles Anywhere in the same step, not just marked in a database somewhere and hoped for. There's also a reversible "disable" if you want to temporarily block someone without permanently killing their cert.


No external crypto dependencies anywhere the X.509/DER encoding is hand-rolled in plain Python, about 240 lines, so you can actually read the whole thing instead of trusting a library blindly.


Real cost, not a guess: ran it through the AWS Pricing Calculator for 2000 users and landed at about $1.25/month. https://calculator.aws/#/estimate?id=8bc0d34839e2c22287a2bc891ac321ee1cdeb114


There's already a well-automated AWS sample repo for this (sample-aws-iam-roles-anywhere-automation), but it deploys ACM Private CA under the hood, which is the exact cost this exists to avoid. If you're fine paying for that, it's a solid option. If the cost is what's stopping you, this gets you to the same place for a couple bucks a month.


GitHub: github.com/vireshsolanki/iam-roles-anywhere-automation


Curious if anyone else here has been running Roles Anywhere and what your CA situation looks like. It's open source and I'd rather it be actually useful for other people's setups than just mine, so if something doesn't work for your environment, open an issue and let me know I'll work on it.

https://redd.it/1uxy0k2
@r_devops

Can I run Falco on EKS Fargate?

I don't get it. I'm looking at whether I can run Falco on EKS Fargate, and I found [this page\](https://falco.org/blog/choosing-a-driver). It says, "yeah man if you wanna do that just use pdig". But if I click through to the pdig repo it says it was archived in 2022. I don't want to use unmaintained software, obvs. Does Falco just not support serverless workloads anymore? Why do they have this on their site if it's not supported anymore? Is there an alternative driver for Falco for serverless? Or should I be looking elsewhere for runtime monitoring?

https://redd.it/1uy4jpa
@r_devops

Prompt injection is a supply chain problem, not a model problem

Hot take from a banking DevSecOps team: treating prompt injection as something the model vendor should fix is a dead end. The real issue is that the context window has no provenance. The model cannot tell user instructions from a poisoned README or a tool response. Until that changes, the practical mitigations look a lot like classic supply chain controls: pin your dependencies, verify what you fetch, restrict what each component is allowed to do. Anyone mapping this to SLSA-style controls yet?

https://redd.it/1uy7l06
@r_devops