Managing DB credentials for k8s services

Hey all,

Trying to figure how people actually manage DB credentials for apps at scale.

Our current setup works, but kinda fragile:

1. Liquibase runs DDLs using shared creds pulled from Parameter Store.
2. A custom Jenkins shared lib provisions dedicated per app creds at the SQL level and drops them into Secrets Manager. Apps pull from there and connect.

The pain - no visibility into what uses what and it's forward only, nothing cleans up when service is decommissioned, stale SQL users and secrets everywhere.

We're fully on AWS, so RDS + EKS and some Redshift and DocumentDB.

Where I've landed so far and where I'd love a sanity check:

* Vault (or OpenBao) for credentials lifecycle
* A separate git repo owning the durable roles (one for DDL, one for app access) plus the Vault config, so grants live in one reviewed place instead of scattered across app repos. DDLs for apps would still live in their respective repos managed via Liquibase.
* Terraform postgres/mysql providers for the grants, not sure about Redshift or DocumentsDB, afaik there is no official provider for either.

Never ran Vault before - how hard is the initial lift realistically?

How to handle redshift and mongo grants declaratively?

I've considered IAM auth before, forgot why we gave up, should I re-visit?

Vault vs OpenBao vs something else?

I guess there is no golden solution, but want to hear what's actually held up in production.

Thanks.

https://redd.it/1ux8asg
@r_devops

OpenTelemetry Agent Skills
https://ollygarden.com/blog/teach-your-coding-agent-opentelemetry

https://redd.it/1uxe03k
@r_devops

Our biggest deployment scare had nothing to do with the deployment itself

A while back we were preparing for what should have been a routine release. The code had been reviewed CI/CD looked good monitoring was ready rollback plans were in place everything you'd expect. Someone on our infrastructure team had been testing Kibu for another project around the same time which led to a conversation about release approvals. It made us realize the closest we came to a problem wasn't technical at all it was confusion over who had actually given the final go ahead. Nothing bad happened in the end but it was a reminder that even well automated deployments still depend on people communicating clearly. Has anyone else had a release where the biggest risk turned out to be process rather than code?

https://redd.it/1uxgl1m
@r_devops

What's the most 'temporary' thing in your stack that's now load-bearing in prod?

Every place I've worked has had at least one. Mine right now is a \~40-line bash script someone wrote 'just for the migration weekend' about three years ago. It's still the only thing that reconciles two systems that were supposed to be fully merged by that Q2. Nobody wants to own it, everyone's a little afraid to touch it, and it has exactly zero tests.

I'm curious what everyone else is quietly sitting on: the cron job with no owner, the one instance nobody can confidently identify, the 'staging' service that's actually taking prod traffic, the manual runbook step that's really the whole system.

And the part I actually want to learn from: did you ever successfully retire one of these, or do they just accumulate? If you killed one, what finally made it possible - a rewrite, an outage, a new hire with no fear, or just budget to do it properly?

https://redd.it/1uxha98
@r_devops

How would you investigate random production downtime when there are almost no useful logs?

Hi everyone,

I'm looking for advice from people who have experience troubleshooting production systems. I'm less interested in the exact fix and more interested in how you would investigate a problem like this.

Environment

\- Windows Server + IIS

\- ASP.NET Core MVC + Web APIs

\- Angular frontend

\- SQL Server Web Edition on a dedicated server (8 GB RAM)

\- Elasticsearch cluster (3 nodes) on separate servers

\- Separate monitoring/tools server

\- Around 8 million products in Elasticsearch

\- Traffic goes directly to IIS (no reverse proxy, CDN, WAF, or load balancer). We also don't control the domain.

The problem

Several times a day, the website becomes unavailable for about 1–2 minutes and then recovers by itself.

Both Pingdom and Uptime Kuma report:

«Socket timeout, unable to connect to server»

Example:

2026-07-09 12:06:43

Socket timeout, unable to connect to server

Confirmed from San Jose and Frankfurt

The issue is completely random. Sometimes it happens during busy hours, sometimes when traffic is low.

What we've already checked

\- DNS resolution is fast.

\- The hosting provider reports no network or infrastructure problems.

\- Windows stays online.

\- IIS logs don't show anything useful.

\- ASP.NET Core logs don't show failed requests.

\- SQL connection pool exhaustion was a problem in the past, but after introducing caching those alerts disappeared.

\- SQL now appears healthy, but the outages continue.

I also know the application has technical debt (blocking calls, synchronous code, etc.), but before changing the application I'd like to understand whether I'm looking at the right layer.

My current investigation plan

I'm planning to:

\- Deploy OpenTelemetry (not deployed yet)

\- Collect runtime metrics (ThreadPool, GC, active requests, request duration)

\- Enable distributed tracing

\- Investigate HTTPERR logs

\- Monitor HTTP.sys and IIS request queues

\- Add Windows Performance Counters to Grafana

\- Correlate Windows, IIS, SQL Server, Elasticsearch, and application metrics when the next outage happens

My questions

If you were the on-call engineer for this production environment:

\- What would be the first things you would monitor?

\- How would you narrow down whether the problem is in the network, Windows, HTTP.sys, IIS, ASP.NET Core, SQL Server, or Elasticsearch?

\- Which metrics or dashboards have helped you the most with intermittent outages like this?

\- Have you ever seen socket timeouts where the application and IIS logs contained almost no useful information?

\- What tools would you add before waiting for the next outage?

\- Is there anything obvious that I'm missing?

I'd love to hear how experienced DevOps/SRE engineers approach this kind of investigation. I'm trying to build a proper troubleshooting process instead of guessing every time an incident happens.

Thanks!

https://redd.it/1uxfdnd
@r_devops

How would you define this role?

I need help defining a role we're looking to hire for.

Our current job postings mention DevOps / Platform Engineer.

The issue is that this is attracting a lot of candidates whose primary experience is as a developer, with some basic cloud experience.

However, this isn't what we're looking for. We're looking for an engineer whose primary experience is in Ops / Sysadmin areas, but uses modern dev tools to manage it, such as IaC, Kubernetes, Ansible, etc.

We have a range of projects on our to do list, and there's some Ops / Sysadmin, cloud infra, SRE, and platform engineering. So we're really looking for someone who has some experience with all of those.

How would you define this role?

(Please don't DM me asking to apply for the position).

https://redd.it/1uxrmsy
@r_devops

Is a yes/no prompt enough before an agent deletes local project files?

For teams that are experimenting with coding agents on developer machines, how are you treating destructive local commands?

I do not mean normal test commands or read-only inspection. I mean things like deleting build folders, clearing caches, resetting generated files, or running a migration helper that changes local state.

I once stopped an agent before it cleaned a generated folder because I realized I had not checked whether the folder was fully reproducible.

With normal automation, I would want a rollback story: current git state, affected path, reason for the command, and maybe a snapshot before it runs. A plain yes/no approval prompt feels too easy to click through after the fifth prompt of the day.

If you were allowing this in a team, would you block by command pattern, require a clean git checkpoint, log every destructive action, or keep these agents read-only by default?

https://redd.it/1uxs2me
@r_devops

Has anyone here actually found consulting work, a remote DevOps job, or long-term clients through Reddit?

I'm curious about the community's real experiences.

Over the past few months, I've seen many posts offering remote work, consulting opportunities, startups looking for DevOps engineers, or requests to "DM me" for more details.

However, I've also noticed that many posters don't share a company name, website, GitHub organization, or LinkedIn profile before moving the conversation to private messages. In many cases, the discussion simply ends without any real opportunity.

For those who've been around this community:

Have you actually landed a real DevOps job, consulting engagement, or long-term client through Reddit?
Which subreddits or communities have worked best for you?
What are the signs that a post is legitimate versus someone just collecting contacts or exploring ideas?
Do you ask for company details before investing your time?

I'm genuinely trying to learn where Reddit fits into a professional job search and whether others have had success here.

I'd appreciate hearing your experiences good or bad.

https://redd.it/1uxwrw0
@r_devops

My CTO forced a microservices migration for our tiny user base and it's a dumpster fire.

I need to vent, and maybe get some perspective to see if I'm just being a bad engineer or if this is actually as crazy as it feels.

We have a relatively small product. A standard Python/Java backend, a Postgres database, serving maybe 10k daily active users. It was running perfectly fine. Fast, easy to debug, simple deployments.

Then our CTO went to some tech conference, came back, and decided we urgently need to be "enterprise scale" and "cloud-native." For the last four months, we've been tearing apart our perfectly good monolith.

It's been absolute hell. We now have 14 different microservices for a platform that barely needs one. The sheer amount of orchestration required just to get a simple user login working and keep these services talking to each other is completely eating up our sprint capacity. We are spending 80% of our time writing YAML files and debugging obscure networking issues instead of actually shipping features.

Yesterday it took me three hours to trace a failed transaction because a message got silently dropped somewhere between three different containers.

Is this normal? Is this just what modern distributed systems engineering is now, or is this pure resume-driven development from leadership? Honestly thinking about jumping ship. How would you guys handle this?

https://redd.it/1uxxfpv
@r_devops

Built a free, KMS-backed alternative to ACM Private CA for IAM Roles Anywhere

ACM Private CA is $400/month minimum before you've issued a single certificate. Roles Anywhere itself is free, but it needs a CA to trust, and that's basically the only paved-road option AWS gives you for one.


So I built my own. Two ways to run it:


A laptop-based version where the CA private key lives on your machine fine for messing around or a small POC, but you're trusting your laptop with the whole thing.


A KMS-backed version where the key never leaves AWS at all, issuance goes through a Lambda, and there's a public API endpoint (API key auth) so someone with zero AWS credentials can request their own certificate. Full audit trail in DynamoDB every cert issued, renewed, revoked, with timestamps and reasons.


Revocation is one call and it's actually enforced within seconds it publishes the CRL straight to Roles Anywhere in the same step, not just marked in a database somewhere and hoped for. There's also a reversible "disable" if you want to temporarily block someone without permanently killing their cert.


No external crypto dependencies anywhere the X.509/DER encoding is hand-rolled in plain Python, about 240 lines, so you can actually read the whole thing instead of trusting a library blindly.


Real cost, not a guess: ran it through the AWS Pricing Calculator for 2000 users and landed at about $1.25/month. https://calculator.aws/#/estimate?id=8bc0d34839e2c22287a2bc891ac321ee1cdeb114


There's already a well-automated AWS sample repo for this (sample-aws-iam-roles-anywhere-automation), but it deploys ACM Private CA under the hood, which is the exact cost this exists to avoid. If you're fine paying for that, it's a solid option. If the cost is what's stopping you, this gets you to the same place for a couple bucks a month.


GitHub: github.com/vireshsolanki/iam-roles-anywhere-automation


Curious if anyone else here has been running Roles Anywhere and what your CA situation looks like. It's open source and I'd rather it be actually useful for other people's setups than just mine, so if something doesn't work for your environment, open an issue and let me know I'll work on it.

https://redd.it/1uxy0k2
@r_devops

Can I run Falco on EKS Fargate?

I don't get it. I'm looking at whether I can run Falco on EKS Fargate, and I found [this page\](https://falco.org/blog/choosing-a-driver). It says, "yeah man if you wanna do that just use pdig". But if I click through to the pdig repo it says it was archived in 2022. I don't want to use unmaintained software, obvs. Does Falco just not support serverless workloads anymore? Why do they have this on their site if it's not supported anymore? Is there an alternative driver for Falco for serverless? Or should I be looking elsewhere for runtime monitoring?

https://redd.it/1uy4jpa
@r_devops

Prompt injection is a supply chain problem, not a model problem

Hot take from a banking DevSecOps team: treating prompt injection as something the model vendor should fix is a dead end. The real issue is that the context window has no provenance. The model cannot tell user instructions from a poisoned README or a tool response. Until that changes, the practical mitigations look a lot like classic supply chain controls: pin your dependencies, verify what you fetch, restrict what each component is allowed to do. Anyone mapping this to SLSA-style controls yet?

https://redd.it/1uy7l06
@r_devops

Self hosted sentry issue

We are running a self-hosted Sentry deployment on EKS using the official Sentry Helm chart. Due to cost constraints, we removed the larger instance types and currently only have "t3a.large" nodes. As expected, we are facing bottlenecks, primarily related to memory, since many Sentry components are quite memory-intensive.

Additionally, the same node group is hosting other workloads, so adding larger instance types is not currently feasible due to existing constraints.

Has anyone faced a similar situation? How did you optimize costs while keeping Sentry stable? Are there any recommendations for reducing resource consumption in self-hosted Sentry, or would migrating to managed Sentry be a better option in this case?

https://redd.it/1uy717p
@r_devops

How would you all

So I just wanted to ask what is the best way to learn and master

Devops

Imran teli udemy course , Techworld with nana or anything else ??

Like I watched nana vids she said focus on Basics

And I feel that's right

So I wanted to get advice on

Courses or learning paths , the mistakes not to make and ways



https://redd.it/1uy1hh2
@r_devops

Hosting Experience Comparison

For deployment to microvms (lambda like) vs container based deployments ? Which experience are the best for developers ? Whats the hardest/limitations for each kind ?

https://redd.it/1uydet0
@r_devops

Regrets leaving previous DevOps role as I am not enjoying the new company

Last month I left my previous DevOps role for a new one. The pay increase was the main reason I left. My old role had good colleagues, interesting work, and an office I could go into whenever I wanted, but I just didn't feel I was being paid enough for the work I was doing. It was a fairly big company and I'd been there 7 years.

The new company is smaller and fully remote. I'm starting to feel like the grass isn't always greener on the other side. Here are the pain points I'm facing:

Autonomy - The security team always needs input on infrastructure designs, and the process adds way too much time before I can actually deploy anything. Some of their recommendations are just overboard, like I get the idea behind them but they're not practical. Security are quite a big blocker and I have had to ask them to keep granting me more access to help debug production issues, which takes them over 6 hours to implement.

AWS accounts galore - When running Terraform, you have to keep logging into different profiles just to plan and apply. It gets confusing, and the way they've split up the accounts feels excessive.

High expectations, no tools - Senior software engineers want fast progress, but the security team hasn't signed off on giving people in my role access to AI tools. Security and software engineers have AI, but DevOps doesn't, because we have SSH access to prod clients.

Meetings galore - There are so many "syncs" throughout the day/week. 2 daily syncs plus another 1-2 meetings most days. In my previous role I had maybe one stand up a week.

Manager - My manager doesn't really have a backbone and doesn't fight for us. When senior software engineers change direction, instead of pushing back and telling them what our path is, he just makes us appease them. He doesn't code or help with the workload either, he's literally just a manager. My old manager was also an engineer I could go to for help.

High turnover - After being here a while I've noticed quite a few people have left, and most of the people who are here are pretty new. They've hired around 7 engineers this year in DevOps, but even the person who interviewed me, who seemed pretty strong, has already left.

The work itself isn't that difficult, it's dealing with people in these meetings that's annoying, and they expect me to move faster than what's actually possible given the tools I've been given. I'm not sure if the company's just in a bad place, if I'm too used to my old company, or if this is just what changing companies is like.

Wondering what your thoughts are, and whether you think I should start looking for new roles ASAP. I don't want to go back to my previous company but they would take me back if I was to apply. I would rather try somewhere else.

https://redd.it/1uygsci
@r_devops

System Admin for 2+ Years but I Feel Like I Never Learned SysAdmin. Want to Transition to DevOps. Where Should I Start?

Hi everyone,

I've been working as a System Administrator for a little over 2 years, but I honestly feel like I haven't gained the kind of experience most people associate with system administration.

I work at a very old/traditional company where my day-to-day work is mostly repetitive:

* Installing the same software over and over
* Basic monitoring
* Following predefined SOPs
* Very little troubleshooting, automation, or infrastructure work

Because of this, I feel like I've missed many of the fundamentals that most SysAdmin pick up along the way.

I want to transition into DevOps, but I'm starting almost from scratch. I know it's not an entry-level role, and I'm prepared to put in the effort to learn properly.

If you were mentoring someone in my position, what would your roadmap look like?

Some questions I have:

* What core System Administration topics should I master first ??
* Which Linux concepts are absolutely essential??
* What networking knowledge is expected ??
* Which scripting language should I prioritize (Bash or Python) ??
* When should I start learning Docker, Kubernetes, Terraform, Ansible, CI/CD, and cloud platforms like AWS ??
* What projects would actually help me build real-world skills instead of just collecting certificates ??
* Are there any resources, books, YouTube channels, or courses that you genuinely recommend??

My goal is to become job-ready for a DevOps role, even if it takes several months of consistent learning.

I'd really appreciate any roadmap, advice, or lessons you wish someone had told you when you started.

Thanks in advance!

https://redd.it/1uyizr3
@r_devops

What areas of IT should you focus on when graduating?

I’ve been working as a SysOps engineer for a year, and it’s what I’ve been doing since I graduated.

I was wondering: if you were in my shoes, what would you invest your time in?

How do you see the future of our profession with the rise of AI?

https://redd.it/1uyhiek
@r_devops

Learning Azure ADO

Can someone give me review for https://www.azuredevopslabs.com. I am trying to learn devops and then devsecops. My main goal is to learn devsecops. Can anyone guide me to some resource for Azure ADO like some guided labs.
Thank you in advance!

https://redd.it/1uyorl7
@r_devops

Announcing v1 of OpenTelemetry Go Compile-Time Instrumentation
https://opentelemetry.io/blog/2026/go-compile-time-instrumentation-v1/

https://redd.it/1uyrggy
@r_devops

Recently Deployed n8n on ECS fargate on AWS

I recently deployed n8n on ECS fargate on AWS eith Terraform by keeping the n8n UI accessible with in the VPN and webhook endpoint to be publicly accessible.

Happy to answer questions and discuss if someone is facing issues with the n8n deployment.

https://redd.it/1uyt5v3
@r_devops