What is the long term plan for charts supporting Gateway API?

We manage all of our infra apps like Vault, Argo Workflows and many others using Terraform + Helm. TF creates the ArgoCD ApplicationSet which points to the chart and values managed in the repo, and ArgoCD syncs everything.

During the transition to GW API I noticed that most charts either don't support GW API natively, or only support creating an HTTPRoute. With ingress, all we had to do was define a single ingress.enabled: true block with some annotations, paths, etc.

Now with GW API, you usually can't manage everything from the app's values file. Even when the chart creates an HTTPRoute, it doesn't create the other resources we need, such as a ReferenceGrant for cross-namespace references or controller-specific resources like Envoy Gateway's SecurityPolicy and BackendTrafficPolicy. Argo Workflows is one example where the chart supports creating an HTTPRoute, but not the rest of that setup.

We have an okay solution for this today: we just create a manifests/ folder for each app and add it as another source in the ArgoCD AppSet (applied by TF, and everything is as GitOps as it gets). In that manifests folder I create all the needed Gateway API and Envoy Gateway resources in a single multi-manifest YAML file.

What do you guys do for 3rd party charts? What is the long term plan for apps in terms of Gateway API support? Is the expectation that charts will eventually expose more Gateway API values, or that routing and policies should live separately from the app chart?

I'm just curious since this whole GW API thing is very new, and I assume a lot of apps will need to add "ingress-like" support at some point?

https://redd.it/1v57q3k
@r_devops

How do devops / engineering services share documentation with clients?

Hello,

We provide devops / engineering services and use GitHub for documentation (runbooks, configs, processes). Clients need access, but raw GitHub feels too technical.

* How do you share documentation with clients in devops / engineering services?
* Do you use GitHub, wikis, or dedicated portals?
* How do you handle versioning and access control?

https://redd.it/1v4s0yd
@r_devops

The difference between safety filters and actual model quality

Just wrapped up a rollout and felt like a bit of a reality check because I think we totally conflated safety with actual quality.

We spent way too much time on the safety layer. Runtime filters, moderation, and injection blocks (the whole 9 yards). I think on paper, we were looking solid. But in practice, the model was hallucinating just as much as it would have without those checks in place. It was giving these super confident, polite answers that had absolutely nothing to do with what the user actually asked for. But, and this is a big but, because the responses were safe, they sailed right through every guardrail we spent so much time on.

Maybe this is obvious, and I just didn't take the time to think about it. Guardrails are great for catching the obvious bad stuff but they do nothing for that middle ground where the user experience actually lives. You can be perfectly safe and still be completely useless to a customer. Duh.

We had to overall our moderating to actually start building a dataset to catch these polite hallucinations. We are now running regression tests every time we touch a prompt.

https://redd.it/1v5pmab
@r_devops

Looking for feedback on my open-source production-grade AWS EKS platform architecture
https://github.com/Saif-hadd/terraform-aws-production-infrastructure

https://redd.it/1v5p19e
@r_devops

Experience with IBM LinuxOne

Hello! My company is considering buying a LinuxOne machine to run Mongo. Has anyone actually worked with one? Any challenges besides the fact that it only supports s390 arch?

https://redd.it/1v5tg98
@r_devops

If a policy changes mid-run, should the worker fail closed?

I keep running into a boring automation question that becomes more annoying with coding agents.

A task is running. I change a network allowlist, proxy, or sandbox rule. The UI says the setting is saved. But unless the worker confirms a reload, I do not actually know which policy snapshot is active.

In one small local setup, I changed a proxy and allowlist while an agent task was running, then realized I could not tell whether the active run had picked up the new rules or still used the old snapshot.

For normal automation, I would expect some mix of config version, reload event, and audit log. For agents, I think it matters even more because the task can keep making decisions while I assume the boundary changed.

If a policy becomes stricter during a run, would you pause/fail closed, reload at the next safe point, or allow the current run to finish under the old snapshot?

https://redd.it/1v5uytg
@r_devops

Terraform Question I am asked almost everytime

You're working in production.

Your Terraform backend is:

S3 for state
DynamoDB for locking

A developer accidentally deletes the DynamoDB lock table.

At the same time, two engineers run terraform apply.

Thirty minutes later, the infrastructure is in an inconsistent state.

# Questions

1. What exactly happened internally?
2. What kind of corruption could occur?
3. How would you recover without making things worse?
4. Would you restore the state file? Import resources? Edit the state manually? Explain your reasoning.
5. What preventive controls would you implement so this never happens again?

https://redd.it/1v637pl
@r_devops

How do you manage/handle openAPI specs sharing between teams?

Hey everyone,

Looking for some advice on a better way to handle and secure our Swagger/OpenAPI docs without overcomplicating our stack or breaking the bank.

Our current setup:

We have several projects hosted on Azure. Each project has standard Dev, Staging, and Prod environments and 3 engineering teams (Backend, Mobile, Web) who rely heavily on Swagger to coordinate and integrate features.

Right now, the backend team just serves Swagger UI directly from the deployed apps. To lock it down, we threw a custom Basic Auth middleware over the `/swagger` route

It works, but honestly, it feels messy and insecure. Managing these shared credentials across multiple teams and environments is becoming a pain, and I really hate exposing the UI endpoints to the public internet at all

I looked into full-blown IDPs like Backstage and OpsLevel, but they are way too bloated and complex for what we actually need.

How did you solve this?



https://redd.it/1v66xq8
@r_devops

DevOps in BigTech

Hij, everyone.
I work as a DevOps/SRE at a consultancy serving large companies and have about two years of experience. My goal is to join a Big Tech company in the coming years.

In your opinion, what should I focus on from now on? What knowledge, technologies, or skills really make a difference for someone aiming to get into that kind of company?

I’d especially like to hear from those who have already gone down this path. What would you prioritize studying if you were starting this preparation today?
Thanks!

https://redd.it/1v6fyj2
@r_devops

How to properly deal with a local sqlite database in a stateful node/sqlite webapp?

I'm building a simple web app with node/express that uses a local sqlite database. (I'm not experienced in devops, btw)

Although I've already managed to get this app running in production by doing a manual deploy via SSH and cloning my repo from github, I'm having issues understanding how I'm going to deal with the database (.db) file in production.

Since my database is running on the same debian VPS as the application, I'm starting to realize I need to be a little bit more thoughtful and stabilish some sort of process if I ever need to restore a db backup or do migrations on this database.

I though about leaving the db file outside the project directory, which should allow me to safely pull new code that I push to production while leaving the db file intact, (as long as I'm not changing db structure/doing migrations) but I'm not quite sure how to deal with backups, migrations and whatever scenarios might come down the line.

Do any of you have suggestions on how to deal with this sort of stateful local database scenario?

https://redd.it/1v6hdgz
@r_devops

How do you reduce risk when deploying to prod?

I’m researching how engineers make safer production changes.

Many teams now use strategies like automated tests, regression tests, canary releases, automated rollbacks, and better observability, but production incidents still happen after deployments.

I’m curious about real experiences from engineers:
\- What type of change makes you the most cautious before deploying?
\- What was the last deployment that made you nervous and why?
\- What gives you enough confidence to press the deploy button?

For me, database migrations are still one of the hardest categories.
Would love to learn how other teams approach this.

https://redd.it/1v6ji9l
@r_devops

Built-in request dashboard for outbound API calls in Laravel

Debugging third-party API failures usually means digging through logs or adding one-off logging.

LaraClient logs every outbound request (with secrets redacted) and ships a dashboard to filter by connection, endpoint, and failure, plus a latency trace strip so you can spot slow/failing windows quickly.

Gated behind a Laravel gate by default outside local.

https://github.com/usamamuneerchaudhary/laraclient

https://redd.it/1v6ld4q
@r_devops

How did you actually get started with Kubernetes?

I’m a junior DevOps engineer and I’ve never had the chance to use Kubernetes at work.

I’d like to learn it, but almost every job posting I see asks for people who already have professional Kubernetes experience. So I’m wondering how you’re supposed to get past that first step when nobody wants to hire a beginner.

How did you personally start working with Kubernetes? Did your company let you learn it on the job, did you end up on a Kubernetes project by chance, or did you first build personal projects?

I’m considering setting up a small homelab and building some projects, but I’m not sure how much companies actually value personal Kubernetes experience.

I’d be interested to hear how it happened for you.

https://redd.it/1v6n8ac
@r_devops

Transitioning from IT to DevOps

Hello! I've been wanting to ask but the amount of "AI slop" comments has been making me second guess. Anyway.

Just curious if I'm jumping in over my head.

My background is 8 years in IT. Local MSP so small to medium businesses. Intune Administration for 15,000 devices, project engineer, SysAdmin, Veeam Engineer, basic networking, and a lot of helpdesk. Lots of resolving compliance failures from audits.

I have a friend who is getting me in touch for some interviews at his company doing DevOps work. Pulumi, GitOps, AWS, GCP, k8s.

I've been messing with my homelab for 6 or so months and have done basic things with it using AI as a glorified search engine for documentation.

They would be offering me a position as an "Apprentice" but my first project would be shoring up SOC2 compliance in the infrastructure through Pulumi (all written in Golang).

I learn quickly. But have no current Golang experience.

Am I in over my head? Or can this be reasonably learned on the job in the next 3-6 months?

Edit: Goal is to allow me to learn DevOps and engineering before transitioning me to Jr engineer at this company.

https://redd.it/1v6owff
@r_devops

Looking for DevOps Engineer opportunities (3+ YOE) – Resume available

Hi everyone,
I'm a DevOps Engineer with 3+ years of experience and I'm currently looking for new opportunities.
My experience includes:
AWS & Azure

Kubernetes & Docker

Terraform & CloudFormation

Jenkins & GitHub Actions

Linux, Bash & Python

CI/CD Automation

Prometheus, Grafana & ELK Stack

I'm open to remote, hybrid, or onsite roles in India as well as international opportunities.
If your company is hiring or you're aware of any openings, I'd really appreciate a referral or any leads. I'm happy to share my resume and discuss my experience.
Thank you in advance!

https://redd.it/1v6s0jl
@r_devops

PostPQC is going to become a certificate operations problem

PQC is often discussed as a cryptography problem. I think the harder part may be operational finding every certificate, supporting larger keys and signatures, and getting themthrough the infrastructure around the service. As far as I can tell, the main reverse-proxy and load-balancer vendors are not supporting this properly yet.

How are other teams approaching PQC readiness? Have you started testing, or are you still waiting for vendor support and clearer standards? I have a feeling this could be a major infrastructure change, similar to the move from HTTP to HTTPS.

Open discussion here :)

https://redd.it/1v6srr9
@r_devops

Should agents need a second approval before deploy/delete actions?

I am trying to think about coding agents less like chatbots and more like automation workers with tokens.

For read-only work, I want low friction. Let the agent inspect logs, read issues, check a config, or prepare a patch. But once the action turns into deploy, delete, push, send, or modify production data, the original token approval feels too broad to me.

When I connect external tools, I am comfortable letting the agent read issues or draft a change, but I still want a second confirmation before it pushes, deletes, deploys, or sends anything outside the repo.

Maybe this is just my bias from small projects, but I would rather add one annoying approval step than debug a silent automated action later.

For people running automation in real environments: would you treat agent write actions like any other gated production action, or is that too much friction?

https://redd.it/1v6rxjm
@r_devops

AWS vs Azure, which cloud to learn wrt Indian market?

I always heard that in the Indian market AWS has more jobs, is it still the same case

https://redd.it/1v6x9sc
@r_devops

our self-healing code loop actually cut our incident volume

our self-healing code loop actually cut our incident volume

i run platform for a team of nine and i was loud about self-healing code being marketing nonsense. changed my mind a bit, posting the numbers before i talk myself out of it

the setup is dumb honestly. fable 5 plans a fix, composer 2.5 does it, coderabbit / bugbot review, keep going until it comes back clean. nothing merges until an extreme pile of tests passes too. were b2b saas so theres no room for a big mistake, id rather ship stable than fast

rough before/after over ten weeks, our numbers not a benchmark:

pages/week from flaky fixes: 11 -> 4

median time to a merged p2 fix: \~3.5 hrs -> 40 min

fixes needing a second human pass: a lot -> maybe 1 in 5

way less manual review, way better code. the only real downside is its costy, running review 3-4 times a fix adds a few hundred a month

https://redd.it/1v74m8f
@r_devops

Is prompt engineering still just a vibe check

Bit of a venting session here, feel free to delete if not allowed on this sub, but, I feel like our monthly prompt review meetings have become a total headache. We spend way too much time arguing over a single freaking sentence which inevitably turns into a bit of a shouting match and then the loudest person in the room wins since we've all gotten exhausted. We then just ship it and pray. We're a very early stage startup, and so I get that this is just part of the growing pains, but it feels insane to me.

I am trying to move toward actual canary deployments for prompts. I want to stop just pushing a new version and start running the new variant alongside the current one on a small slice of live traffic. I want to let the data tell us if it is actually better. Our CTO hasn't done anything to try and fix the problem, and I really want something strong to present to him.

My biggest worry at the moment is the overhead though. Keeping track of five different iterations at once is a mess. We have been using Braintrust to handle the versioning and scoring deltas just so I can show our CTO that Version B has a 12% higher alignment score (number go up = good ammiright?)

Still trying to figure out the right balance between offline and online testing though.

How are you guys handling canary rollouts? Are you doing automated scoring on live traffic or are you still doing manual spot checks? At what point is an offline eval safe enough for you? Anyone actually found a way to make this A/B loop fast enough that it doesnt kill dev velocity?

https://redd.it/1v76qdv
@r_devops

NAT Gateway was costing us $3.8k/month and nobody noticed for 4 months - here's the CUR query that found it

This is a writeup I probably should have done months ago. Sharing in case the query saves someone else from finding this the hard way.

**Background**

Standard multi-account setup: most compute in private subnets, NAT Gateways across a few regions. Production, staging, and a batch processing environment that we don't look at as carefully as production.

AWS bill had been climbing - roughly $800/month more each month for about four months. Not enough to trip our anomaly alerts (we had percentage-based thresholds and total spend was growing legitimately), but enough that it finally showed up in a quarterly review.

**How I found the actual culprit**

Cost Explorer filtered by service showed NAT Gateway high, but so was compute, so I couldn't tell if it was proportional.

What actually helped: filtering by Usage Type and pulling out NatGateway-Bytes specifically. That's the data processing charge, distinct from the flat hourly gateway charge. Our NatGateway-Bytes was running around 85 TB/month. Way more than our actual application traffic justified.

To get per-NAT-gateway detail, I had to query our Cost and Usage Report in Athena. Native Cost Explorer doesn't give you per-resource granularity on NAT data processing. If you haven't set up CUR yet, this is the reason to do it.

Query that found it:

SELECT line_item_resource_id, SUM(line_item_blended_cost) as cost, SUM(CAST(line_item_usage_amount AS DOUBLE)) as gb_processed FROM your_cur_table WHERE line_item_product_code = 'AmazonEC2' AND line_item_usage_type LIKE '%NatGateway-Bytes%' AND line_item_usage_start_date >= DATE('2026-03-01') GROUP BY 1 ORDER BY 2 DESC LIMIT 20;

The top result was a NAT Gateway in us-west-2 that only existed because of our batch VPC. It was processing \~80 TB/month on its own. Our main app NAT Gateways were doing around 5 TB combined.

**Root cause**

VPC Flow Logs for the batch subnet answered it quickly. A batch job doing heavy S3 reads and writes was routing all of it through NAT instead of directly to S3.

Why? We'd added S3 VPC Gateway Endpoints to our main application VPC (they're free, should be everywhere). But the batch VPC got spun up separately as "temporary" infrastructure, never got the same attention, and nobody ever added the endpoint. S3 VPC Gateway Endpoints route S3 API traffic directly within AWS's network and bypass the NAT Gateway entirely - every GetObject, PutObject, ListBucket was going through NAT and incurring the $0.045/GB processing charge.

We were also pulling ECR images through NAT in that environment. That adds up too if you're pulling big images frequently.

**The fix**

Three things:

1. Added S3 VPC Gateway Endpoint to the batch VPC route table. Free. Took about 5 minutes. Cut NatGateway-Bytes for that gateway by \~92%.

2. Added VPC Interface Endpoints for ECR (ecr.api and ecr.dkr). Not free ($0.01/hr each) but the data processing savings easily justified it.

3. Set up Cost Anomaly Detection on NatGateway-Bytes with an absolute dollar threshold, not just percentage-based. We had anomaly detection on compute but not on data transfer. That's fixed now.

After a full billing month: NAT Gateway data processing went from \~$3,800 to \~$200. ECR endpoints add about $15/mo in hourly charges. Net save: roughly $3,600/month.

**What I'd do differently**

Enable CUR from day one, not after you have a problem. Cost Explorer is fine for headlines. For per-resource attribution of NAT charges you need Athena against CUR.

Set absolute dollar thresholds on specific line items in Cost Anomaly Detection, not just percentage thresholds on total spend. NatGateway-Bytes as a specific monitor would have caught this in week two instead of month four.

Audit every VPC for VPC endpoints independently, including the "temporary" ones. Production got the full setup. The batch VPC was treated as temporary infrastructure and never got properly reviewed. That's where the cost hid.

Happy