How to properly deal with a local sqlite database in a stateful node/sqlite webapp?

I'm building a simple web app with node/express that uses a local sqlite database. (I'm not experienced in devops, btw)

Although I've already managed to get this app running in production by doing a manual deploy via SSH and cloning my repo from github, I'm having issues understanding how I'm going to deal with the database (.db) file in production.

Since my database is running on the same debian VPS as the application, I'm starting to realize I need to be a little bit more thoughtful and stabilish some sort of process if I ever need to restore a db backup or do migrations on this database.

I though about leaving the db file outside the project directory, which should allow me to safely pull new code that I push to production while leaving the db file intact, (as long as I'm not changing db structure/doing migrations) but I'm not quite sure how to deal with backups, migrations and whatever scenarios might come down the line.

Do any of you have suggestions on how to deal with this sort of stateful local database scenario?

https://redd.it/1v6hdgz
@r_devops

How do you reduce risk when deploying to prod?

I’m researching how engineers make safer production changes.

Many teams now use strategies like automated tests, regression tests, canary releases, automated rollbacks, and better observability, but production incidents still happen after deployments.

I’m curious about real experiences from engineers:
\- What type of change makes you the most cautious before deploying?
\- What was the last deployment that made you nervous and why?
\- What gives you enough confidence to press the deploy button?

For me, database migrations are still one of the hardest categories.
Would love to learn how other teams approach this.

https://redd.it/1v6ji9l
@r_devops

Built-in request dashboard for outbound API calls in Laravel

Debugging third-party API failures usually means digging through logs or adding one-off logging.

LaraClient logs every outbound request (with secrets redacted) and ships a dashboard to filter by connection, endpoint, and failure, plus a latency trace strip so you can spot slow/failing windows quickly.

Gated behind a Laravel gate by default outside local.

https://github.com/usamamuneerchaudhary/laraclient

https://redd.it/1v6ld4q
@r_devops

How did you actually get started with Kubernetes?

I’m a junior DevOps engineer and I’ve never had the chance to use Kubernetes at work.

I’d like to learn it, but almost every job posting I see asks for people who already have professional Kubernetes experience. So I’m wondering how you’re supposed to get past that first step when nobody wants to hire a beginner.

How did you personally start working with Kubernetes? Did your company let you learn it on the job, did you end up on a Kubernetes project by chance, or did you first build personal projects?

I’m considering setting up a small homelab and building some projects, but I’m not sure how much companies actually value personal Kubernetes experience.

I’d be interested to hear how it happened for you.

https://redd.it/1v6n8ac
@r_devops

Transitioning from IT to DevOps

Hello! I've been wanting to ask but the amount of "AI slop" comments has been making me second guess. Anyway.

Just curious if I'm jumping in over my head.

My background is 8 years in IT. Local MSP so small to medium businesses. Intune Administration for 15,000 devices, project engineer, SysAdmin, Veeam Engineer, basic networking, and a lot of helpdesk. Lots of resolving compliance failures from audits.

I have a friend who is getting me in touch for some interviews at his company doing DevOps work. Pulumi, GitOps, AWS, GCP, k8s.

I've been messing with my homelab for 6 or so months and have done basic things with it using AI as a glorified search engine for documentation.

They would be offering me a position as an "Apprentice" but my first project would be shoring up SOC2 compliance in the infrastructure through Pulumi (all written in Golang).

I learn quickly. But have no current Golang experience.

Am I in over my head? Or can this be reasonably learned on the job in the next 3-6 months?

Edit: Goal is to allow me to learn DevOps and engineering before transitioning me to Jr engineer at this company.

https://redd.it/1v6owff
@r_devops

Looking for DevOps Engineer opportunities (3+ YOE) – Resume available

Hi everyone,
I'm a DevOps Engineer with 3+ years of experience and I'm currently looking for new opportunities.
My experience includes:
AWS & Azure

Kubernetes & Docker

Terraform & CloudFormation

Jenkins & GitHub Actions

Linux, Bash & Python

CI/CD Automation

Prometheus, Grafana & ELK Stack

I'm open to remote, hybrid, or onsite roles in India as well as international opportunities.
If your company is hiring or you're aware of any openings, I'd really appreciate a referral or any leads. I'm happy to share my resume and discuss my experience.
Thank you in advance!

https://redd.it/1v6s0jl
@r_devops

PostPQC is going to become a certificate operations problem

PQC is often discussed as a cryptography problem. I think the harder part may be operational finding every certificate, supporting larger keys and signatures, and getting themthrough the infrastructure around the service. As far as I can tell, the main reverse-proxy and load-balancer vendors are not supporting this properly yet.

How are other teams approaching PQC readiness? Have you started testing, or are you still waiting for vendor support and clearer standards? I have a feeling this could be a major infrastructure change, similar to the move from HTTP to HTTPS.

Open discussion here :)

https://redd.it/1v6srr9
@r_devops

Should agents need a second approval before deploy/delete actions?

I am trying to think about coding agents less like chatbots and more like automation workers with tokens.

For read-only work, I want low friction. Let the agent inspect logs, read issues, check a config, or prepare a patch. But once the action turns into deploy, delete, push, send, or modify production data, the original token approval feels too broad to me.

When I connect external tools, I am comfortable letting the agent read issues or draft a change, but I still want a second confirmation before it pushes, deletes, deploys, or sends anything outside the repo.

Maybe this is just my bias from small projects, but I would rather add one annoying approval step than debug a silent automated action later.

For people running automation in real environments: would you treat agent write actions like any other gated production action, or is that too much friction?

https://redd.it/1v6rxjm
@r_devops

AWS vs Azure, which cloud to learn wrt Indian market?

I always heard that in the Indian market AWS has more jobs, is it still the same case

https://redd.it/1v6x9sc
@r_devops

our self-healing code loop actually cut our incident volume

our self-healing code loop actually cut our incident volume

i run platform for a team of nine and i was loud about self-healing code being marketing nonsense. changed my mind a bit, posting the numbers before i talk myself out of it

the setup is dumb honestly. fable 5 plans a fix, composer 2.5 does it, coderabbit / bugbot review, keep going until it comes back clean. nothing merges until an extreme pile of tests passes too. were b2b saas so theres no room for a big mistake, id rather ship stable than fast

rough before/after over ten weeks, our numbers not a benchmark:

pages/week from flaky fixes: 11 -> 4

median time to a merged p2 fix: \~3.5 hrs -> 40 min

fixes needing a second human pass: a lot -> maybe 1 in 5

way less manual review, way better code. the only real downside is its costy, running review 3-4 times a fix adds a few hundred a month

https://redd.it/1v74m8f
@r_devops

Is prompt engineering still just a vibe check

Bit of a venting session here, feel free to delete if not allowed on this sub, but, I feel like our monthly prompt review meetings have become a total headache. We spend way too much time arguing over a single freaking sentence which inevitably turns into a bit of a shouting match and then the loudest person in the room wins since we've all gotten exhausted. We then just ship it and pray. We're a very early stage startup, and so I get that this is just part of the growing pains, but it feels insane to me.

I am trying to move toward actual canary deployments for prompts. I want to stop just pushing a new version and start running the new variant alongside the current one on a small slice of live traffic. I want to let the data tell us if it is actually better. Our CTO hasn't done anything to try and fix the problem, and I really want something strong to present to him.

My biggest worry at the moment is the overhead though. Keeping track of five different iterations at once is a mess. We have been using Braintrust to handle the versioning and scoring deltas just so I can show our CTO that Version B has a 12% higher alignment score (number go up = good ammiright?)

Still trying to figure out the right balance between offline and online testing though.

How are you guys handling canary rollouts? Are you doing automated scoring on live traffic or are you still doing manual spot checks? At what point is an offline eval safe enough for you? Anyone actually found a way to make this A/B loop fast enough that it doesnt kill dev velocity?

https://redd.it/1v76qdv
@r_devops

NAT Gateway was costing us $3.8k/month and nobody noticed for 4 months - here's the CUR query that found it

This is a writeup I probably should have done months ago. Sharing in case the query saves someone else from finding this the hard way.

**Background**

Standard multi-account setup: most compute in private subnets, NAT Gateways across a few regions. Production, staging, and a batch processing environment that we don't look at as carefully as production.

AWS bill had been climbing - roughly $800/month more each month for about four months. Not enough to trip our anomaly alerts (we had percentage-based thresholds and total spend was growing legitimately), but enough that it finally showed up in a quarterly review.

**How I found the actual culprit**

Cost Explorer filtered by service showed NAT Gateway high, but so was compute, so I couldn't tell if it was proportional.

What actually helped: filtering by Usage Type and pulling out NatGateway-Bytes specifically. That's the data processing charge, distinct from the flat hourly gateway charge. Our NatGateway-Bytes was running around 85 TB/month. Way more than our actual application traffic justified.

To get per-NAT-gateway detail, I had to query our Cost and Usage Report in Athena. Native Cost Explorer doesn't give you per-resource granularity on NAT data processing. If you haven't set up CUR yet, this is the reason to do it.

Query that found it:

SELECT line_item_resource_id, SUM(line_item_blended_cost) as cost, SUM(CAST(line_item_usage_amount AS DOUBLE)) as gb_processed FROM your_cur_table WHERE line_item_product_code = 'AmazonEC2' AND line_item_usage_type LIKE '%NatGateway-Bytes%' AND line_item_usage_start_date >= DATE('2026-03-01') GROUP BY 1 ORDER BY 2 DESC LIMIT 20;

The top result was a NAT Gateway in us-west-2 that only existed because of our batch VPC. It was processing \~80 TB/month on its own. Our main app NAT Gateways were doing around 5 TB combined.

**Root cause**

VPC Flow Logs for the batch subnet answered it quickly. A batch job doing heavy S3 reads and writes was routing all of it through NAT instead of directly to S3.

Why? We'd added S3 VPC Gateway Endpoints to our main application VPC (they're free, should be everywhere). But the batch VPC got spun up separately as "temporary" infrastructure, never got the same attention, and nobody ever added the endpoint. S3 VPC Gateway Endpoints route S3 API traffic directly within AWS's network and bypass the NAT Gateway entirely - every GetObject, PutObject, ListBucket was going through NAT and incurring the $0.045/GB processing charge.

We were also pulling ECR images through NAT in that environment. That adds up too if you're pulling big images frequently.

**The fix**

Three things:

1. Added S3 VPC Gateway Endpoint to the batch VPC route table. Free. Took about 5 minutes. Cut NatGateway-Bytes for that gateway by \~92%.

2. Added VPC Interface Endpoints for ECR (ecr.api and ecr.dkr). Not free ($0.01/hr each) but the data processing savings easily justified it.

3. Set up Cost Anomaly Detection on NatGateway-Bytes with an absolute dollar threshold, not just percentage-based. We had anomaly detection on compute but not on data transfer. That's fixed now.

After a full billing month: NAT Gateway data processing went from \~$3,800 to \~$200. ECR endpoints add about $15/mo in hourly charges. Net save: roughly $3,600/month.

**What I'd do differently**

Enable CUR from day one, not after you have a problem. Cost Explorer is fine for headlines. For per-resource attribution of NAT charges you need Athena against CUR.

Set absolute dollar thresholds on specific line items in Cost Anomaly Detection, not just percentage thresholds on total spend. NatGateway-Bytes as a specific monitor would have caught this in week two instead of month four.

Audit every VPC for VPC endpoints independently, including the "temporary" ones. Production got the full setup. The batch VPC was treated as temporary infrastructure and never got properly reviewed. That's where the cost hid.

Happy

to share more on the CUR schema or the anomaly detection setup if anyone's dealing with something similar.

https://redd.it/1v7aurp
@r_devops

Bringing trust to your CI System

How would you even know if your build pipeline was compromised — not your code, the system that builds it?

That question is basically what pulled me down a rabbit hole last weekend. I was just going back to basics, re-reading the fundamentals of secure software delivery, and somehow landed right back on the one question that always pops up eventually: how much does anyone actually trust CI?

Because if someone quietly got a foothold in a build system, the artifact that comes out the other end still looks completely legitimate. Nothing downstream catches it, because most pipelines are built to just trust the artifact by default. That's the same pattern behind incidents like SolarWinds and Codecov — not application bugs, trust failures in the delivery path itself.

So I went down the rabbit hole properly — the current SLSA framework, provenance, in-toto attestations, the whole "how do you actually prove what happened during a build" question. Turns out the industry's answer is pretty elegant: provenance is a verifiable record of what happened during the build. Attestation is how that record gets proven, cryptographically, at every step after.

A few late nights later, my notes turned into something worth keeping — a handbook I know I'll come back to, more than once. The whole thing distills into one line I keep coming back to:

Git → Build → Artifact → Deploy → Runtime — identity, evidence, and independent verification at every boundary.

If you operate any part of a software delivery path to production, ask yourself that question today.

Read the handbook at https://securesoftwaredelivery.wckd14.xyz/

\#PlatformEngineering #SupplyChainSecurity #SLSA #SSDLC #DevSecOps

https://redd.it/1v7cj9j
@r_devops

cluster design + cost/CapEx-vs-cloud

I built a free, vendor-agnostic GPU platform tool — compare NVIDIA/AMD, get a workload rec, and cost a cluster

I kept juggling spec sheets and spreadsheets every time I had to pick GPUs, so I built GPU Platform Hub. It lets you compare datacenter GPUs (NVIDIA H100/H200/Blackwell + AMD Instinct) side by side, describe a workload to get a ranked shortlist, and design a costed cluster (CapEx vs cloud, power, rack footprint).

It's free and needs no login to explore: https://hub.inferonomics.ai

I'd genuinely like feedback from people running local/production GPU setups — what's the most painful part of your GPU decision process? Anything you'd want it to cover?

https://redd.it/1v7mu0w
@r_devops

How GitHub gave every one of their 14,000+ repos a validated owner in 45 days

This is a summary of their blog post on how they assigned owners to over 14,000 internal repos when as of early 2025, less than half had a clear owner. It became a problem during a secret-scanning remediation push. they could rotate a leaked secret, but without knowing who owned the repo, doing so was risky and there was no way to route the work.

Their old model only tracked ownership for repos backing a deployed service, through an internal Service Catalog. That covered maybe a third of repos. Everything else, team repos, docs, internal tools, one-off projects, personal experiments, had no owner on record. Finding one meant checking commit history, reading the README, or asking around in Slack.

What they built:

Two GitHub custom properties: ownership-type (Service Catalog / Team / Individual) and ownership-name (free text, validated against actual org membership, team existence, or Service Catalog entries).

Before asking a single person to do anything, they synced ownership from the existing Service Catalog, which auto-covered about 1,500 repos.

For the rest, they ran a GitHub App on a Kubernetes CronJob that opened an issue on unowned repos with a 30-day deadline, then archived anything still unclaimed. They picked archiving because it's reversible, repo goes read-only, Actions stops, nothing gets deleted.

Two incidents:

1. They archived a repo that had a Datadog integration configured to open issues in it. Datadog couldn't create the issue post-archive, and nobody had been directly notified about the pending archival, their own monitoring paged the owning team before the ownership system did. Fix: they started @-mentioning repo admins directly and assigning all write-access users as a fallback.
2. They realized a stale or corrupted response from Service Catalog could trigger mass-archiving of repos that actually had valid owners. So they added a threshold check. If a single run is about to archive or flag more repos than some conservative limit, it aborts and pages instead of running.

End state: \~3,000 active repos, \~11,000 archived (up from \~3,000 archived at the start), done in under 45 days. They also tightened the ongoing enforcement loop from a 30-day grace period down to one hour, and made ownership required at repo creation going forward.

https://redd.it/1v7o3i8
@r_devops

I own a wood factory and want to use webscrapers to get info

I own a few wood factories that specialize in producing custom fit-outs for hotels/schools/universities/houses/offices and all sorts around the Middle East, the business model we have been running is very out dated, we run purely off of reputation and returning customers as we have been in the business for over 25 years now.

I want to implement webscrapers but I am not sure what data I can scrape that would help my business grow.

I would really appreciate any advise to what data I could scrape to help me find more contracts for example or anything else that you guys would think would be valuable.

https://redd.it/1v7p7n7
@r_devops

Weekly Self Promotion Thread

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

https://redd.it/1v7scb2
@r_devops

Agentic ai devops after a year

we run a fairly boring b2b product, 20ish services, small platform team. a year ago management got excited about agentic ai devops and told us to "explore aggressively". so we did, and honestly most of it got thrown out



what didnt stay:



\- ai generated terraform. we tried stackgen and messed around with facets too, the natural language to terraform thing is genuinely impressive in a demo. but infra code has to BE right, not look right, and "looks right" is exactly what these are good at. every output still needed a human to read every line so where's the win

\- an ai pipeline optimizer that promised faster CI, mostly just added a service we had to babysit

\- an infra chatbot that answered questions wrong just confidently enough to be dangerous. someone almost ran a suggested command in prod, thats when it got killed



what stayed:



\- ai written deploy changelogs, an agent reads the merged PRs and posts a summary to slack. zero effort, everyone actually reads them

\- coderabbit on every PR. catches missing config changes and deploy risks before merge, this is the one we'd riot over if it got removed. it gets noisy on big PRs tho, we tuned it down

\- alert correlation. instead of 15 alerts when one db degrades we get one summary with a probable cause. right maybe 60% of the time which sounds bad until you remember the old system was right 0% of the time, it just screamed



what we're watching but not touching, auto rollbacks on anomaly detection and agent managed feature flags. maybe next year. the line right now is agents can read and summarize anything but they dont get to touch infra, a human still presses the button

https://redd.it/1v7u28r
@r_devops

Planning an EU launch. Looking for real-world GDPR implementation advice.

I'm currently leading the GDPR implementation for our company, and honestly, the deeper we get into it, the more I realize how much there is to do.

Our platform wasn't originally built with GDPR in mind. Right now our primary database is in Australia, and we serve customers in multiple countries from the same infrastructure. As we're planning an EU launch, we're now figuring out what needs to change to become GDPR compliant.

Initially, I thought this would mostly be about infrastructure and documentation, but it's becoming clear that there are quite a few application-level changes as well. It's made me realize that if there's even a chance your product will serve EU users in the future, it's probably worth considering GDPR from day one rather than trying to retrofit it later.

For those who've actually implemented GDPR, I'd love to hear about your experience.

Some things I'm curious about:

What ended up being the hardest part?
Was it more of a legal/compliance challenge or a technical one?
Did you end up redesigning your architecture?
Did you create a separate AWS account or separate EU infrastructure, or keep everything in the same account?
How did you handle data residency and cross-border data transfers?
If your database was already in another region, did you migrate it or keep it where it was?
How did you handle backups when users requested data deletion?
Were there any code changes that surprised you?
Did any third-party services become a problem?
Did you use tools like CompAI, Vanta, Drata, or Secureframe? Were they worth it?
If you have a single admin portal managing multiple regions (US, AU, CA, EU), how did you handle access to EU customer data? Did you keep one global admin system with role-based access, or did you separate access by region (e.g., EU admins only seeing EU data)? How did you handle this in practice?
Looking back, what do you wish you'd known before starting?

I'm basically looking for the "I wish someone had told me this before we started" kind of advice.

If you've been through a GDPR implementation and wouldn't mind answering a few questions, please leave a comment or send me a DM.

https://redd.it/1v7qey5
@r_devops

Alternatives to Kubernetes for medium-sized organizations?

Kubernetes is a great platform, but it can be expensive and may be more than what’s needed for many medium-sized organizations.

What are the common alternatives being used in production?

Interested in hearing what solutions have worked well in real-world environments and the reasons behind those choices.

https://redd.it/1v7u665
@r_devops