The whole frontend + backend + db split in k8s, help

I've been a full-stack dev with responsibilities for the servers since '01, so it was a big change for me when I finally went with a big company that had people to do it. It also ment that I was suddenly working with kubernetes. Since everything was set up, it haven't been a big jump but I wanted to learn, so I set up my own 3 node k8s and have been playing with it since. Running full pipeline with dev, production, linting, security scans and all.

But now I want to build something. So I have a few react project running for testing. And I want to make the backend + database split right.

Locally, I am used to run environment variables in the .env file so I can switch between local and dev backends and/or local and dev databases for testing.

In the setup at work, there is a doohickey that controls the environment variables. I just alter files in a git repo and upload, or set it from command line. An enterprise-grade thing developed by the entity I work for. I just want to do it the regular kubernetes way.

So where do I put my database server and port location in a k8s setup? my buddy, mister chatgpt, suggest ConfigMap, coupled with Secret for the password. Is this just a barebone thing or is it how most do it?


Secondly, I don't think putting the database in the backend-pod is the right thing given it writes and reads to it and that sounds wrong. My little buddy suggest that I use either CloudNativePG or set it up as a PersistentVolumeClaim. Is there any other way as well? What would be the preffered way? I assume if I ever get to the stage I make something of this projects public, I am going to publish it to a could service which has their own database stuff so for my own pleasure that is not much of a concern, or is it?

Just curious to know how on track I am.

https://redd.it/1uucsj2
@r_devops

Article: Model caching for AI workloads on GKE/Kubernetes without re-downloading weights

The basic idea is to use node-local storage as a shared model cache so new inference pods can reuse existing weights. It significantly reduces startup time and cuts down on repeated network transfers, especially when autoscaling.

I wrote up the approach, some implementation details on GKE:

https://hrittikhere.com/posts/model-caching-kubernetes-gke

Curious how others are handling model distribution at scale. Are you using node-local caches, RWX storage, image-based models, or something else?

https://redd.it/1uuke1c
@r_devops

Thoughts on the bindplane acquisition?

How are people feeling about bindplane being acquired by dynatrace? Mainly, are people hesitant to adopt it now since they are owned by one of the big observability companies?


And for those who are using bindplane today, have you seen any changes since the acquisition?

https://redd.it/1uuwu5w
@r_devops

To break into devops

Hi All,

Completely out from DevOps, may i know what's the programming language to learn for today's DevOps?

https://redd.it/1uuxoa4
@r_devops

Where is the line for automated agents fixing CI failures?

I am curious how people here would handle AI agents around CI failures.

A simple version sounds useful: CI fails, the agent reads the log, tries to reproduce it, proposes a patch, and comments with the command it ran. That would save time on the boring failures.

But I do not fully trust the next step. I have wanted an agent to handle the boring CI failure, but I stopped when I realized it might comment confidently before it had reproduced the failure locally.

The part that worries me is not the patch itself. It is the noise around it: wrong diagnosis, extra PR comments, fixing the symptom, or making the pipeline greener while the actual issue is still there.

If you were allowing this in a small team, what would you require before the agent comments or pushes a fix? Reproduction command, exact log link, human approval, or only draft patches?

https://redd.it/1uuzyxo
@r_devops

Weekly Self Promotion Thread

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

https://redd.it/1uv3kzk
@r_devops

Help in implementing CICD for Snowflake Objects.

Hey all. I am building a POC for deploying Snowflake objects like table, stream, task, stage etc... to multiple environments like DEV/QA (single only) & PROD with the help of CICD and used Github actions for it.

I have never built something like this before and never worked on project consisting CICD in it.

So, I have used Schemachange library to detect schemachange and sqlfluff for SQL linting. Also using some python scripts to build backup of existing DB snapshot before deployment and rollback script if anything break during deployment.

I am testing this in DEV env only but i am confused how can i validate the objects that are being created with schemachange library and SQL files (which contain DDLs of objects). like how can I verify that the object created is correctly build in the target or not.

and if there is any other suggestion / best practice you guys have that is also welcome on how can i improve the CICD pipeline for it.

Thanks

Bots and moderators I am 100% human only don't remove my post!!!

https://redd.it/1uv7sbp
@r_devops

Roadmap for infrastructure/backend development in the .NET ecosystem?

Hi everyone,

I'm looking for advice from developers who build infrastructure or DevOps-oriented software.

My goal isn't to become a typical CRUD web developer. Instead, I'd like to build things like hosting/server management panels, deployment tools, automation systems, monitoring dashboards, and software that interacts with Linux servers, Docker, networking, SSH, DNS, reverse proxies, and similar infrastructure.

I'll primarily be building these projects with C# and [ASP.NET](https://ASP.NET) Core, but I'm mainly looking for guidance on the infrastructure side rather than the language itself.

If you were starting from scratch today, what roadmap would you follow?

Specifically:

* What concepts should I learn first?
* Which infrastructure and networking topics are the most important?
* Which tools and technologies are worth learning (Docker, Kubernetes, Nginx, systemd, cloud platforms, etc.)?
* What kinds of projects would progressively teach these skills?
* Are there any books, courses, GitHub repositories, or YouTube channels you'd strongly recommend?

I'd appreciate hearing how professionals in infrastructure, platform engineering, backend, or DevOps would approach learning this field today.

https://redd.it/1uvdtif
@r_devops

Degree vs Certs Dilemma: Is a 4-year CS degree worth $15k (20M IQD) if I already have live cloud projects and a home lab at 19?

Hi everyone,

I’m facing a major turning point in my career path and need some unfiltered, realistic advice from people working in the industry.

I am 19 years old, living in Erbil (Kurdistan Region of Iraq), and currently completing a 5-year IT Institute Diploma with about 2 years remaining. Administratively, this diploma functions more like a vocational high school equivalent in our system rather than a university degree.

My ultimate goal is straightforward: I want to secure a Junior Cloud Engineer or Junior DevOps role as efficiently as possible.

I am trying to decide between two completely different paths:

Path 1: Go to a local university for a 4-year Bachelor's degree in Computer Science/IT. Cost: 5,000,000 IQD per year (20,000,000 IQD total / \~$15,200 USD).

Path 2: Skip the traditional university route, finish my current IT diploma, spend \~$220 USD on the AWS Solutions Architect Associate (SAA-C03) and HashiCorp Terraform Associate certifications, and aggressively apply for junior roles right now.

To give you context, I have skipped the basic tutorial phase and have been building/deploying actual production environments:

• Cloud & Infrastructure: Hands-on experience provisioning AWS EC2, S3, RDS PostgreSQL, Lambda, and DynamoDB.

• CI/CD & Containers: I containerize applications using Docker and automate live deployments to AWS using GitHub Actions pipelines.

• Production Projects: I’ve built a personal portfolio with a serverless backend API, supported a live client website's deployment/DNS/CloudFront infrastructure, and built/deployed a live full-stack e-commerce store secured with Caddy rate-limiting.

• Home Lab: I am currently building a physical bare-metal cluster using two personal PCs to practice separating compute and state for high-availability setups.

The Dilemma:

The 4-year degree is a massive financial and time opportunity cost. I know a university will teach me traditional computer science theory, but it won't teach me modern DevOps, Infrastructure as Code, or orchestration.

If my goal is simply to get a job in the modern tech market (targeting agile local software houses, startups, or remote international roles), will having verified project execution, a physical home lab, and AWS/Terraform certifications allow me to bypass the lack of a Bachelor's degree? Or am I going to hit a brutal HR ceiling later on without that paper?

Would love to hear from hiring managers or anyone who took the certification/portfolio route over a traditional degree. Thanks!

https://redd.it/1uvisng
@r_devops

is Overthewire bandit game good for someone learning devops?

I've just starting to learn the fundamentals of devops and started with linux. A lot of people recommended this game online saying it's good for building command line skills what do you guys think?

https://redd.it/1uvkejc
@r_devops

Are you looking to switch to AI platform engineering roles?

If you are someone looking for transition to AI infra roles from Devops,
this podcast is for you-
https://www.youtube.com/live/82UoYSWaGpk?si=h6qmIfIN4gVZGlnm
Here I discussed tips, projects, interview questions.

https://redd.it/1uw1v3y
@r_devops

Does your org's EDR restrict which Linux distro you can run on your dev laptop?

New work laptop, wanted to switch off Ubuntu to Fedora. Turns out our EDR (Acronis) doesn't support Fedora at all for antimalware/EDR — only Ubuntu, Debian, RHEL-family, and SUSE make the list (Rocky/Alma/Ubuntu 24.04 just got added recently).

Ended up staying on Ubuntu since it's the safest bet either way.

Questions for you all:

1. Does your EDR/security agent limit your distro choice? Which one do you run?
2. Anyone gotten an unsupported distro approved by IT anyway? How'd you make the case?
3. Anyone switched EDR vendors over Linux coverage specifically?

Mainly wondering if this is universal or my org's just strict.

https://redd.it/1uw3438
@r_devops

GitHub Actions OIDC AWS: "Not authorized to perform sts:AssumeRoleWithWebIdentity" even after fixing sub/environment claims, what am I missing?

Been stuck on this for a while. Production CI/CD pipeline, GitHub Actions job needs to assume an AWS IAM role via OIDC to push to ECR. Error:

Error: Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentity

**Setup:**

* Repo: single GitHub repo, workflow triggers on version tags (`v*.*.*`)
* The failing job declares `environment: production` at the job level
* IAM role has OIDC federated trust with [`token.actions.githubusercontent.com`](https://token.actions.githubusercontent.com)
* `permissions: id-token: write` is set at the workflow level

**Current trust policy** (updated after learning that jobs with `environment:` set emit a different `sub` claim shape):

{
"Version": "2012-10-17",
"Statement": \[
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<ACCOUNT\_ID>:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:MY\_ORG/MY\_REPO:environment:production"
},
"StringLike": {
"token.actions.githubusercontent.com:ref": "refs/tags/v\*"
}
}
}
\]
}

**What I've already checked/ruled out:**

* OIDC provider exists in the account, audience is [`sts.amazonaws.com`](https://sts.amazonaws.com)
* The GitHub Environment named `production` genuinely exists under repo Settings > Environments (not just referenced in YAML)
* Role ARN in the `role-to-assume` input matches the role I'm editing (confirmed via `aws iam get-role`)
* Repo name/owner in the `sub` condition is correct, no typos
* Re-ran the same tag after each trust policy edit (not creating new tags each time)

**Still failing with the same error after all of the above.**

https://redd.it/1uw3v8v
@r_devops

Has an AI agent (Copilot, Cursor, etc) ever broken something in your infra/pipeline?

I'm a DevOps engineer, and recently I saw a case where an AI agent "fixed" something in a pipeline that it thought was fine, but it ended up destroying something in the infrastructure. I caught it fast because I know the system well, but it got me thinking about how dangerous this could be for a team without dedicated DevOps.



I'm exploring whether it's worth building a simple, cheap tool that acts as a "safety net" checks/blocks destructive actions (delete, destroy, force changes) made by AI agents on infra/CI-CD before they execute, without needing a complex setup like OPA.

Specific questions:

1. Has something similar happened to you (an AI agent making a risky/wrong change to your infra)?

2. How do you manage this risk today (if at all)?

3. Would you pay for something simple that prevents this (e.g. $20-30/month)?

https://redd.it/1uw7b5l
@r_devops

Control-M vs Airflow?

My team is currently using airflow. It works fine, nothing great. We're evaluating CTM to see if it offers meaningful advantages, and would make sense for us to make the switch (which would take 6+ months...). Welcoming any and all advice, suggestions, etc.

https://redd.it/1uwdd07
@r_devops

Server monitoring (cpu/ram/disk) recommendations please

Hello everyone, so I am currently using NewRelic for half of my setup and grafana (self-hosted) + prometheus for the other half, however I'm looking to move out of grafana and probably use a cloud service, however NR is kinda expensive for my needs; I need to monitor CPU usage, RAM and DISK + any syslog error (as extra), any recommendations/ideas?

https://redd.it/1uwoo76
@r_devops

How do I approach dev ops problems with lack of experience?

I'm a college student with a very narrow knowledge of C/C++, data structures/computer architecture and the more theoretical side of Computer Science. I'm interning at a small sized company this summer and my software engineering role has turned into more of a dev ops role. I may enjoy it, but it's been frustrating to be dropped in a world I (and actually my bosses don't have much experience either) where nothing is familiar. Are there any recommendations for a crash course about development pipelines/ infra that is recommended? My dm's are also open if I could talk through my struggles with someone experienced.

https://redd.it/1uwugip
@r_devops

What enterprise backup solution are you using for Azure VMs and Blob Storage?

We're taking another look at our Azure backup strategy and are curious how other teams are handling it

Azure Backup does a good job for many use cases, but we're also thinking about scenarios where we'd want copies of our data outside the same Azure environment. That could mean another tenant, another cloud, or a dedicated backup platform to improve recovery options.

Our environment is primarily Azure VMs, Azure Blob Storage, Azure SQ databases, and a few terabytes of data overall.

For those running production workloads, what approach has worked best for you?

Did you stick with Azure-native services, maintain backups in a separate tenant, use a third-party platform, or take a different approach altogether?

Looking back, were there any restore or disaster recovery lessons that changed how you designed your backup strategy?

https://redd.it/1uwx5nc
@r_devops

People who bought a cloud cost optimization tool: did the savings number they promised actually show up on your bill?

Disclosure up front: I'm a founder of a startup in this exact space, so read everything below with that bias in mind. Not naming or pitching the product, that's not what this is.

Here's why I'm asking. Before a deal, every vendor in this category (mine included) shows you a big number. "We found 40% waste." And I've started noticing the games that can hide inside that number. Identified savings quietly presented as if they were realized savings. Savings from work your own team did getting counted in the tool's total. One-time cleanups counted every month forever. Baselines cherry-picked from your most wasteful week.

So I'm curious about the other side of the table. For those of you who actually bought one of these tools:

Did the promised number materialize on the actual invoice? How did you verify it, or could you even? Did anyone here catch a vendor inflating? And for those where it worked out, what did the vendor do differently that made you trust the math?

Trying to figure out what proof would actually convince a skeptical platform team, because "trust our dashboard" clearly isn't it.

https://redd.it/1uwyi5o
@r_devops

Google Maps for Clusters
https://redd.it/1ux1leq
@r_devops

I'm tired of bolting RBAC and Audit logs onto our custom ops scripts. How do you safely execute these?

We have the standard graveyard of custom bash, Python and even Groovy scripts for atomic ops tasks (draining nodes, killing stuck pods, db failovers, etc.).

While standard "community playbooks" are useful for the boilerplate stuff, we inevitably end up needing to run our own custom logic out of our own repos too.

But the actual friction is the execution boundary. Trying to enforce "who is allowed to run this script in prod at 3am" (RBAC), requiring an approval before execution, and getting a clean, immutable audit log of the outputs is driving me crazy. Handing devs raw kubeconfig/SSH access is a minefield.

Because you basically have to own the runtime to actually enforce this governance, how did you solve it? Did you write a custom internal CLI/runner that handles the auth + audit and then invokes the script? Or did you end up wiring something heavy (like AAP or Temporal) purely to get that secure execution boundary?

https://redd.it/1ux21mx
@r_devops