How does your team track everything that's about to be retired in your cloud?

Infra engineer here (about 10 years across AWS, Azure and on-prem). In every environment I've worked in, something is always approaching end of life: AI model versions on Azure OpenAI and Bedrock, Kubernetes versions on EKS/AKS/GKE, Lambda and Functions runtimes, database engine versions.

Each cloud sends its own notices, but they tend to land with account or subscription owners, not the team that owns the workload. And unlike library versions, none of this shows up in Renovate or Dependabot.

Curious how your team handles it:

\- How do you find out? Native notices (Service Health, AWS Health, Advisor), vendor emails, a script, or someone just happens to notice?

\- How does the notice get to the team that owns the workload?

\- Do you catch any of it in IaC or CI, like pinned versions in Terraform or a pipeline check?

\- What's your worst "we missed it" story?

\- If you use a tool or spreadsheet for this, what's annoying about it?

Full disclosure: I've built a few small tools around cloud AI model and Kubernetes lifecycle data, and I'm exploring whether something for this specific problem is worth building. Mostly I want to know if it's a real pain or already handled well. I'll share a summary of what I learn.

https://redd.it/1x8ubmw
@r_devops

Hosting my own multiplayer game

Hi all! I wanted to share a project I've been working on.
In my free time I've been learning how servers work, especially how multiplayer games are handled on the server side. I built a small game and now I'd love for people to play it and push it as hard as they can. The more players and chaos, the better: I want real world logs to see how it holds up under load.
If you end up enjoying the game, let me know! I find it surprisingly fun myself, so I'm considering expanding it and improving the visuals down the line. That's not the main goal for now, though.

The game: https://airdogz.hobolichi.com

https://redd.it/1x8w1k0
@r_devops

Thinking about moving from backend Go to SRE/DevOps

I’ve been working in software for about 15 years, mostly backend, and I specialize in Go. I also have around 3 years of DevOps/SRE experience, though that was about five years ago and in a small company.

Lately, my backend role is almost entirely prompting and reviewing AI-generated code. At my company, even people who aren’t engineers can open PRs.. Management thinks that "help" us, but thanks no thanks, 1k+LOC PR is not helping anyone... They want us to build some magical "harness" that will produce exactly the feature you want with all good practices embedded.
I worry that my job is heading toward purely supervising AI-driven workflows rather than writing and designing software.

I’ve been considering moving back toward SRE, platform, or DevOps work. Perhaps it's naive, but I think it's less exposed to the AI frienzy, as it's not directly part of product. I get that a lot of things get automated (incident response, dashboard) but that's kind of in the spirit of devops ?
I see it as more job secure and more interesting in this day and age, as you'll always need people to run your infra for ai workflows (right anakin ?)

Also part of the appeal is that these roles seem more available where I live, while remote backend jobs have been hard to find. Devops skills can be reused in a lot of company of various sizes.

Question for you: how has AI changed your day-to-day work? Is it making the work more interesting, or just shifting more of it toward reviewing generated configs and automation? How do you see it evolving or enduring ?

https://redd.it/1x8ygo6
@r_devops

Should I teach myself the AWS CLI or is the time for that is gone?

I'm setting up an AWS workflow (we work only with azure and on-prem at work so little experience with the CLI) and wondering if I should bother learning about the tool itself.

Seems like I should just set up limited users with limited roles, hard budget caps, place manual gates on production CIs and let a sandboxed AWS MCP handle everything else. Using Terraform is an implicit admission on how we'd all like to instrument our architecture in a vendor-agnostic vocabulary. An MCP seems to be an additional step in the same direction?

Thoughts?

https://redd.it/1x9aq1z
@r_devops

Your AI coding agent might be cheating to pass CI. Here's how to catch it!

I had a small bug in a bill-splitting app and ran the same fix task 10 times across three coding agents. Every single run changed the test instead of fixing the code.

The failing test, encodes a product requirement => split 100 cents three ways and every receipt shows 33, even though 1 cent goes unaccounted for.

The agents treated it as a bug and rewrote the test to match their code 😆

What I learned: an agent's "tests pass" is worthless if it's allowed to edit the tests. You have to check what it changed, not just whether it finished green.

So I built Skill Gate to catch issues like this in our Cursor or claude skills so you can evaluate them before sharing with the rest of your team to speed up their programming workflows.

You give it a skill and a few test cases, it runs the agent on them, and a decision model checks what the agent did, like "were all files under tests/ left unchanged?", not just the exit code.

Limitations (for now): you bring your own API keys and write your own test cases.

Would love to hear any thoughts or feedback from the community here!

https://www.skill-gate-ai.app/blog/ai-coding-agent-cheating-ci

https://redd.it/1x9ipmw
@r_devops

Whats’s your experience with SST.dev or Pulumi?

I’m working on a personal project, so 0 users, and I have worked with SST in the past, but having to upgrade SST was a pain. Not sure if pulumi is any different of an experience, or is less painful and wanted insight on different experiences with either tool

https://redd.it/1x9js9y
@r_devops

How to monitor Supabase using an OTEL Collector?
https://telflo.com/blog/monitoring-supabase-with-the-opentelemetry-collector

https://redd.it/1x9kck3
@r_devops

Weekly Self Promotion Thread

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

https://redd.it/1x9pgb3
@r_devops

To monorepo or not?

My startup is in consumer electronics. The product is 1 device with 1-2 peripherals that come with it. Users connect to a mobile app with a backend that does data analysis on what the product measures.

ATM I have 4 repos: a flutter app and a FastAPI backend with sqlite. The embedded side uses PlatformIO+Arduino for the device/hub, and same stack for the peripehrals.

The DevOps is getting unwieldy. On the one side, I'd like everything to be in one place so agents can easily build up a context and CI/CD is not scattered. On the other hand, I hate working with massive project trees and loading up vscode would be a nightmare.

Thoughts? At work I use git submoduels but I was told that's not a monorepo.

https://redd.it/1x9q2cd
@r_devops

optscale-kompressor: open-source proxy that compresses coding-agent tool output (−22.7% on real API bills)

https://preview.redd.it/vulcz3l6cdwh1.png?width=2540&format=png&auto=webp&s=09f90c56a3d90ce80a553eca942b12a250513a7b

Hello everyone! We open-sourced optscale-kompressor (MIT, single Rust binary).
If your team runs coding agents on API keys, a chunk of that bill is the agent re-sending the same apt-get output and compiler warnings every turn. In our traces that was about half the prompt. The compressor sits between the agent and the provider and strips the repetition, keeping errors and anything the agent can't reconstruct, and it can fetch back whatever it removed.
Everything is rules, nothing is learned, so the same output always compresses the same way. That matters more than it sounds: break the byte-for-byte prefix and you lose the prompt cache discount, which is how these compressors usually end up costing more than they save.
Real numbers off actual API bills, not token math: 22.7% cheaper on Terminal-Bench with Codex, solve rate unchanged. Works with Codex, Aider and opencode. Nothing phones home. https://github.com/hystax/optscale-kompressor — a star helps if it's useful to you.

https://redd.it/1x9pitg
@r_devops

Starting a Team

looking for people to form a team. could be any position in tech.

https://redd.it/1x9q44s
@r_devops

What are your thoughts on people using AI to converse?

So we have this infra lead who pretty much uses ChatGPT to write up their replies to Slack messages. You can clearly see the use of AI in how their replies are structured. Because of this, sometimes, due to their AI’s lack of context, certain replies do not have the depth required for their level.

When I say use of AI, here’s some examples:

- The current 1TB threshold sits right at our normal free space, so the daily 50GB temp file dip trip it. Adding 200GB would just delay the same problem as data grows, adds cost, and RDS storage can’t be shrunk later.

- Good point on slot lag. 850/800 is a good move.

- Nice approach, much cleaner than maintaining Stripe IP allow lists. Once you confirm it’s working I’ll factor it into the billing service step for EKS: it’ll need a pod identity role with read access to SQS.

This is just from today.

https://redd.it/1x9umrb
@r_devops

Jenkins Alternative for Projects with Mixed VCS

I run a small game studio (~10 years, been in industry for 15, just to note background) and am looking at modernizing our build machine. Previously, we used Jenkins as it was something I was familiar with, but setting it up and maintaining new projects has always been more laborious than preferred. I have used it's older build style heavily, relying on command line scripts, as well as it's pipeline workflow which took much more effort to set up, though I liked it's flexibility and stage reporting.

Our specific challenges are that we primarily do service work, so we are often in a mix of version control systems across hosts: git (github, gitlab, gitea, butbucket, etc), Unity Version Control (previously PlasticSCM and what we use internally for projects), and once in a while, SVN. Jenkins was fairly VCS agnostic, which allowed for "quick" setups across systems. Projects are not hosted locally.

Our previous builder was spun down a few years ago and we haven't replaced it. Our deliveries are now built on dev machines (usually my own) using build scripts for consistency. Most project builds are relatively quick but I'd like to get things offloaded and automated once more to a dedicated build machine. The spinning down of the old machine was partly due to a security incident that affected another team using Jenkins that had me reconsider the risks of using it.

The system needs to be self hosted with local runners as we mostly build for closed platforms and our preference is to keep everything internal, when possible. The target platforms primarily require Windows machines for building and local access to development hardware. It's uncommon that we work on mobile projects but we have local mac mini machines for that when necessary. Builds and build artifacts can be quite large, so self hosting also lets us keep costs down by leveraging our local NAS setup.

We don't really need webhooks to the VCS for our setup; manually triggering builds from a web UI is all the team really needs (it allows non-devs to get development builds for testing on hardware) and the occasional cron job for weeklies. On the other side, pushing notifications when builds are complete along with their status is important. We use Discord for internal communication which is often overlooked by other CI tools that prioritize Teams and Slack.

I've read through a lot of older threads about Jenkins alternatives and it seems that most suggestions involve using the CI included with the host, which isn't entirely possible with our needs. I would like to use something secure (and therefore less dependent on community plugins than Jenkins is) that lets the VCS hookup be swapped out in some simple manner per project.

The conclusions I have reached at this point is that there really isn't a good, free (or cheap) solution outside of Jenkins as teams move away from dedicated build software to full software platforms that provide end-to-end services for hosting, project management, and CI/CD. I would very much like to be wrong about this and have somebody inform me of a modern FOSS Jenkins replacement out there.

https://redd.it/1wydwfj
@r_devops

J.Crew’s homepage is currently just a JSON health check for their "generative AI backend"
https://redd.it/1xa2353
@r_devops

Is anyone using Endor Labs products?

Doing a poc and wanted to know about the product from someone who is actually using it in production. How has your experience been and how effective it is in filtering security violations?

https://redd.it/1xa2e3q
@r_devops

main skill u need for ur server stack?

The main skills knowing what services u need to run, not just picking Nginx or Apache, like I’m on BeAdmin rn, ran Webmin before. What changed ur setups u guys setup?

https://redd.it/1xah47j
@r_devops

Sonatype Nexus now requires mandatory telemetry gathering
https://help.sonatype.com/en/nexus-repository-telemetry.html

https://redd.it/1xaremm
@r_devops

Kubernetes probes keep restarting services that just need more time

Hey folks, I've noticed readiness and liveness probes often end up with pretty similar settings, even tho they do different jobs. With tight timeouts and failure thresholds, a service that's slow to start or recovering from a dependency issue can get pulled out of rotation or repeatedly restarted.

I've seen probe settings get copied between services too, even when their startup times are completely different.

How do you guys set probe thresholds for services with unpredictable startup times or dependencies that occasionally respond slowly?

https://redd.it/1xatqcs
@r_devops

Homelab runner for Gitea

Hello guys,

I'm new to the devops world and I'm working on some projects to learn. I'm currently trying to automate the creation of VMs with Terraform with my local Proxmox.

As CI/CD and Git server I have everything on a Gitea local. I remember doing some CI/CD once on Github and it worked fine but apparently I need a runner in my home.

Should I create a VM or a Docker container? I have no idea on how to setup that thing and to connect it to my Gitea right after.

Thanks for the help

https://redd.it/1xatqay
@r_devops

Interviewing for a Software Support Engineer role (Linux, Python, Bash, CI/CD) — what technical questions should I prepare for?

Hey everyone,

I have a 30-minute interview coming up for a Software Support Engineer position at a defense contractor, and I'm trying to prepare as thoroughly as possible.

The recruiter mentioned there won't be a formal technical assessment, but I still want to be ready for any technical questions or troubleshooting scenarios.

For background, I currently work in cloud/data center infrastructure operations, where I use Linux, Python, and Bash for diagnostics, troubleshooting, and automation. I also have experience developing software applications, working with REST APIs, automated testing, and CI/CD.

Here's the relevant job description:

Responsibilities:

Support lab tooling and version control through CI/CD pipelines.
Create and deploy test scripts.
Support and troubleshoot software deployments on custom servers.
Support configuration management processes.
Develop unit tests and tailor rule sets.

Required qualifications:

Bachelor's degree in CS, Computer Engineering, or a related technical field.
2+ years of relevant experience.
Experience with Python and Bash.
Ability to independently and collaboratively solve complex technical problems.

Preferred qualifications:

Knowledge of CI/CD pipelines.
Experience with GitLab.
Experience with Linux and NFTables.

I'm particularly interested in moving toward DevOps/platform engineering, so this seems like a great opportunity.

For anyone working in DevOps, Linux administration, software integration, or similar roles:

1. What technical questions would you expect in an interview for this position?
2. What Linux troubleshooting scenarios should I be prepared to walk through?
3. How deeply should I study GitLab CI/CD, unit testing, and NFTables?
4. Does this sound more like DevOps, software integration, or traditional software support?
5. If you were interviewing a candidate for this role, what would you want them to demonstrate?

I'm not looking for shortcuts or leaked interview questions. I genuinely want to understand the technical expectations and make sure I can speak confidently about the work.

Appreciate any advice!

https://redd.it/1xaxlif
@r_devops