At what point do you stop treating timeouts as normal?

Hey guys! we've got a couple of internal services that will occasionally throw timeout errors under load. They retry, recover, and users never seem to notice.

The annoying part is that it's been happening long enough that nobody really reacts to it anymore. Every incident review ends up with the same general response that it's just something we've always seen.

Now I'm wondering if thats just the reality of distributed systems, or if we've gotten too comfortable ignoring something that probly deserved a closer look a while ago.

How do you decide when intermittent timeouts have crossed the line from something you live with to something worth digging into?

https://redd.it/1v3gelr
@r_devops

Looking for Terraform + AWS hands-on project ideas after completing Terraform Associate and AWS SAA

Hi everyone,

I recently completed:

HashiCorp Certified: Terraform Associate (003)
AWS Certified Solutions Architect – Associate (SAA-C03)

Now I want to move beyond certifications and gain more hands-on practical experience with Terraform on AWS.

I understand the Terraform concepts and AWS services at a theoretical level, but I want to build real-world projects that would help me improve my skills and also create some strong portfolio projects for my resume.

I am looking for recommendations on projects that simulate what a Cloud Engineer / DevOps Engineer would actually work on in an organization.


My goal is not just to deploy resources but to understand how Terraform is structured and used in real enterprise environments.

Would appreciate suggestions on:

Which projects are most valuable for learning Terraform + AWS together?
Any GitHub repositories, courses, or labs you recommend?
What kind of Terraform projects stand out on a Cloud/DevOps resume?

Thanks in advance!

https://redd.it/1v454t7
@r_devops

Which DevOps tool looked amazing during the PoC but became painful after six months?

I am Little confused ?

https://redd.it/1v445k8
@r_devops

Moving a high-memory Python application from Kubernetes to a dedicated Linux server — what would be the best setup?

Hello,

At the company where I work, we have a Python application running on a Kubernetes cluster. The application is fully integrated into a CI/CD workflow. Whenever code is merged, Jenkins pulls the source code from GitLab, builds the application, and pushes the image to Harbor. Argo CD then deploys it to the Kubernetes cluster.

However, the application consumes a large amount of RAM, and I have had to increase its memory allocation several times. We have now decided to move the application to a separate Linux server.

The application runs with Uvicorn, and I plan to make it accessible through a domain name, just as it currently is in the Kubernetes environment. However, I am not sure what the best-practice architecture would be for this setup, so I would appreciate your recommendations.

I still want Jenkins to automatically pull the application from GitLab, build it, and deploy it. After deployment, the application should automatically start and run as a web service.

My main concerns are how to restore the application as quickly as possible if a problem occurs, how to handle backups, and what kind of deployment and recovery strategy would be best.

In short, could you please advise me on the best way to design and implement this setup?

https://redd.it/1v475m2
@r_devops

How are developers and operations meant to work in this situation?

Here’s our situation - a mature operations team with pipelines building container images for a third party product going into kube

Company acquired a company with a development team who had no ops but are proficient at producing dockerfiles for their products

Ops is in azure devops, developers in bitbucket

We want to leverage the developer teams skills to get the apps into ops docker images

Some suggest that ops could build base images and developers could build their apps with that as the base - but what if the base images need patching? That means ops depends on developers for that?

Another thought I had was developers keep Dockerfiles in devops, the containers build from sources (aka git clone bitbucket) and build the project all in the ops pipeline

Would love to hear how others do this…

https://redd.it/1v480u5
@r_devops

Terraform/Github deployment overwriting another deployment

I'm on a team that shares a Github repository. Whenever we open a PR from a feature branch to the dev branch, a deployment to the AWS development account is automatically triggered.

The issue we're facing is that one developer may deploy to dev, and then another developer deploys afterward. Even though they're working on different files, the second deployment ends up overwriting the first developer's change in AWS.

How can we prevent this?

We're following a Gitflow workforce (feature -> dev -> release -> main), and our biggest challenge right now is that the second developer's code is often "outdated" when it's deployed, causing it to overwrite changes that were already deployed by someone else.

We tried merging everything to dev, but when it's time to deploy to prd, the dev branch ends up filled with a lot of unnecessary changes.

We using Github Actions + Terraform.

https://redd.it/1v4np15
@r_devops

AWS says its DevOps Agent delivers 75% lower MTTR and 94% root cause accuracy

Straight from the GA blog: customers report "up to 75% lower MTTR, 80% faster investigations, and 94% root cause accuracy, enabling 3-5x faster incident resolution."

https://aws.amazon.com/blogs/mt/announcing-general-availability-of-aws-devops-agent/



GA since March. I haven't seen anyone outside AWS's own blog actually use it. Did you try?

https://redd.it/1v4nqwx
@r_devops

Are these the 10 best practices for DevOps as a Solo developer?

Hi r/devops!

The ten:

1. Secure base images by default (cgr, rapidfort, etc) — `cgr.dev/chainguard/*`, \~0 CVEs
2. Multi-stage builds, non-root user — build tools never ship
3. Pin everything — lockfiles, image digests, actions by commit SHA
4. Least-privilege CI tokens — explicit permissions: contents: read
5. No secrets in code, images, or logs — OIDC over long-lived keys
6. Health checks + rollback-ready deploys — /healthz, one-step revert
7. Plan before apply — read the diff; never hand-edit live infra
8. Structured logs with levels — JSON events, not print("here 2")
9. Fail fast, cache hard in CI — lint before build before test
10. Small, frequent, reversible releases — boring deploys are the goal

DevOps is hard. It used to frustrate me, but the tools are making it easier.

Cheers!

https://redd.it/1v4vkg0
@r_devops

Why do coding agents behave differently in scheduled runs than interactive runs?

One thing that feels under-discussed with coding agents is the gap between interactive success and scheduled reliability.

In an interactive run, a lot of assumptions are quietly true: the user is present, credentials are warm, the working directory is right, logs are visible, and a bad step can be stopped.

I have had small automations work perfectly when I ran them by hand, then fail in the scheduled version because the path, auth, or environment was slightly different.

For a coding agent, that failure mode feels worse because it may keep reasoning, retrying, or making changes while the real problem is just the runner state.

Would you treat scheduled coding-agent runs more like CI jobs, with strict preflight and timeouts? Or is that too heavy for personal/dev workflows?

https://redd.it/1v4xotv
@r_devops

How do I learn how to architect a system?

Yes I know the question is extremely broad and complex. I'm simply looking for a starting point (or middle point).

I've been working on designing an SPA using AWS services for an NPO for the past few months. I've realized that the system works but I'm not happy with it. I know it can be better (more organized, a more sophisticated and automated system). I often find myself asking questions like: should these be a microservices architecture? Should it be a event-driven? some sort of layered architecture? How should the tests be? Is this the best programming language to use?

I know quite a few of those questions and doubts are best to be left to "whatever ur comfortable with". But I like to get into the nitty gritty. Are there certain books/youtube videos/etc that I can look into to learn about the basics of system design and architecture up to an intermediate or expert level? The only one I know of is Hello Interview.

https://redd.it/1v53dyz
@r_devops

How to tackle 90days NP.

I have been working at TCS for the past 4.7 years as an AWS DevOps Engineer. I have been trying to switch jobs for the last four months.

I receive calls through Naukri only when I update my notice period to 15 days or less. Most of the calls are not from MNCs. Even when I do receive calls from MNCs that generally accept a 90-day notice period, they still prefer immediate joiners.

I don't have the confidence to resign before securing another offer because I have a lot of financial and personal responsibilities.

I requested my manager to release me from the current project, but the request was declined. Ironically, there is very little work for me. Most days, I simply swipe in, swipe out, and spend the day with almost nothing to do.

Given this situation, I'm confused about my next step. I would really appreciate your suggestions.

https://redd.it/1v54wgy
@r_devops

How do you catch the stuff you never thought to put an alarm on?

so everyone talks about tuning alarms and reducing false positives but honestly the scarier problem for us has been the opposite. stuff breaking that we just never set up monitoring for in the first place.

had this happen a few weeks ago. a queue consumer quietly stopped processing overnight, nobody had an alarm on messages piling up in that specific queue because it's like a secondary internal thing, not customer facing directly. found out the next morning when downstream stuff started failing and someone finally went looking. by then there was hours of backlog to clear

and it's not like we were being lazy, we've got a decent number of alarms set up. it's just impossible to think of every metric on every resource ahead of time, especially as stuff gets added and nobody goes back and audits what's covered

so how are people actually dealing with the "unknown unknowns" side of this. do you just accept some stuff will slip through until it's bad enough someone notices manually. anyone actually using baseline/anomaly based detection that catches things without you having to pre-configure a threshold for every single metric, and does it actually work or is it just a different kind of false positive machine

genuinely asking, not trying to set up alarms on literally everything because that just becomes its own nightmare to maintain

https://redd.it/1v569m7
@r_devops

🚨 Never use Hostinger, worst experience ever 👎

We purchased the KVM 8 plan with a one year subscription, which includes 8 vCPUs, 32 GB RAM, and 400 GB of storage. On paper, it offers strong specifications at a competitive price. However, after several months of use, we began encountering an issue that significantly affected day to day operations.

We experienced recurring Connection Timed Out errors when accessing hosted websites or connecting via SSH. In many cases, the issue was resolved immediately by using a VPN or by obtaining a different public IP address after restarting the router.

Because residential internet providers routinely assign dynamic IP addresses, customers generally have no control over the reputation or previous history associated with the IP they receive. As a result, connectivity problems based on IP reputation can unexpectedly affect legitimate users.

Over the following months, we contacted Hostinger support on multiple occasions to request that our IP address be unblocked. Each request was resolved manually and access was restored. However, we were never provided with logs or a technical explanation describing why the IP had been blocked in the first place.

The absence of diagnostic information made it difficult to identify the underlying cause or prevent the issue from recurring. By comparison, providers such as Cloudflare offer detailed security logs that show blocked requests, security challenges, and rule matches, enabling customers to troubleshoot issues more effectively. While every provider implements security differently, a similar level of visibility would have been helpful in this situation.

Hostinger's billing model also meant that the service could not be easily discontinued once the initial refund period had passed. As a result, although the subscription remains active, I have limited its use to testing and non critical workloads rather than production services.

https://redd.it/1v56g1h
@r_devops

Advice for a struggling team lead

I'm a de-facto team/tech lead (but not officially appointed by our managers) with several years of experience. Most of the time, I'm feeling pretty confident in my ability to guide the team members in the right direction, ensuring critical tasks are identified and get done, ensuring our product is in a healthy and sustainable state, day to day stuff is frictionless, etc. However, I lack guidance and mentorship from above. It's like the management layer above me is non-existent and I'm left to my own devices as they focus on shinier things. That's fine, as long as they're happy and my product works. The problem is that I don't really get feedback from anybody in terms of my product/team leadership. I feel like I'm just struggling to keep things going in roughly the same way as before, when the team was run by the previous lead. I'm being assured by my manager that I'm doing a great job and appreciated, but I somehow don't really believe it, because I'm there and I see room for improvement.

I feel like all I'm really good at is getting the team to maintain status quo. I don't know whether that's optimal or how I'd change it if it wasn't ( or even if I was supposed to). Yes, it's largely due to not being in our director's spotlight and me not receiving mentorship anymore. And yes, I'm unfortunately in a position of being subconsciously expected to lead without being appointed to do so officially.

So let's make the best if it - can you recommend any kind of team lead training courses for me to take? How to handle emergencies? How to do things sustainably? How and when to communicate assertively? When to change the process you suspect doesn't work? How to empower team members? How to delegate properly? How and when to project authority and when to let things slide? That kind of stuff. Or any other thoughts and advice you can offer.

https://redd.it/1v58cb9
@r_devops

What is the long term plan for charts supporting Gateway API?

We manage all of our infra apps like Vault, Argo Workflows and many others using Terraform + Helm. TF creates the ArgoCD ApplicationSet which points to the chart and values managed in the repo, and ArgoCD syncs everything.

During the transition to GW API I noticed that most charts either don't support GW API natively, or only support creating an HTTPRoute. With ingress, all we had to do was define a single ingress.enabled: true block with some annotations, paths, etc.

Now with GW API, you usually can't manage everything from the app's values file. Even when the chart creates an HTTPRoute, it doesn't create the other resources we need, such as a ReferenceGrant for cross-namespace references or controller-specific resources like Envoy Gateway's SecurityPolicy and BackendTrafficPolicy. Argo Workflows is one example where the chart supports creating an HTTPRoute, but not the rest of that setup.

We have an okay solution for this today: we just create a manifests/ folder for each app and add it as another source in the ArgoCD AppSet (applied by TF, and everything is as GitOps as it gets). In that manifests folder I create all the needed Gateway API and Envoy Gateway resources in a single multi-manifest YAML file.

What do you guys do for 3rd party charts? What is the long term plan for apps in terms of Gateway API support? Is the expectation that charts will eventually expose more Gateway API values, or that routing and policies should live separately from the app chart?

I'm just curious since this whole GW API thing is very new, and I assume a lot of apps will need to add "ingress-like" support at some point?

https://redd.it/1v57q3k
@r_devops

How do devops / engineering services share documentation with clients?

Hello,

We provide devops / engineering services and use GitHub for documentation (runbooks, configs, processes). Clients need access, but raw GitHub feels too technical.

* How do you share documentation with clients in devops / engineering services?
* Do you use GitHub, wikis, or dedicated portals?
* How do you handle versioning and access control?

https://redd.it/1v4s0yd
@r_devops

The difference between safety filters and actual model quality

Just wrapped up a rollout and felt like a bit of a reality check because I think we totally conflated safety with actual quality.

We spent way too much time on the safety layer. Runtime filters, moderation, and injection blocks (the whole 9 yards). I think on paper, we were looking solid. But in practice, the model was hallucinating just as much as it would have without those checks in place. It was giving these super confident, polite answers that had absolutely nothing to do with what the user actually asked for. But, and this is a big but, because the responses were safe, they sailed right through every guardrail we spent so much time on.

Maybe this is obvious, and I just didn't take the time to think about it. Guardrails are great for catching the obvious bad stuff but they do nothing for that middle ground where the user experience actually lives. You can be perfectly safe and still be completely useless to a customer. Duh.

We had to overall our moderating to actually start building a dataset to catch these polite hallucinations. We are now running regression tests every time we touch a prompt.

https://redd.it/1v5pmab
@r_devops

Looking for feedback on my open-source production-grade AWS EKS platform architecture
https://github.com/Saif-hadd/terraform-aws-production-infrastructure

https://redd.it/1v5p19e
@r_devops

Experience with IBM LinuxOne

Hello! My company is considering buying a LinuxOne machine to run Mongo. Has anyone actually worked with one? Any challenges besides the fact that it only supports s390 arch?

https://redd.it/1v5tg98
@r_devops

If a policy changes mid-run, should the worker fail closed?

I keep running into a boring automation question that becomes more annoying with coding agents.

A task is running. I change a network allowlist, proxy, or sandbox rule. The UI says the setting is saved. But unless the worker confirms a reload, I do not actually know which policy snapshot is active.

In one small local setup, I changed a proxy and allowlist while an agent task was running, then realized I could not tell whether the active run had picked up the new rules or still used the old snapshot.

For normal automation, I would expect some mix of config version, reload event, and audit log. For agents, I think it matters even more because the task can keep making decisions while I assume the boundary changed.

If a policy becomes stricter during a run, would you pause/fail closed, reload at the next safe point, or allow the current run to finish under the old snapshot?

https://redd.it/1v5uytg
@r_devops

Terraform Question I am asked almost everytime

You're working in production.

Your Terraform backend is:

S3 for state
DynamoDB for locking

A developer accidentally deletes the DynamoDB lock table.

At the same time, two engineers run terraform apply.

Thirty minutes later, the infrastructure is in an inconsistent state.

# Questions

1. What exactly happened internally?
2. What kind of corruption could occur?
3. How would you recover without making things worse?
4. Would you restore the state file? Import resources? Edit the state manually? Explain your reasoning.
5. What preventive controls would you implement so this never happens again?

https://redd.it/1v637pl
@r_devops