From DevOps to SaaS: How Do You Find Your First Paying Customers?
https://redd.it/1uowi5u
@r_devops

5 YOE in DevOps/SRE at a Tier-1 bank, pivoted after a layoff — now in a support-heavy role. How do I course-correct before it hurts my career?

Hey r/devops — not looking for validation here, looking for people who’ve actually been through this.
The background:
Spent 5 years as a DevOps/SRE engineer at a large global investment bank. Real scope — owned incident response, SLO/SLI definition, Python automation, Kubernetes deployments, Grafana/InfluxDB observability stacks, CI/CD pipelines. High-stakes production trading systems. The kind of work I was proud of.
Earlier this year, my team was restructured. Whole group gone. I was laid off.
The decision I made:
The market was rough. I had a contract offer on the table — onsite at another major financial institution, compensation was actually better than my previous role. I took it. Practically speaking, it was the right call at the time.
What I underestimated: the actual scope of the work. It’s L1-L2 support. Ticket resolution, runbook execution, escalations. I’m not owning anything. I’m not building anything. And every month I stay here, the gap between my resume and my current reps gets wider.
I’m not bitter about the pay — it’s kept me stable. But I know this lane leads nowhere if I stay in it too long.

Three things I genuinely need help thinking through:

1. How do I keep building real SRE depth when my day job isn’t giving me the reps?
I have a solid foundation — Docker, Kubernetes, Terraform, Ansible, Python, Grafana. But I want to go deeper on observability, chaos engineering, capacity planning, and SRE fundamentals. What projects, home labs, or open-source contributions have actually made you better — not just resume filler?

2. Is the AWS Generative AI Professional cert (AIP-C01) worth pursuing for an SRE career path?
I’m currently working toward it, specifically because I’m interested in AIOps — anomaly detection, automated remediation, predictive reliability. Is this a real differentiator for senior SRE roles in 2025-26, or am I chasing the wrong thing? What certs have actually moved the needle for you?

3. How do I frame a support-heavy role in interviews without it undercutting 6 years of solid experience?
My instinct is to move within 6–9 months before the narrative gets harder to control. Is that the right timeline? And how do you position this kind of pivot when you’re targeting L4/L5 SRE or Senior Platform Engineer roles?

Context: Based in Bengaluru. Targeting product companies, global banks, and open to remote/international opportunities.
I know I made a compromise. I’m not here to relitigate that. Just want to make sure the next move is the right one.
Appreciate any honest perspectives — especially from people who’ve navigated something similar. 🙏

https://redd.it/1uoydmi
@r_devops

I'm starting a new movement

I am officially declaring the start (in my mind) of #MRBA

That stands for "Make Releases Boring Again"

This was prompted by a Release Engineer job posting that was your usual "just be on 24/7 on every communication channel during release windows". So every few months, you over activate my nervous system and it takes until the next release for it to finally calm down only to be activated again? No thanks.

I need to be doing automation, environment config hardening, observability tweaking. Not "monitoring Slack in case someone reports an issue". 😒

Releases need to be boring. The more boring, the more both dev AND ops sleep. With the added bonus of not over-rewarding heroics. 😏

Release day hype/fanfare/stress is for shit like clothing, games, etc. Not the newest feature for your internal app with 10 users.

https://redd.it/1up28y5
@r_devops

I built a single-binary TUI that manages Redis, Postgres, SSH, Docker, Git, S3, MySQL, MongoDB, and HTTP — with a built-in MCP server for AI tooling

**Qore** is a single-binary infrastructure orchestrator with a terminal-native UI. You type commands, get results inline — no context switching between redis-cli, psql, ssh, and docker. What makes it different: **8 connection types in one place:**

* Redis (native RESP protocol — no redis-cli needed)
* PostgreSQL / MySQL / MongoDB (full SQL queries, EXPLAIN, slow queries, CSV export)
* S3-compatible (AWS SigV4 — works with MinIO, R2, AWS)
* HTTP/REST (GET, POST, PUT, PATCH, DELETE with auth)
* SSH (exec, SFTP, systemd, Docker Compose, deploy scripts, interactive shell)
* Git (branch graph, merge, rebase, cherry-pick, blame, tags)

**Built-in MCP server:** This is the part I'm most excited about. It exposes 35 tools (SSH, Docker, database queries, system discovery, HTTP) over JSON-RPC 2.0 — so Claude, Cursor, or any MCP-compatible AI can interact with your infrastructure using connection names only. Credentials stay server-side. **Other highlights:**

* Secure vault: AES-256-GCM + scrypt, master password never touches disk
* Docker via Unix socket (no docker CLI dependency)
* Multi-tab: all connections stay mounted, switch with Ctrl+Tab
* Multi-service dashboard with auto-refresh
* Health checks with latency sparklines
* Self-updating (`qore update`)
* Single binary, \~45MB, Linux/macOS/Windows

Install: `curl -fsSL` [`https://github.com/Kodjaoglanian/qore/releases/latest/download/install.sh`](https://github.com/Kodjaoglanian/qore/releases/latest/download/install.sh) `| bash` Code: [https://github.com/Kodjaoglanian/qore](https://github.com/Kodjaoglanian/qore)


Happy to answer questions!

https://redd.it/1up1rh3
@r_devops

Need guidance !

I have 3 yrs of experience as a cloud engineer/devops worked in same company from start got one promotion and good hike all along . Want to switch to now and thinking on doing CKA certification is it worth having ?

https://redd.it/1up4s58
@r_devops

How are you managing dynamic runtime configuration without triggering a full CI/CD deployment?

Hi all!

I’m currently digging into the "toil" of our (the company i work at's) release process, and I’m hitting a recurring bottleneck: rate-limiting configuration.

Right now, we have our limits (e.g., token buckets, thresholds) defined as part of our static config, which is baked into our container images. Whenever we need to tune these for a traffic spike or emergency throttle, it forces a full CI/CD deployment aka build, push, wait for rollout, and pray the new pod doesn't have a startup issue.

It feels fundamentally wrong to bounce a production service just to change a numerical threshold.

I’m looking into moving this "knob-turning" out of the deployment pipeline and into a centralized, runtime-synced store (like Redis), so we can tweak values on the fly without a code push.



Is anyone else using a "Config-as-a-Service" or dynamic sidecar pattern for this, or have we missed a super obvious solution lol thanks guys :)

https://redd.it/1up6pnr
@r_devops

Odd manager behavior - looking for opinions

I've been working at a local startup as a DevOps engineer for the past year. I have a total of around 3 years of experience, which is considered mid-level at best. I've worked with various technologies and I'm not that confident in my technical skills but I do try my best to deliver. That's not my problem though. My problem is that communication with my manager is terrible.

I am rarely assigned any tickets. My work is a mixture of verbally assigned tasks with minimal details and initiatives I take to improve our workflows. My manager rarely joins our weekly one to one meetings so I decided to send him my task updates on teams. He doesn't reply to my messages, sometimes for days. He will only reply fast if he believes the question is important to him. He never reacts or responds to my task updates and won't review my work for months (if at all). Sometimes, when I ask him questions or try to make conversation on meetings he won't respond. When I ask again, he will laugh it off saying he heard me the first time. The kind of questions I ask are usually clarifications on my tasks, or discussion around my technical implementations.

In general, I believe I am fairly independent and I don't burden the team. My tasks are mine to deliver and I am solely responsible for them. I request guidance in a structured way and usually when I don't have enough information to move on. Even when I don't have enough context I will push through instead of waiting for weeks for an answer. The problem is that I am not responsible for the infrastructure architectural decisions, and I need the clarifications in order to do my job effectively.

I will give you a recent example:
I am asked to deploy one of our products to a test environment, but we want this product to be isolated from our other products so it's not the standard procedure.
That's pretty much what I got. I ask my manager to discuss for like 15minutes to show me what he deployed for production. He never does.

I create a list of resources I will need to deploy along with the infra design. I make the deployment. I share that with him. He shows up after a week of being AWOL to tell me some resources should be redeployed. I ask for two very specific clarifications in chat. He says we will discuss in a meeting. We join the meeting. I ask. He stays silent. I ask again. He says he heard me and he just didn't respond.

What am I supposed to do at this point? This whole thing is deeply demoralizing to me. I feel deeply disrespected and looked down upon. I am mad and sad and it's affecting my confidence and will to work and be creative and productive. I've tried a few different things since starting in this company, I've created my own tickets and shared them with him, I've tried texting him I've tried reaching out during meetings to avoid spamming him. Nothing seems to work. I dread going to work every single day. I feel lost about what to do next.

I want to stay professional but I also feel very done. I want them to fuck off but also I want to take technical experience. I want to quit but I know it's a terrible idea. What can I do to continue learning and growing within team and business goals when these are not communicated properly? What could I be doing wrong? Am I needy or is this truly as annoying and disfunctional as I think it is?

Thanks for reaching this far, any opinions, experiences or recommendations are welcome. I can take criticism as long as it is respectful, my mental health is declining fast enough already 😋

https://redd.it/1upabcd
@r_devops

How to structure scoring live traffic

We've had offline evals as part of our CI for a while now, but last month we got hit with something that none of our CI runs flagged. Our production inputs had drifted and users were asking things our test set just didn't cover, and output quality on that section of things had degraded for weeks. So unfortunately, our existing evals gave me a false sense of safety because they can only ever test what I thought to put in them.

So now I'm trying to figure out actually sampling and scoring real production responses, not just CI runs against a fixed dataset.

Main things I'm rubber ducking:

Sampling. Are people scoring all live traffic or some percentage?

Alerting. I want to know when quality drops on live traffic, but I’m not trying creat another annoying alert channel. And then if/when it goes off who/what owns the response?

https://redd.it/1upa4lk
@r_devops

How do you actually separate CI/CD pipelines for AKS across dev/qa/uat/prod in Azure DevOps?

Hey Folks, need your advice badly ,

I'm building out a CI/CD flow for AKS using Azure DevOps Pipelines (not ArgoCD/GitOps for this one, using native Azure Pipelines + KubernetesManifest@1 tasks). Trying to understand what people actually do in production.

The MS Learn sample bundles CI and CD into one pipeline (Build stage → Deploy stage, same YAML file), which builds once and deploys straight to the cluster. That seems fine for a single environment, but once you add QA → UAT → Prod with a manual sign-off before prod, it starts to feel like the wrong shape.

Questions:

1. Do you run one CD pipeline with multiple stages (QA → UAT → Prod, each an Azure DevOps Environment with its own approval gates), or separate pipelines per environment (e.g. cd-nonprod and cd-prod)? What made you choose one over the other?
2. How do you handle the nonprod → prod ACR promotion? Are you doing az acr import to copy the same digest into a separate prod registry, or do you just use one ACR with RBAC-scoped repositories/tags instead of physically separate registries?
3. If CI only has push access to a nonprod ACR, what triggers the CD pipeline — a pipeline completion trigger (resources.pipelines), a manual run with an image tag parameter, or something else?
4. For those who've tried both native Azure Pipelines deploys and ArgoCD/GitOps for AKS was there a specific pain point that pushed you from one to the other?

Not looking for "just use GitOps" as the whole answer (I get the appeal); more interested in how people structure this with plain Azure DevOps pipelines if they're not on ArgoCD, since that's what I'm working with right now.

https://redd.it/1upcl17
@r_devops

Leaving Your Cloud Provider Is About to Get Cheaper - by Law
https://cirran.eu/blog/data-act-cloud-exit-costs

https://redd.it/1uprgk7
@r_devops

How should DevOps teams contain financial blast radius from high-velocity cloud spend categories like Bedrock and Marketplace

A compromised cloud credential is no longer just a security problem. With services like Bedrock and Marketplace, it can become a financial incident within hours. I’m trying to reason through what practical DevOps / platform engineering controls actually reduce that financial blast radius before billing alerts, anomaly detection, escalation, credential revocation, and human response catch up.

I’m sharing the technical pattern we observed, the control model I’m starting to think is necessary, and the parts where I’d appreciate correction from people with deeper DevOps, platform, cloud governance, and incident response experience.

A small software company account we manage recently generated approximately USD 62k in estimated/pending charges in less than 24 hours, mainly recorded as AWS Marketplace software usage for Anthropic Claude models on Amazon Bedrock Edition.

Sanitized context:

* the account historically had low, predictable usage, mostly S3 plus small residual AWS services;
* there was no legitimate prior usage of Amazon Bedrock, Anthropic Claude models, AWS Marketplace LLMs, or globally distributed AI inference workloads;
* the abnormal usage appeared across multiple AWS regions in a very short period;
* MFA was enabled;
* during emergency containment, we found and removed access keys that were not recognized by or authorized by us;
* we asked AWS to reconstruct the relevant IAM / CloudTrail / STS / Marketplace / Bedrock timeline to determine the identity path behind the usage.

I understand the shared responsibility model. I also understand that MFA does not protect every non-interactive credential path, especially access keys, STS sessions, assumed roles, temporary credentials, CI/CD secrets, or other API-based access paths.

The IAM/security layer is central, but for high-velocity spend categories, the financial impact can accumulate faster than billing data, alert triage, escalation, credential revocation, and human response can realistically converge.

From a DevOps / platform engineering perspective, my current working model is this:

**1. Billing visibility is not containment**

Budgets, Cost Explorer, Cost Anomaly Detection, dashboards, CUR analysis, and billing alerts are useful, but they mostly tell you that something happened or is happening.

They do not necessarily stop usage at the point where the expensive action occurs: API call, model invocation, Marketplace subscription, procurement action, or high-cost service usage.

For slower-moving workloads, detection may be enough to avoid catastrophic exposure. For AI inference and Marketplace-based usage, it may not be.

The control objective should be **time-to-stop**, not just **time-to-detect**.

**2. High-cost service categories probably need default-deny governance**

For accounts with no business reason to use Bedrock or AWS Marketplace LLMs, I’m increasingly convinced that the safer pattern is not “detect unexpected spend later.”

The safer pattern is:

* deny Bedrock by default unless explicitly approved for that account;
* deny AWS Marketplace subscription / procurement actions by default unless there is a documented business case;
* use account-level service allow-lists where possible;
* avoid relying on an ever-growing deny-list of new AI or Marketplace services;
* require explicit approval before a low-cost account can enter a high-velocity spend category.

In an AWS Organizations setup, the more robust pattern seems to be layered governance: SCPs as the outer preventive guardrail to keep unused high-cost categories closed by default; IAM permission boundaries and least-privilege policies to constrain principals inside the account; region restrictions to limit fan-out; and an explicit approval path before enabling Bedrock, Marketplace procurement, or any new high-velocity spend category.

**3. First-time service/category usage should be treated differently from a normal cost spike**

A spike in an existing EC2,

S3, RDS, or Lambda workload is one kind of signal.

A first-time transition from “no Bedrock / no Marketplace LLM history” into large, multi-region AI usage is a different risk class.

That kind of transition feels closer to a governance event than to a normal cost anomaly.

Useful signals would include:

* first-ever AWS Marketplace spend in an account;
* first-ever Bedrock model usage;
* first-ever usage of a service category materially outside the account’s historical profile;
* first-ever activity in regions the account does not normally use;
* first-ever use of high-cost services from a CI/CD identity, automation role, deployment user, or long-lived access key.

Even if some of this is detection rather than hard prevention, it seems much more actionable than waiting for a generic billing anomaly.

**4. Region scope is part of financial blast-radius control**

A lot of cloud permission discussions focus on services, but region scope seems equally important.

If a credential can use an expensive service across every supported region, the blast radius is much larger. Region-locking turns a global blast radius into a smaller and more understandable failure domain.

For workloads that only operate in one or two regions, it seems reasonable to enforce approved regions by default through SCP conditions, IAM policy conditions, permission boundaries, account separation, or Control Tower guardrails.

**5. Quotas may help, but they are not a universal financial hard cap**

Service quotas may be useful where they exist and where they meaningfully limit throughput, but they seem too service-specific to be the primary financial safety model.

For unused high-cost services, proactively reducing quotas may still be useful as one layer. But the more reliable preventive layer seems to be permission-based governance: SCPs, IAM, permission boundaries, region restrictions, and Marketplace controls.

**6. Long-lived credentials are a financial risk, not just a security risk**

DevOps teams already know long-lived access keys are dangerous, especially in CI/CD systems, developer machines, legacy scripts, and old integrations.

What this incident made clearer to me is that long-lived credentials should also be treated as a financial blast-radius risk.

The safer pattern seems to be:

* avoid long-lived IAM user access keys where possible;
* move humans to SSO / IAM Identity Center;
* use short-lived role assumption for automation;
* scope CI/CD identities aggressively;
* separate deployment roles from runtime roles;
* prevent deployment credentials from having broad service enablement or Marketplace permissions;
* regularly detect unused, stale, or overprivileged credentials.

**7. Automated containment is attractive but operationally risky**

The most aggressive pattern would be some kind of circuit breaker:

* detect suspicious cross-region activity;
* detect unexpected Bedrock / Marketplace usage;
* detect first-time use of high-cost services by automation identities;
* disable or quarantine the relevant access key or role;
* detach risky permissions;
* apply an emergency SCP;
* notify the account owner immediately.

That could reduce time-to-stop dramatically, but it is not operationally trivial. Circuit breakers can create false positives, break production workloads, and require a maintained model of expected services, regions, identities, and usage patterns.

**8. Forensics still matter, but they are not the first line of containment**

After the fact, CloudTrail / IAM / STS / Marketplace / Bedrock evidence is essential to reconstruct the identity path: principal, credential, session, source IPs, regions, Marketplace / Bedrock actions, timing, and correlation with billing line items.

But for high-velocity spend categories, forensic reconstruction explains what happened after the exposure. It does not, by itself, limit the exposure while it is happening.

So the architecture question becomes:

**What should a cloud account already have in place so that a compromised credential, CI/CD secret, automation role, or long-lived access key

cannot silently enter a new high-cost category, fan out across regions, and generate material spend before humans can react?**

My current takeaway is that mature DevOps / platform governance for these categories probably needs a layered model:

* Organizations/SCP default-deny for high-cost or unused service categories;
* explicit approval workflow for Bedrock, Marketplace, and similar services;
* IAM permission boundaries for human, workload, and CI/CD identities;
* no long-lived credentials where avoidable;
* SSO / IAM Identity Center for humans;
* short-lived role assumption for automation;
* region restrictions by default;
* service quotas reduced where meaningful;
* first-time service/category usage detection;
* emergency containment automation for accounts with narrow expected usage;
* forensic readiness through CloudTrail and billing correlation.

I’d appreciate feedback from people who have implemented similar controls in real cloud environments.

Which controls actually reduce financial blast radius in practice, which ones look good only on paper, and what would you prioritize first to optimize for time-to-stop, not just time-to-detect?

https://redd.it/1uptjrx
@r_devops

Follow-up: How does your team actually move important Slack discussions into documentation?
https://www.reddit.com/r/devops/comments/1unvlkj/engineering_managers_how_do_you_prevent_valuable/

https://redd.it/1upqr3q
@r_devops

Looking at tooling and curious how your team actually using AI in ITops?

Curious how teams are actually using AI in day-to-day ITops, not the vendor pitch version, but what's genuinely saving time , improves results vs. what's been overhyped.

https://redd.it/1upvgyr
@r_devops

Security you can't justify is a vicious cycle
https://bogomolov.work/blog/posts/security-you-cant-justify-is-a-vicious-cycle/

https://redd.it/1upxytm
@r_devops

Security you can't justify is a vicious cycle
https://bogomolov.work/blog/posts/security-you-cant-justify-is-a-vicious-cycle/

https://redd.it/1upzq3o
@r_devops

DevOps is one of the most vaguely defined roles in tech, and I think that's exactly the point

​

DevOps spans a huge range: CI/CD, security, SRE, DevEx, each with its own sub functions. At big companies, engineers often specialize in one. At startups, you juggle all of them at once, usually by necessity.

Here's the distinction I keep coming back to: DevOps isn't about mastering one function in isolation. It's about holding all of them in your head at once, even when you're only actively working on one. Building CI/CD? You're also thinking observability, security, and DevEx, because a pipeline that ships fast but can't be debugged, or isn't secure, isn't actually done.

Curious how this plays out elsewhere. Does your team specialize by function, or does everyone think across all of them regardless of company size?

https://redd.it/1uq3g3d
@r_devops

Anyone else spend half their week just chasing down which tag blew up your cardinality?

we had a service push a UUID out as a label last month and our metrics query latency went sideways for two days before anyone traced it back to the actual deploy. nobody noticed until dashboards started timing out.

now i'm paranoid about every new label that goes into prod, and "just use fewer labels" isn't really an answer when you don't know which label is the problem until it's already in your bill.

please share how does your team actually catch this before it happens?

https://redd.it/1uqfflz
@r_devops

Need an Azure learning path for an internal transfer to DevOps (2-3 month timeline)
https://redd.it/1uqjb9j
@r_devops