Security you can't justify is a vicious cycle
https://bogomolov.work/blog/posts/security-you-cant-justify-is-a-vicious-cycle/
https://redd.it/1upxytm
@r_devops
https://bogomolov.work/blog/posts/security-you-cant-justify-is-a-vicious-cycle/
https://redd.it/1upxytm
@r_devops
bogomolov.work
Security you can't justify is a vicious cycle
Dependabot is on fire, the model yells, the meeting says add TLS. Fine. Is this stopping an attacker, or feeding a checklist?
Security you can't justify is a vicious cycle
https://bogomolov.work/blog/posts/security-you-cant-justify-is-a-vicious-cycle/
https://redd.it/1upzq3o
@r_devops
https://bogomolov.work/blog/posts/security-you-cant-justify-is-a-vicious-cycle/
https://redd.it/1upzq3o
@r_devops
bogomolov.work
Security you can't justify is a vicious cycle
Dependabot is on fire, the model yells, the meeting says add TLS. Fine. Is this stopping an attacker, or feeding a checklist?
DevOps is one of the most vaguely defined roles in tech, and I think that's exactly the point
​
DevOps spans a huge range: CI/CD, security, SRE, DevEx, each with its own sub functions. At big companies, engineers often specialize in one. At startups, you juggle all of them at once, usually by necessity.
Here's the distinction I keep coming back to: DevOps isn't about mastering one function in isolation. It's about holding all of them in your head at once, even when you're only actively working on one. Building CI/CD? You're also thinking observability, security, and DevEx, because a pipeline that ships fast but can't be debugged, or isn't secure, isn't actually done.
Curious how this plays out elsewhere. Does your team specialize by function, or does everyone think across all of them regardless of company size?
https://redd.it/1uq3g3d
@r_devops
​
DevOps spans a huge range: CI/CD, security, SRE, DevEx, each with its own sub functions. At big companies, engineers often specialize in one. At startups, you juggle all of them at once, usually by necessity.
Here's the distinction I keep coming back to: DevOps isn't about mastering one function in isolation. It's about holding all of them in your head at once, even when you're only actively working on one. Building CI/CD? You're also thinking observability, security, and DevEx, because a pipeline that ships fast but can't be debugged, or isn't secure, isn't actually done.
Curious how this plays out elsewhere. Does your team specialize by function, or does everyone think across all of them regardless of company size?
https://redd.it/1uq3g3d
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Anyone else spend half their week just chasing down which tag blew up your cardinality?
we had a service push a UUID out as a label last month and our metrics query latency went sideways for two days before anyone traced it back to the actual deploy. nobody noticed until dashboards started timing out.
now i'm paranoid about every new label that goes into prod, and "just use fewer labels" isn't really an answer when you don't know which label is the problem until it's already in your bill.
please share how does your team actually catch this before it happens?
https://redd.it/1uqfflz
@r_devops
we had a service push a UUID out as a label last month and our metrics query latency went sideways for two days before anyone traced it back to the actual deploy. nobody noticed until dashboards started timing out.
now i'm paranoid about every new label that goes into prod, and "just use fewer labels" isn't really an answer when you don't know which label is the problem until it's already in your bill.
please share how does your team actually catch this before it happens?
https://redd.it/1uqfflz
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Need an Azure learning path for an internal transfer to DevOps (2-3 month timeline)
https://redd.it/1uqjb9j
@r_devops
https://redd.it/1uqjb9j
@r_devops
Optimising workspace for multiple laptops?
What setup have you put in place on your desk to manage being on multiple laptops?
https://redd.it/1uqm1f2
@r_devops
What setup have you put in place on your desk to manage being on multiple laptops?
https://redd.it/1uqm1f2
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Why are intermittent production bugs so hard to reproduce?
I had one of those bugs last month that made the whole team question reality for about four days. A checkout service would randomly throw 500s, maybe 3-4 times a day, always on the same endpoint, never on a predictable schedule sometimes it happened during peak traffic, sometimes at 3am with almost nobody hitting the system. Logs showed the error but the stack trace pointed to a null reference in a place that should have been impossible given our validation logic. We tried the obvious stuff first: replaying the same request payload from the failed logs, same user, same cart contents, down to the byte worked fine every single time locally and in staging couldn't reproduce it on demand no matter how hard we tried. Turned out three things had to line up at once to trigger it, a specific cache eviction timing on one node, a race condition between two async calls that only mattered under a certain concurrency level and a stale feature flag value that got cached slightly differently depending on which pod served the request none of that shows up in a single request replay because the bug wasn't really about the request, it was about the state of the system at that exact moment. This is why intermittent bugs are so brutal to chase down the failure depends on system state not just input, things like cache state, connection pool exhaustion, background job timing or which pod handled the request all shift constantly and aren't captured by a normal log line. Concurrency and timing windows are nearly impossible to force, since race conditions might only trigger under load patterns that don't exist in staging or in a single manual test and by the time you are looking at it, the state that caused it is already gone. logs tell you what happened not the exact conditions leading up to it, so you are reconstructing a crime scene from a few photos instead of watching it happen. We eventually solved it by adding much more granular tracing around the cache and flag evaluation logic then waiting for it to happen again with all that extra visibility on took another two days after that just to catch it in the act. How do other people approach this class of bug, better tracing and waiting to catch it live or actual techniques for forcing race conditions to surface faster before they become customer-facing?
https://redd.it/1uqm2hr
@r_devops
I had one of those bugs last month that made the whole team question reality for about four days. A checkout service would randomly throw 500s, maybe 3-4 times a day, always on the same endpoint, never on a predictable schedule sometimes it happened during peak traffic, sometimes at 3am with almost nobody hitting the system. Logs showed the error but the stack trace pointed to a null reference in a place that should have been impossible given our validation logic. We tried the obvious stuff first: replaying the same request payload from the failed logs, same user, same cart contents, down to the byte worked fine every single time locally and in staging couldn't reproduce it on demand no matter how hard we tried. Turned out three things had to line up at once to trigger it, a specific cache eviction timing on one node, a race condition between two async calls that only mattered under a certain concurrency level and a stale feature flag value that got cached slightly differently depending on which pod served the request none of that shows up in a single request replay because the bug wasn't really about the request, it was about the state of the system at that exact moment. This is why intermittent bugs are so brutal to chase down the failure depends on system state not just input, things like cache state, connection pool exhaustion, background job timing or which pod handled the request all shift constantly and aren't captured by a normal log line. Concurrency and timing windows are nearly impossible to force, since race conditions might only trigger under load patterns that don't exist in staging or in a single manual test and by the time you are looking at it, the state that caused it is already gone. logs tell you what happened not the exact conditions leading up to it, so you are reconstructing a crime scene from a few photos instead of watching it happen. We eventually solved it by adding much more granular tracing around the cache and flag evaluation logic then waiting for it to happen again with all that extra visibility on took another two days after that just to catch it in the act. How do other people approach this class of bug, better tracing and waiting to catch it live or actual techniques for forcing race conditions to surface faster before they become customer-facing?
https://redd.it/1uqm2hr
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
What are you guys using for KYC API for identity verification and onboarding?
Getting straight to the problem: our current KYC API is hitting its limits. Slower verification times, too many false rejections on international documents and support goes quiet the moment something breaks in production.Every vendor sounds identical on paper so we fail to understand what does AI powered verification actually mean when an edge case hits at 2am(lol) and thousands of verifications are queued up? Looking for something API first with good docs, reliable webhooks, global document support, and AML screening that is baked in. Anyone who has run more than one of these in production and can give an honest take would be really helpful.
https://redd.it/1uqookv
@r_devops
Getting straight to the problem: our current KYC API is hitting its limits. Slower verification times, too many false rejections on international documents and support goes quiet the moment something breaks in production.Every vendor sounds identical on paper so we fail to understand what does AI powered verification actually mean when an edge case hits at 2am(lol) and thousands of verifications are queued up? Looking for something API first with good docs, reliable webhooks, global document support, and AML screening that is baked in. Anyone who has run more than one of these in production and can give an honest take would be really helpful.
https://redd.it/1uqookv
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Do people actually use their GPUs as much as they expected?
When I first started looking at upgrading my setup, I was convinced I'd be using a GPU every day.
Reality turned out to be pretty different.
Some weeks I barely run anything. Then I'll have two or three days where I'm constantly launching jobs.
Now I'm wondering if most people actually use their hardware that much, or if it mostly sits there waiting for those busy days.
https://redd.it/1uqpme2
@r_devops
When I first started looking at upgrading my setup, I was convinced I'd be using a GPU every day.
Reality turned out to be pretty different.
Some weeks I barely run anything. Then I'll have two or three days where I'm constantly launching jobs.
Now I'm wondering if most people actually use their hardware that much, or if it mostly sits there waiting for those busy days.
https://redd.it/1uqpme2
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Could you please share a typical daily routine for a DevOps professional and outline the types of tools used for application hosting?
The objective here is to investigate the DevOps tools currently employed by various companies within the market, thereby enabling adequate preparation for a potential career transition.
Also share the year of experience do you have?
https://redd.it/1uqqjho
@r_devops
The objective here is to investigate the DevOps tools currently employed by various companies within the market, thereby enabling adequate preparation for a potential career transition.
Also share the year of experience do you have?
https://redd.it/1uqqjho
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
How do you use agents and LLMs in your work
I was wondering how you guys use LLMs and agents in your day to day work, do you use them to write yaml/boilerplate or beyond that?
https://redd.it/1uqsg66
@r_devops
I was wondering how you guys use LLMs and agents in your day to day work, do you use them to write yaml/boilerplate or beyond that?
https://redd.it/1uqsg66
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Docs as Code implementation for Infrastructure
Hi there
Recently I was tasked to write documentation for our infrastructure in "doc as code" way but I have not very well grasped what it is
The only requirement my team leads has is that the documents should be enough for any new person to understand our infra setup and tools we are using.
They also mentioned that any changes in the documents should have a PR and only after reviewing and approving any changes should be visible.
What I understand till now is that we would have a central repository in confluence or version control with documentation files.
There should be a way to navigate to different documents
All .md files are similar in structure, how they are written
Architecture diagrams to show infrastructure
I had a look at kubernetes documentation as I get what it is everything is in markdown it is being rendered to the website and has different documents for different versions.
But I still have no idea how to start on this.
Can I know what are some common points to note down or industry standard for these kind of documentation. And how to implement it
https://redd.it/1uqwjuf
@r_devops
Hi there
Recently I was tasked to write documentation for our infrastructure in "doc as code" way but I have not very well grasped what it is
The only requirement my team leads has is that the documents should be enough for any new person to understand our infra setup and tools we are using.
They also mentioned that any changes in the documents should have a PR and only after reviewing and approving any changes should be visible.
What I understand till now is that we would have a central repository in confluence or version control with documentation files.
There should be a way to navigate to different documents
All .md files are similar in structure, how they are written
Architecture diagrams to show infrastructure
I had a look at kubernetes documentation as I get what it is everything is in markdown it is being rendered to the website and has different documents for different versions.
But I still have no idea how to start on this.
Can I know what are some common points to note down or industry standard for these kind of documentation. And how to implement it
https://redd.it/1uqwjuf
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Looking for advice on transitioning from Sysadmin to DevOps
I'm looking to apply to devops/sre positions to change my current job and have a profesional glow up, i no longer feel challenged project from my job and i am stating to just doing app maintaenance and helpdesk tasks.
I am a sysadmin with hands on production enviroments and automatation background (scripting and low code) but where i learn and enjoy the most is in my homelab, I have around 2.5 years of professional experience.
I'd like to learn technologies such as Terraform (mainly because I see it requested in many LinkedIn job postings) and Ansible, as well as deepen my knowledge of CI/CD pipelines. I've already worked with GitHub Actions.
I've also used AI to help me create a learning roadmap and prioritize milestones. One of the strongest recommendations was to document everything on GitHub.
Beyond following a roadmap, I'd like to hear what you think are the most important things to focus on when transitioning into DevOps. If you've seen what helps people land their first DevOps role, or if you have any advice on common mistakes, skills to prioritize, or portfolio ideas, I'd really appreciate your perspective.
https://redd.it/1ur3a7j
@r_devops
I'm looking to apply to devops/sre positions to change my current job and have a profesional glow up, i no longer feel challenged project from my job and i am stating to just doing app maintaenance and helpdesk tasks.
I am a sysadmin with hands on production enviroments and automatation background (scripting and low code) but where i learn and enjoy the most is in my homelab, I have around 2.5 years of professional experience.
I'd like to learn technologies such as Terraform (mainly because I see it requested in many LinkedIn job postings) and Ansible, as well as deepen my knowledge of CI/CD pipelines. I've already worked with GitHub Actions.
I've also used AI to help me create a learning roadmap and prioritize milestones. One of the strongest recommendations was to document everything on GitHub.
Beyond following a roadmap, I'd like to hear what you think are the most important things to focus on when transitioning into DevOps. If you've seen what helps people land their first DevOps role, or if you have any advice on common mistakes, skills to prioritize, or portfolio ideas, I'd really appreciate your perspective.
https://redd.it/1ur3a7j
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Additional burden of hosing AI apps
With AI, business and product teams are creating apps left and right. They dont understand what the code is doing, no clue about security or how to host it.
This burden falls on DevOps/Engineering to now maintain it, fix it. Authors are still considered the owners of these apps. I wanted to know how are you guys handling this situation?
\- Is Engineering/DevOps the defacto owners of such apps in your company?
\- How are you deploying these - in your prod AWS or some hosted env?
TIA
https://redd.it/1ur4nwc
@r_devops
With AI, business and product teams are creating apps left and right. They dont understand what the code is doing, no clue about security or how to host it.
This burden falls on DevOps/Engineering to now maintain it, fix it. Authors are still considered the owners of these apps. I wanted to know how are you guys handling this situation?
\- Is Engineering/DevOps the defacto owners of such apps in your company?
\- How are you deploying these - in your prod AWS or some hosted env?
TIA
https://redd.it/1ur4nwc
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
How to use my old laptop
Anyone help me what I can do useful with my old laptop related to server or something and I don't have router.
https://redd.it/1urbjdi
@r_devops
Anyone help me what I can do useful with my old laptop related to server or something and I don't have router.
https://redd.it/1urbjdi
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
What is your optimal infrastructure
we're all striving to optimize our infra, but what would you say is the PERFECT infra configuration in your mind?
I'll go first
# Product:
micro services
each service has its own git repo following a standardized directory structure
each service has its own pipeline that scans the code and runs a full regression suite before a PR is merged to master/main
each service has a health check that is actually accurate to the state of the service
launch time for the services are minimal
daily pipeline regression suite that updates a global dashboard highlighting all the current repo statuses
SaaS ONLY (onprem is pain)
# Deployment:
terraform code base that is multiplatform, can be deployed to any cloud
deployment process is executed by a pipeline with optional parameters for configure what kind of environment to deploy
all services hosted in GKE/EKS or some other cloud hosting platform
auto scaling horizontally and vertically
all logs routed to logz.io and tagged by environment and service
all deployed infra is tagged by its environment name
# Monitoring:
new relic, all the things go to new relic. (not sponsored, i just used it in the past and it was a dream when setup correctly. was just too expensive for my previous company at the time)
one thing i did learn is that the less you "have" to do. the better. and if your company can afford to outsource something, your life will be much easier.
https://redd.it/1urdygd
@r_devops
we're all striving to optimize our infra, but what would you say is the PERFECT infra configuration in your mind?
I'll go first
# Product:
micro services
each service has its own git repo following a standardized directory structure
each service has its own pipeline that scans the code and runs a full regression suite before a PR is merged to master/main
each service has a health check that is actually accurate to the state of the service
launch time for the services are minimal
daily pipeline regression suite that updates a global dashboard highlighting all the current repo statuses
SaaS ONLY (onprem is pain)
# Deployment:
terraform code base that is multiplatform, can be deployed to any cloud
deployment process is executed by a pipeline with optional parameters for configure what kind of environment to deploy
all services hosted in GKE/EKS or some other cloud hosting platform
auto scaling horizontally and vertically
all logs routed to logz.io and tagged by environment and service
all deployed infra is tagged by its environment name
# Monitoring:
new relic, all the things go to new relic. (not sponsored, i just used it in the past and it was a dream when setup correctly. was just too expensive for my previous company at the time)
one thing i did learn is that the less you "have" to do. the better. and if your company can afford to outsource something, your life will be much easier.
https://redd.it/1urdygd
@r_devops
Logz.io
Logz.io: Modern Observability Powered by AI
Stop chasing alerts. Logz.io's AI-powered observability platform unifies logs, metrics, and traces to cut MTTR, automate root cause analysis, and get ahead of issues before they impact users.
What's up with GitHub runners lately?
https://preview.redd.it/yzae6azo76ch1.png?width=718&format=png&auto=webp&s=6a1dbf5a366b67da18cc08e2b898d8a764cf0580
Too much AI slop to build? If this continues I will probably prefer to just use my self hosted ones for all jobs.
https://redd.it/1urkyux
@r_devops
https://preview.redd.it/yzae6azo76ch1.png?width=718&format=png&auto=webp&s=6a1dbf5a366b67da18cc08e2b898d8a764cf0580
Too much AI slop to build? If this continues I will probably prefer to just use my self hosted ones for all jobs.
https://redd.it/1urkyux
@r_devops
How should I stay curious and sharp and learn more? What resources should I go to?
Hello, I have been working as a dev sec ops for the last 4 years and recently as a devops engineer.
At my last job, since everything was new, and I had no idea about so many things everything was exciting. Learning Terraform, ansible, Linux, how to sys admin, Grafana monitoring so on.
Now at this new job, I realized that ok, I am not super familiar with CI/CD, I could use some work there, but the rest of the things I had to do, were needed in such a hurry that I did not have the time to even understand stuff, so as you know, AI (mostly Claude).
Anyway the thing is with all this AI here and there, how do you stay motivated to learn, where do you turn to, how should I go about it?
I feel like my brain is getting numb lately, and I want to do something about this, I do not want to end up being a mediocre human, just another prompter (not even good at it).
Thanks a bunch.
https://redd.it/1urnqrt
@r_devops
Hello, I have been working as a dev sec ops for the last 4 years and recently as a devops engineer.
At my last job, since everything was new, and I had no idea about so many things everything was exciting. Learning Terraform, ansible, Linux, how to sys admin, Grafana monitoring so on.
Now at this new job, I realized that ok, I am not super familiar with CI/CD, I could use some work there, but the rest of the things I had to do, were needed in such a hurry that I did not have the time to even understand stuff, so as you know, AI (mostly Claude).
Anyway the thing is with all this AI here and there, how do you stay motivated to learn, where do you turn to, how should I go about it?
I feel like my brain is getting numb lately, and I want to do something about this, I do not want to end up being a mediocre human, just another prompter (not even good at it).
Thanks a bunch.
https://redd.it/1urnqrt
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Is there an orchestration tool that doesn't force everyone on the team to learn python ?
Genuine question. Our team is 3 data engineers (python), 2 analytics engineers (SQL and dbt), and 2 infra guys (terraform / ansible). We tried airflow but the analytics and anfra pp can't contribute to pipeline definition bc everything is python DAGs. It creates a bottleneck where the 3 python devs end up writing and maintaining every single workflow.
I am looking for sth where defining run this dbt job, then this terraform apply, then notify Slack doesn't require importing fifteen python modules. Does this exist or am I dreaming?
https://redd.it/1uro23y
@r_devops
Genuine question. Our team is 3 data engineers (python), 2 analytics engineers (SQL and dbt), and 2 infra guys (terraform / ansible). We tried airflow but the analytics and anfra pp can't contribute to pipeline definition bc everything is python DAGs. It creates a bottleneck where the 3 python devs end up writing and maintaining every single workflow.
I am looking for sth where defining run this dbt job, then this terraform apply, then notify Slack doesn't require importing fifteen python modules. Does this exist or am I dreaming?
https://redd.it/1uro23y
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
DevOps/Platform engineering vs. Software Dev, is it less creative and more rote work?
I have been a fullstack software dev for 5 years already, with some years also doing ops stuff for the team (since no one bothered to/liked doing it) like managing Jenkins IaC, pipelines, AWS CDK, K8S deployments, etc. I liked those stuff and our team was really suffering because no one bothered to take care of it so I took leadership there.
I am now looking for another job, since my contract ended. I just got an offer to work as a cloud engineer at another organization.
To be honest, I do like being a dev, but I could not really see myself being a "senior" or freelancer in this field one day. This is because I feel in software dev there is a lot of "openness" or options on how to do something and it is very highly opinionated, and it is hard to find the "correct" solution. For example with design patterns (do you need to apply patterns? do we need this abstraction/interface?), or with REST APIs (how do you design your endpoints), or with frontend design decisions (confirm button on right or left side? color? opacity? etc.).
And with DevOps, at least so far from what I see there is less "opinions" e.g. you follow the vendor's directions, if it deploys and it runs then it's good (less edge cases), there is more standardized ways of doing something/deploying something, and also it is domain-independent.
In software dev, you have to understand the domain to make business impact, and that can take away a lot of time from coding itself.
It is also easier to prove yourself for other jobs through certifications, whereas with full stack there's no such luxury.
But the disadvantage I see with DevOps is that it is more stressful than a software dev position, for example through on-calls, although you do get paid for your extra hours so I think it compensates it somewhat. And being on call I think really teaches you to be a tough person mentally, able to say no to other people, not be a cry baby, so it helps also perhaps with self development.
And also with DevOps, it can be harder to try something out (you will need to have a free AWS account to try deployments, etc.) although I might be wrong here. And since there's so much breadth, you cannot understand the root cause of everything going wrong, but I may be wrong here.
What is your opinion here? Do you see DevOps as being less "uncertain" than fullstack, or is it not the case?
https://redd.it/1urufkh
@r_devops
I have been a fullstack software dev for 5 years already, with some years also doing ops stuff for the team (since no one bothered to/liked doing it) like managing Jenkins IaC, pipelines, AWS CDK, K8S deployments, etc. I liked those stuff and our team was really suffering because no one bothered to take care of it so I took leadership there.
I am now looking for another job, since my contract ended. I just got an offer to work as a cloud engineer at another organization.
To be honest, I do like being a dev, but I could not really see myself being a "senior" or freelancer in this field one day. This is because I feel in software dev there is a lot of "openness" or options on how to do something and it is very highly opinionated, and it is hard to find the "correct" solution. For example with design patterns (do you need to apply patterns? do we need this abstraction/interface?), or with REST APIs (how do you design your endpoints), or with frontend design decisions (confirm button on right or left side? color? opacity? etc.).
And with DevOps, at least so far from what I see there is less "opinions" e.g. you follow the vendor's directions, if it deploys and it runs then it's good (less edge cases), there is more standardized ways of doing something/deploying something, and also it is domain-independent.
In software dev, you have to understand the domain to make business impact, and that can take away a lot of time from coding itself.
It is also easier to prove yourself for other jobs through certifications, whereas with full stack there's no such luxury.
But the disadvantage I see with DevOps is that it is more stressful than a software dev position, for example through on-calls, although you do get paid for your extra hours so I think it compensates it somewhat. And being on call I think really teaches you to be a tough person mentally, able to say no to other people, not be a cry baby, so it helps also perhaps with self development.
And also with DevOps, it can be harder to try something out (you will need to have a free AWS account to try deployments, etc.) although I might be wrong here. And since there's so much breadth, you cannot understand the root cause of everything going wrong, but I may be wrong here.
What is your opinion here? Do you see DevOps as being less "uncertain" than fullstack, or is it not the case?
https://redd.it/1urufkh
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community