Debug & chill - Articles of infra & devops debugging

Hey everyone!
Back in 2020, I started jotting down notes about various debugging adventures to keep track of my process and insights. I’ve finally turned them into a series of blog posts—totally free and purely for educational purposes. If you’re curious about network troubleshooting (and some cool tools to help you along the way), check out the first episode here:

Debug and Chill #1

It covers a relatively simple network issue but offers plenty of takeaways you can apply to other situations. I hope you find it helpful! Let me know what you think.

https://redd.it/1ion4wj
@r_devops

Experienced sa/devops - Learning basic Python quickly

I have over 5 years of experience with linux system administration (all the classics), and cloud/DevOps (terraform, kubernetes, ci/cd etc) and I know how to write basic bash scripts. I have a few interviews coming up and I understood that there will be fairly basic coding assignments.

I guess that they can be done with Bash, but diving deeply into bash seems a bit pointless to me, and the syntax for scripts that are above something basic looks complicated.

I worked with Python, sometimes helped debugging some code, I can kind of understand it, but I can't really write it (without gpt), I never learned it properly. Is it realistic and is it a good idea to try to intensively learn the basics to the level of solving a coding challenge in an interview for a DevOps position in let's say a week? Or I'd better spend this time diving deeper into bash? And which learning resources would you recommend in my case? I see that many courses are geared towards developing and such while it's not as relevant in my case.

https://redd.it/1iooi80
@r_devops

Thank you all and Goodbye!


I got told I'm affected in VMware's latest layoffs, and I've decided to quit tech after 10 years and focus on my knifemaking hobby to be more consistent in delivering orders on time :D

Reading this sub throughout my career has been so helpful professionally, so thank you all! Wishing everyone's servers/deployments good health! Goodbye!

https://redd.it/1ioqk17
@r_devops

Is there any free database client for MacOS which supports IAM auth?

wondering if there is any database tool for MacOS that has support for IAM based authentication.

It looks like licensed versions of dbeaver support it but I'm wondering if there are any other options.

https://redd.it/1iori6w
@r_devops

I'm having some serious problems with GitHub Actions

Hello there.

I am just a poor dotnet dev trying to automatize the release of his internal app.

What I wanna do in a single action is:

- Build the app in a runner (it's a WinForms app, so it has to be windows-latest)

- Name the release in a certain pattern (e.g. yyyy-MM-dd_vXX) where XX is the build number of the day (so the first release for a day will be 01, second release for the day 02 and so on).

- Then zip the build output and make a github release.

I've spent literally the past 12 hours trying to get this to work somehow.

I'm at the stage where I've got a little app written in Go that uses the GitHub API to get and increment the version string. It then outputs three strings (name of the release, name of the zip file, name of the db-migration-runner-app zip file), which are then supposed to be written to the $env:GITHUB_OUTPUT file using a little powershell script.

The Go program is a separate repo/action that I am using in the main dotnet app workflow. It then tries to pass out the strings in the action outputs to be used in next steps, but to no avail.

Here's the relevant output part of the Go app's action.yml

releaseName:
description: 'Name of the release'
releaseNameZip:
description: 'Name of the release with a .zip suffix'
migratorNameZip:
description: 'Name of the release'

I've tried doing this after supposedly setting the file in the action (powershell):

Write-Host "Reading github output file"
Write-Host (Get-Content -Path $env:GITHUB_OUTPUT)

But the runner's logs show nothing.

So I've tried getting the $env:GITHUB_OUTPUT using the simple pwsh command:

$env:GITHUB_OUTPUT

Yet it shows that the variable inside the runner is empty for some reason

I'm contemplating three options:

- Commiting sudoku

- Rewriting this action in JS and using official bindings instead of go

- Using a different scripting language to just build + increment version + zip + upload release in one "step".

https://redd.it/1iotow3
@r_devops

DevNetOps for everyone!!

Hi everyone, I am a program manager for a national network operator working in the company’s technology, strategy and engineering org. This org has also recently absorbed some IT teams. The org has around 500 staff. Some teams are using devops, most aren’t. Some teams are quite agile while others are still managing workflow and communication through email. A large chunk of staff are working in PMOs and have not been exposed to the foundations of DevNetOps at all. Ive been assigned to manage a new program/initiative to raise awareness about devops, roll out some training, facilitate fixing some problem processes, automate anything possible and to generally get people collaborating. We have already made some great big strides in the first 6 months. We are going to start running some value stream mapping sessions next month, just looking for the right process to assess.

I just want this program to be really successful. I want staff to feel comfortable coming to our program to get relief from their daily pain points and I want to help deliver meaningful impact. If you were part of this organization, what could the program or I as the PM do to bring you and your team closer to devops?

https://redd.it/1iox766
@r_devops

What should a DevOps newb learn?

I'm a second year comp sci student and I'm starting a DevOps internship in a few weeks. I haven't had any classes on OS or Comp Architecture yet, but I've done research on linux, docker, and kubernetes for this role. What should I be focusing on as a newbie to devops to have a strong learning foundation?

https://redd.it/1iowwps
@r_devops

Getting into DevOps in 2025?

I'm thinking of going down this training path to become a devOps engineer. Targeting Azure.

I did some Javascript development and scripted alot using powershell, python, bash, Javascript. I love to automate. Right now I fell back into A+ technical support.

Development (especially web and front end) seem to been taking a huge hit. And so I want to get into devOps.

1. Is devOps also taking a hit these days? How's the Job market currently?

2. Does this path seem good to you?

AZ-900 – Azure Fundamentals

AZ-104 – Azure Administrator

AZ-204 – Azure Developer

AZ-400 – Azure DevOps Engineer

AZ-305 – Azure Solutions Architect

HashiCorp Certified Terraform Associate

Docker Certified Associate (DCA)

Certified Kubernetes Administrator (CKA)

I'm just trying to see if it's worth doing all this studying. Mind you I'm American and worried about AI and near/offshoring. I don't know if it's wise to invest my time and effort for something that might go away and die in the US. If anyone with experience knows please let me know. Thank you.



https://redd.it/1ip0n41
@r_devops

Can a 10+ years experienced Support Engineer switch to Devops ?

Hi Guys,

I have total experience of 10+ years in IT industry (4 years in developement and reminaing in Support). I recently lost job due to layoffs. My doubt is whether I can switch Devops in this situation. I have started learning Devops but have this doubt in my mind. So, please help me clear my doubt.


https://redd.it/1ip21wc
@r_devops

Does Debian tend to patch their vulnerability slower than Alpine?

I have two teams using Debian and Alpine. The one that uses Debian as base image often have critical issues that are unmatched for months. The alpine ones are close to 0

https://redd.it/1ip2hy1
@r_devops

Suggest me a good workstation, i am willing to spend around 5-6k

Hello all, i am tired of using multiple devices , setting up env every time there is new project. Can anyone suggest me a good workstation so that i can create my home lab. I already have one but the home lab is assembled using multiple laptop and one optiplex. I would like to have my own server from now on. Please do suggest me a good one i can couple of k more.

https://redd.it/1ip25wc
@r_devops

Building a new environment from scratch - tooling advice needed!

Hi all,

I'm joining a team of about 12 people who will be building an application within the AWS infrastructure.

I will be the primary/only engineer in charge of 'SRE/pipeline/devops' and I have only basic experience on this front right now (my background is QA).

I will likely need to have some basic infrastructure available for the team within a few months after I join.

It's likely that after a year or so, we will be able to bring in more people to take over the SRE side of things from me so I can focus on the QA angle.

The current infrastructure we have is very complex, was built by a team of about six people over around six years, so I don't want to have the exact same as my goal (impossible for one person in a few months).

What we will need:

* multiple namespaces running at the same time each with a set of microservices likely docker/kubernetes based
* our microservices will need to talk to each other within a namespace and will likely need access to their own or a chunk of a database and message bus at least, and writing logs into opensearch
* git source control with the usual branches and code reviews etc
* automatic pipelines triggered on merge requests to run unit tests and component tests
* package the various microservices into docker containers
* push new builds to a staging ns (long term I'd like to avoid this step once we have a good handle on quality within the MR pipelines - can push straight to prod)
* ability to create or delete a kubernetes namespace and all running containers and linked message bus queues, db tables etc created/deleted as needed
* ability to deploy a specific version of a built microservice into a namespace for test purposes
* on demand or timer or after a pipeline, deploy all latest versions of services to all namespaces (push to prod)

I know the usual tools for this are gitlab/github, jenkins, eks/kubernetes, ambassador, jfrog, cloudflare, opensearch, etc but that feels like quite a lot to get production ready for one person in a few months?

Given my current lack of knowledge, what would be my best approach here? I've read that AWS CodeCatalyst might make sense - lets me compile, test, store artifacts, deploy to eks etc all within AWS and it all autoscales. Any opinions on this?

Or are there other tools like ArgoCD which are latest and greatest/best in class which might be worth looking at? Could gitlab + amazon eks literally be enough for example?

And maybe I can just use AWS managed versions of opensearch, kafka, etc for the services my microservices talk to?

Appreciate any help to direct my learning over the upcoming months!



https://redd.it/1ip6zrq
@r_devops

Jenkins on LXC?

I know I can run jenkins on a VM. Recently I saw some company says they started running jenkins on Linux Containers. So, I was wondering can I do the same on my proxmox. When tried to install on a ubuntu LXC, it got installed but failed to run stating "Failed to start jenkins Continuous Integration Server" Status Code : 1/FAILURE. No other error messages on journalctl. Any ideas? Has Anyone tried this?

Note: Some of you may ask why not run on a VM, I can and I am already running. But want to lower my resource since I am not doing much with it.

https://redd.it/1ip81tx
@r_devops

Is there a more secure way to setup a CI pipeline for a FUSE Project That Does Not Involve Enabling Privileged Flag for Docker or Gitlab Runner?

I'm playing around with setting up a CI pipeline that runs e2e testing for work. This project involves having a FUSE mount and e2e testing for this project is done manually, which frankly sucks. I'm developing a script to automate this, but I'm thinking if I can do one step further and make this run in a CI pipeline on gitlab.

I tested that mounting fuse only works if the runner is priveleged, but my question is if there is a more secure way of doing this. It would be greatly appreciated if there are similar open source pipeline examples of doing this.

Thank you!

https://redd.it/1ip93ec
@r_devops

Looking for advice on understanding developer experience

Hey everyone,

Lately, there's been a lot of talk about how developer experience impacts productivity. Research shows that productivity isn’t just about metrics, it’s also about how developers feel about, think about, and value their work. In our team, we’ve been relying on developer feedback to uncover inefficiencies in our processes.

That’s why we’re considering a tool that could help teams better understand devex. The idea is to integrate surveys into Bitbucket with customizable templates and questions on the most common challenges developers face at work to gather their feedback on the whole working environment. You can find more details on the vision here: https://link.stiltsoft.com/dev-surveys

However, we're unsure if surveys are the best way to measure developer experience and would love to hear your thoughts:

Do you measure developer experience in your company?
What tools or methods do you use to track developer experience?

https://redd.it/1ip9r2q
@r_devops

I promise this will improve your chances of getting more interviews

I made a website that converts your cv to match the job description automatically without manually copying and pasting your CV. Visit https://cvconverter.replit.app/ to get started

https://redd.it/1ipbodj
@r_devops

SRE Interview Questions

I work at a startup as the first platform/infrastructure hire and after a year of nonstop growth, we are finally hiring a dedicated SRE person as I simply do not have the bandwidth to take all that on. We need to come up with a good interview process and am not sure what a good coding task would be. We have considered the following:

* Pure Terraform Exercise (ie writing an EKS/VPC deployment)
* Pure K8s Exercise (write manifests to deploy a service)
* A Python coding task (parsing a lot file)

What have been some of the best interview processes you have went through that have been the best signal? Something that can be completed within 40 minutes or so.

Also if you'd like to work for a startup in NYC, we are hiring! DM me and I will send details.

https://redd.it/1ipcn2i
@r_devops

Struggling with Docker Rate Limits – Considering a Private Registry with Kyverno

I've been running into issues with Docker rate limits, so I'm planning to use a private registry as a pull-through cache. The challenge is making sure all images in my Kubernetes cluster are pulled from the private registry instead of Docker Hub.

The biggest concern is modifying all image references across the cluster. Some Helm charts deploy init containers with hardcoded Docker images that I can’t modify directly. I thought about using Kyverno to rewrite image references automatically, but I’ve never used Kyverno before, so I’m unsure how it would work—especially with ArgoCD when it applies changes.

Some key challenges:

1. **Multiple Resource Types** – The policy would need to modify Pods, StatefulSets, Deployments, and DaemonSets.
2. **Image Reference Variations** – Docker images can be referenced in different ways:
* [`docker.io/distribution/distribution`](https://docker.io/distribution/distribution)
* `distribution/distribution`
* `alpine` (which actually maps to `library/alpine`, so I’d need to account for that).
3. **Policy Complexity** – Handling all these cases in a single Kyverno policy could get really complicated.

Has anyone tackled this before? How does Kyverno work in combination with ArgoCD when it modifies image references? Any tips on making this easier?

https://redd.it/1ipdwpo
@r_devops

Do you bump helm chart version manually?

So currently im bumping my helm chart versions manually, the version is the same as appVersion, and the appVersion is set also manually whenever i push to github; i have release-please creating a new PR and then i manually sync that version in my chart.

I feel like this can be automated but i don't know how? is there a tool that does this?

https://redd.it/1ipf8ca
@r_devops

Is yq available in Cka ?

So far, I’ve only used yq instead of jq or kubectl jsonpath, and it worked fine in KodeKloud labs, Killercoda, and Killer.sh.

I assumed it would be available in the exam as well, but after reviewing the guidelines, I noticed that only jq (and some other tools) are explicitly mentioned as configured—yq is not.

Can anyone who has taken the exam confirm whether yq is available?

Thanks!



https://redd.it/1ipgfeb
@r_devops

Deploying via GitHub code runner running on target server?

I'm taking over the lead on development of a node.js project at work and want to automate where I can to make things easier and less tedious for myself. The former lead had been manually updating the builds and restarting the pm2 service outside of work hours, and that is something I can't abide if there's any other way.

I was able to get automated deployment working on our demo server by making a batch script which runs as a cron job and does everything manually, but it's not very robust and I realize I probably shouldn't try to reinvent the wheel here. I've been looking into GitHub Actions, which until now I've only really used with GitHub's provided code runners, and I've seen some tutorials that say to have the code runner actually run on the deployment machine. I'm curious if this is an intended use case and good practice or if it's a hack, since these tutorials were all made by individuals and I didn't see any documentation on GitHub suggesting deployment this way.

I've also seen some tutorials that say to use ssh/scp actions to build via a runner from anywhere and then send the builds to the target server and have it run a script to restart services, but for my use case this would require sending about 1GB of dependency packages and also the target server does not have ssh access outside of a VPN so it wouldn't even be possible without having IT make some networking changes which they may not even approve.

So my question is, would it be an appropriate use of a code runner to have it run on the actual server the app runs on? Additionally, is there a way to have a workflow that is both triggered by a git push AND waits until a specified time to execute (our app is generally only used in our time zone and we want to update during a time where it is less likely to disrupt service), or if I go this route should I have the code runner instead run on a schedule and exit early if there are no new commits? Any feedback would be greatly appreciated!

https://redd.it/1ipim82
@r_devops