Do you spend time optimizing jenkins jobs?

Hey guys,

In our company we have a lot of jenkins jobs almost 400. Some are for deployments used by devs, others are our own for some metric and monitoring stuff.

My manager has been for the past 1-2 years has been focusing much on optimizing on creating common jobs for all the stuff to minimize this number of jobs. Even if they are 4-5 jobs of a type he asks us to create a common job to accumulate these 4 so that if change is required in all then we can change in just one place and everything will work fine. Initially I was involved in creating a common pipeline for all deployments, that went well, we did it. But now he is just asking us to "commonize" every repeating pair or part of jenkins jobs that he sees.

Is this relevant for devops? Will that help with anything? Or is he just trying to solve a problem that never existed? Do you take part in these activities? Will they ever help a devops engineer in any way? Will putting these things in your resume or cv, attract recruiters?


https://redd.it/1lku3k7
@r_devops

Grafana monitoring

Hello Folks,

Those who are using azure and grafana to visualize the data, how are you querying the data?
We are using SQL to fetch the data however the queries are running frequently and increases the sql usage, we want to avoid relying on SQL?
What is you approach?



https://redd.it/1lkvwt4
@r_devops

UK Thinking of moving from IT Field Engineer to DevOps

Hey folks,

Been in IT for about 12 years now, basically all I’ve ever done on my life. Started out in tech support and eventually moved up to IT Field Engineer. Still doing hands-on work, and while I enjoy it, I’ve been seriously thinking about shifting into DevOps.

Main reason? DevOps salaries here in the UK look a lot healthier than what I’m on right now, even if I had to start over as a Junior (vs experienced tech).

Due to expire later this year, I’ve got my AWS CCP (never managed to use it in any of my jobs though) and I’ve dabbled in Azure (VM's only) in the past through work. I’ve also done some homelab stuff using Oracle Cloud (free tier) nothing massive, but enough to get some knowledge.

I was considering doing a bootcamp to accelerate things, since I tend to pick up new tech pretty fast. But I’m not sure if it’s worth the investment or if I should just go the self-study route and build a portfolio or certs instead.

Also, curious about how DevOps folks are feeling about AI right now. Within my current role, I’m not too worried, I don’t see AI replacing that any time soon. But what’s your take? Is it changing the DevOps space already? I can feel if the company allows you to use it can be a good allied to work, when comes to makes scripts, etc. Boost on productivity.

Would love to hear any advice or experiences from others who made the switch. Cheers!

https://redd.it/1ll0i3s
@r_devops

Arachni/Codename-SCNR Shutdown

Arachni was a DAST scanner I had used in previous projects, I went looking for it earlier this year to find out it had been converted to a new project, Codename-SCNR owned by ecsypno.

Here is the origin story, taken from the wayback machine since their site is down:

Origin

Today when going to the site I discovered that it no longer exists:

ECSYPNO

And the only thing I could find was a somewhat cryptic post on twitter from the owner, stating "Ecsypno.com is closing shop for the foreseeable future due to sabotage of my personal and professional lives."

Anyone here a customer? I wonder what will happen to the software for people who have already paid. It was definitely a smaller commercial enterprise, so hopefully not too many orgs are impacted, but it is interesting nonetheless.

https://redd.it/1ll49ke
@r_devops

I hate existing doc tooling

I don't think this breaks community guidelines (I post here regularly), if I am please remove the post.

I'm increasingly frustrated with how documentation tooling stinks at striking a balance between being useable for non-technical users and being well suited for automation/compliance workflows. I'm considering putting a service together and have a [quick survey](https://forms.fillout.com/t/aZtDWSYiMrus) that could help me validate some ideas. Also welcome discussion below.

* Why does nobody tackle document localization?
* Why does every service expect data backups to be done with some half-baked manual export function?
* Aside from Confluence, most have no options for data residency.



https://redd.it/1ll6h22
@r_devops

Bare metal k8s interview questions, what will be asked?

Bare metal k8s interview questions, what will be asked? I said I know bare metal k8s, but Im familiar only cloud managed k8s, What kind of questions can I expect and how to answer them. Can anyone share some insights.

https://redd.it/1ll75yn
@r_devops

Boss encourages a culture of „fixing in prod“ and it drives me insane

Disclaimer: I’m not a native speaker, I apologize for any confusion.

I’m the „DevOps engineer“ in a kinda established start up (running for more than 6 years, not yet profitable, Series A in October 2023). Technically what we do is not DevOps, rather classic ops just with more chaos but that’s not the topic.

I am responsible of doing the prod deployments and more than half the deployments, it does not go through smoothly. Manual scale downs need to be done before, restarting pods, even sometimes I need to pull in engineers to tell me what’s wrong and then they manually create an index, run a database query or things like that.

After another today if botched deployments today, it pissed me off so much, I wrote a manifesto called „no cowboy ops manifesto“ and here’s the content:

Disclaimer: I used ChatGPT for phrasing but I explicitly fed it my thought before.

**Production is Sacred – Let’s Treat It That Way**

Purpose

We work on live, customer-facing systems. Every action in production reflects our reliability, trustworthiness, and respect for our users. This manifesto defines the **boundaries** we operate within during deployments and incidents.

> “Production is not a playground. It’s a place for professionals who prioritize safety, reliability, and shared responsibility.”
>

## Core Principles

### 1.Rollback First, Investigate Later

If an update breaks, we **roll back within 10 minutes** unless there’s a playbooked fix. Recovery is a process, not a race.

> “Don’t treat rollback as failure. It’s responsible engineering.”
>

### 2. No Manual Fixes Without Audit Trail

No:

- kubectl exec hacks
- Manual scaling or pod deletions
- Ad-hoc DB queries or index creations
- Queue drops or reboots

Unless:

- It’s part of an **approved, version-controlled procedure**
- Actions are **fully logged and explained** in the postmortem

### 3. Every Fix Must Be Reproducible

If it’s not automatable, it’s not production-ready.

Fixes made by hand must be:

- Committed to a Git repo
- Reviewed and explained in plain English
- Deployable via CI/CD

### 4. No Heroes, Only Teams

We value coordinated response over solo acts. If you “save prod” alone:

- You take on **postmortem ownership**
- You also take on **automation follow-up**
- You may still get challenged on why rollback wasn’t used

### 5. Visibility Over Secrecy

Manual interventions will trigger alerts/logs. That’s not to shame—it’s to **protect the system** and learn from every exception.

## **Your Checklist in Production**

- Issue during update? Roll back within 10 minutes.
- Need a manual fix? Stop. Is it versioned or pre-approved?
- Touching prod? Tell the team. No silent SSHs.
- Fixed something? Great. Now write it up and automate it.


Now by boss‘ response was this:

Strongly disagree!

* Generally: When we tell customers we want to do something, we should at all times do everything in our power to ensure that it actually happens. A Rollback means we did not do what we promised to do in a first place and it is for sure considered as failing our promise.

* Priority 1 is always to make an update successful.
We are not in a business where Maintenance windows can be granted multiple times a day or week and this is connected with a huge communication effort on all sides and should be handled very carefully. So if there is any way to proceed even with manual intervention it is encouraged to do whatever needed to be able to make that update a success.

* Rollback should be the last possible solution when everything else failed. Within a maintenance window a rollback can be decided within the last 10 minutes if until that point we were not able to successfully update.

* We should do everything (including Rollback if needed) to be back online before the end of our maintenance window, respecting the window is super important.

* In case we see that we will not hold our maintenance window, we should rollback.In case we need to Rollback, we need to make sure to gather as much information as

somehow possible before rolling back to make sure we understand the problem before the rollback is triggered.

* Fixing whatever was needed to do manually to make the update a success or in case of rollback fixing what caused it is always highest possible priority as a next action


———


I think this is not a way to run a stable environment and ist driving me crazy. I am in this business for over a decade and quite confident in my abilities and views but I would still appreciate your opinion and advice. Thanks and apologies for the wall of text. I tried to be as brief as possible without missing many details.


https://redd.it/1ll9ul3
@r_devops

Solution to re-run terminated AWS spot instances in CI jobs?

Hey guys,

I'm currently running a script every 15 minutes to re-run terminated jobs via Github API, but it's far from ideal and still missing some of the terminated workflows.

I saw this post from 3 years ago and was wondering if anyone has come up with a better solution by now.

Thanks!

https://redd.it/1llco45
@r_devops

📡 Anyone setting up HTTPS for JupyterHub? Here’s my method using Jupyter AI setup

Hi all,

I recently had to configure HTTPS for JupyterHub while working with Jupyter AI and wanted to share a working method in case anyone else is trying to do the same.

The process involved:

Generating self-signed SSL certs (or using Let's Encrypt)

Editing the JupyterHub config

Restarting with the right flags and paths

It took a bit of trial and error to get it stable, especially since Jupyter AI has some subtle differences in environment behavior.

Would love to hear how others secure their notebook environments — especially for production or collaborative setups.

#Jupyter #HTTPS #DevOps #SelfHosted #JupyterHub #Security #Tips

https://redd.it/1llf10d
@r_devops

📡 Anyone setting up HTTPS for JupyterHub? Here’s my method using Jupyter AI setup

Hi all,

I recently had to configure HTTPS for JupyterHub while working with Jupyter AI and wanted to share a working method in case anyone else is trying to do the same.

The process involved:

Generating self-signed SSL certs (or using Let's Encrypt)

Editing the JupyterHub config

Restarting with the right flags and paths

It took a bit of trial and error to get it stable, especially since Jupyter AI has some subtle differences in environment behavior.

Would love to hear how others secure their notebook environments — especially for production or collaborative setups.

#Jupyter #HTTPS #DevOps #SelfHosted #JupyterHub #Security #Tips

https://redd.it/1llf0up
@r_devops

How do you handle the glue between Java builds, Docker images, and deployment?

I'm curious how teams out there handle the glue code between building Java projects and getting them into production.

What tools are you using to build your Java projects (Maven, Gradle, something else)?

Once you build the JAR, how do you package it into a Docker image?

Are you scripting this with bash, using Maven plugins, or something more structured?

How do you push the image and trigger deployment (Terraform, GitOps, something else)?

Is this process reliable for you, or do you hit flaky edge cases (e.g., image push failures, ECS weirdness, etc)?

Bonus points if you're using ECS or Kubernetes, but any insights from teams with Java + Docker + CI/CD setups are welcome.

https://redd.it/1lli67l
@r_devops

Am I deploying to On-Prem right

# Context

I'm the all-rounder at my agency, handling development, DevOps, database administration, sys admin, as well as whatever else is needed when someone doesn't have the necessary skills available.

A colleague comes to me, having built a script (in TypeScript) that needs to run on a cron on a customer-controlled platform, specifically an RHEL VM on an on-premises server, for specific reasons (unimportant at this point, just need to accept this is not able to be changed).

# Problem

Most of my experience is building and deploying artifacts in a cloud environment for containerised services, so my experience with on-prem, non-containerised workloads is not too well honed.

Currently, the on-premises server is locked down to a VPN and accessible via SSH.

# Current Approach

My current approach is to use Ansible executed from a CICD runner (right now, there is some uncertainty about what CICD we will be using, so it's unclear if I need to get the runner to connect to the VPN or if I can request the runner be whitelisted).

This seems like the exact use case for Ansible, but due to my lack of experience with Ansible, I'm wondering if there are better options (by better options I don't mean using other tools like Chef, Puppet, Saltstack or something else, I mean specifically higher level)





https://redd.it/1llkb5p
@r_devops

A Developer Introduced a Real Bug to Fix an Imaginary One

I've seen it first hand. I was in a project that had endless stakeholder conflicts, and contradictory requirements kept landing on the dev team's plate.. By that time ofc all trust across the teams had eroded. Everyone (including devs, testers, legal, business) kept suspecting each other of screwing things up.

So.... developers started adding defensive code. Quiet fail-safes. "fixes" for problems that had not happened yet, juuust in case they came up in the future. One senior dev added a timeout to prevent a theoretical infinite loop. Except... that infinite loop was an intentional part of a legal feature to block fraud. This "fix" caused a regression, which triggered a crisis with leadership. All because someone tried to save the product from its own requirements.

In my opinion the core issue was that no one trusted the process. And when devs lose trust, they silently take over the requirements...and that’s when real bugs happen.

One solution? One empowered Product Owner who owns priorities, makes decisions, and protects devs from the chaos.

Anyone ever had to protect a product from its own requirements? Or worked with someone who “coded just in case”?

https://redd.it/1llo299
@r_devops

How are you running short-lived Docker containers for integration tests in Java apps?

I see a lot of people using Jib or Buildx for building Docker images and Helm/Terraform for deployment.

What about running containers during integration tests? For example, spinning up Postgres, Redis, Elasticsearch, or other services locally or in CI to test against?

Are you using docker run in CI scripts or custom bash logic?

Using something like Testcontainers?

Building your own test infra harness?

I'm curious what patterns you’ve seen work (or fall apart) when trying to reliably run and stop Docker containers from within Java-based test flows or CI pipelines.

Have you hit reliability or cleanup issues?

Thanks.

https://redd.it/1llq016
@r_devops

alternative to Signoz

My organization wants to adopt the API monitoring tool. The best one. We wanted to go forward with Signoz, but right now, Signoz doesn't provide user management, and it's not what we're looking for.

What are the alternatives for Signoz out there? Tell me all, even if they are paid one.

https://redd.it/1llr3ki
@r_devops

Distributed Logging Store?

Hi,
we are building a software (backend + app) for a large retailer with thousands of stores. Each store has its own server and therefore our backend has basically 10.000 instances distributed across the world.

When it is about logging we have two conflicting requirements and every second week we have a meeting around that:

1. All logs should be stored centralized for monitoring purposes and the costs must be acceptable. We have Elastic for that and expect a few Million Euro per year for logs. So we should not log too much.

2. When there is a bug we often get the complaint that the logs are not detailed enough. But we cannot add more logs, otherwise we would violate our cost constraints.


One idea is to have a system with decentralized log stores. Basically each server would have its own log server and store the stuff locally and the most important logs are also sent to elastic for central monitoring. But we need a way to connect with each store and run queries there. Do you know such a system to have decentralized log store, but with a centralized management hub? We don't want to connect to each server individually via remote desktor (they are windows btw).

https://redd.it/1lltexe
@r_devops

If you’re starting with AWS, focus on these 5 services

When I started learning AWS, I felt completely lost.

There were so many services, so much jargon, and no real roadmap. I kept bouncing between random tutorials and still had no idea how everything fit together.

What helped me most was focusing on a few key services that actually taught me how the cloud works at a basic level.

Here are five that made things start to make sense:

EC2
Taught me how virtual machines work in the cloud. Launching one, connecting to it, and running a basic app helped me understand compute in a hands-on way.

S3
This was my intro to cloud storage. Uploading files, managing folders, and setting permissions gave me a real sense of how cloud apps store data.

IAM
I used to get constant access errors until I spent time learning this. Once I understood users, roles, and policies, everything got easier.

RDS
Made working with databases much simpler. I didn't need to install anything locally, and I could finally connect apps to a managed database in the cloud.

Lambda
Running code without setting up a server felt like magic. It helped me understand how event-driven applications work and introduced me to automation.

While I was working through these, I made a simple system in Notion to stay organized, track what I was learning, and avoid getting overwhelmed.

What AWS service made things finally click for you? Always curious how others got started.

https://redd.it/1lltycb
@r_devops

Migrating 5PB from AWS S3 to GCP Cloud Storage Archive – My Architecture & Recommendations

# Migrating 5 petabytes of data from AWS S3 to Google Cloud Storage Archive is quite a complex project.

I’ve recently completed a detailed discovery and analysis phase and published an architecture and recommendations based on my findings.

I’d love to know: Do you think my recommendations make sense? Or do you have any suggestions or lessons learned from similar large-scale migrations?

https://medium.com/@rasvihostings/migrating-5-petabytes-from-aws-s3-to-gcp-cloud-storage-archive-a107634969eb

https://redd.it/1llytar
@r_devops

How to make DevOps projects to showcase my skills and learn?

I want to learn and showcase my skills but without collecting certificates or making a software application from scratch, what could be some ways to practice using docker, kubernetes, linux and all that stuff?

https://redd.it/1lm0vnx
@r_devops

Anyone running wide events in a sizeable codebase?

- What hurdles or wins did you hit while instrumenting them?
- Did they shorten MTTR or surface new insights (numbers welcome!)?
- How do you reconcile single-service wide events with the cross-service view you get from distributed tracing?

Success stories, horror stories, and hard metrics all appreciated.

https://redd.it/1lm1wbm
@r_devops