I’m co-founder at SigNoz - an open-source Datadog alternative with over 22k Github stars. Ask Me Anything! [AMA]

Hey r/devops!

I am Pranay, one of the co-founders of [SigNoz](https://github.com/SigNoz/signoz), an opentelemetry native observability tool that provides APM, logs, traces, metrics, exceptions, alerts, etc. in a single tool.

A bit on how and why we started SigNoz:
4 years back, I and my co-founder, Ankit, identified a gap in observability tooling. There was a huge difference between what was available in open source vs proprietary tools. We thought there should be much better tooling available in Open Source. There was none available, hence we started building one.

We applied with this idea to YCombinator and were selected.

4 years from then we now have a much more mature product, many users using the product every day and Github repo with 22K stars (vanity metric), but atleast it shows it has got some interest.

Not here to sell anything, but thought our journey may be interesting to some and might insipire the next set of ppl. Feel free to ask me anything about building and maintaining SigNoz, observability practices, etc. A few things in my mind that we can talk about:

- engineering and technical questions around SigNoz
- existing and upcoming features
- Building and maintaining an open-source project
- existing observability landscape, your pain points, etc.
- state of opentelemetry and its future

or anything related to observability in general. SigNoz is now being used by engineering teams at companies of all sizes, so I can definitely help you with questions around your observability set up.

I will start answering questions from 9:30 am PT (11th June, Wednesday). Leaving it here now so that folks from other timezones can leave their questions. Looking forward to a great chat.

To prove that I am real and not an LLM bot :) : https://www.linkedin.com/posts/pranay01_if-youre-on-reddit-i-am-doing-a-reddit-activity-7338425383240773634-dz6V

https://redd.it/1l8jrc2
@r_devops

how do you stay efficient when working inside large, loosely connected codebases?

I spent most of this week trying to refactor a part of our app that fetches external reports, processes them, and displays insights across different user dashboards.

The logic is spread out
– the fetch logic lives in a service file that wraps multiple third-party API calls
– parsing is done via utility functions buried two folders deep
– data transformation happens in a custom hook, with conditional mappings based on user role
– the UI layer applies another layer of formatting before rendering

None of this is wrong on its own, but there’s minimal documentation and almost no direct link between layers.
Tho used blackbox to surface a few related usages and pattern matches, which actually helped, but the real work was just reading line by line and mapping it all mentally

The actual change was small: include an extra computed field and display it in two places. But every step required tracing back assumptions and confirming side effects.

in tightly scoped projects, I guess this would’ve taken 30 minutes. and here, it took almost two days

what’s your actual workflow in this kind of environment?
do you write temporary trace logs? build visual maps? lean on tests or rewrite from scratch?
I’m trying to figure out how to be faster at handling this kind of loosely coupled structure without relying on luck or too much context switching

https://redd.it/1l8julj
@r_devops

PSA- MS have expired cert on onegetcdn.azureedge.net

As title says, MS cert expired a few hours ago and pipelines with Power Platform Tool Installer task may fail when trying to connect to this shared CDN service: unable to get NuGet

Have raised sev1 with MS and they’re investigating and hopefully will resolve soon…


https://redd.it/1l8madc
@r_devops

How to get started with observability as a developer?

Hi,

I am a backend developer looking to learn and implement observability.

What would be a good starting point on the domain language around observing applications?

How does observability and alerting fit into product architecture?

What would be some good and robust open source tools to perform observation?

https://redd.it/1l8nt8v
@r_devops

Built a simple SSH jump tool (sshop) for managing many client/server combos

Hey all!

I built sshop, a lightweight CLI helper that lets you pick a client → server from a structured JSON config file, and SSH into it instantly. Reason for building this was my own struggle with managing many clients with dev/stage/prod environments.

Under the hood it uses fzf \+ jq for fast, interactive selection, and allows for adding, updating and deleting of servers via CLI flags.

I made it open-source, and I'm curious if others find it useful or have any feedback or suggestions.

Repo with more info can be found here: https://github.com/Skullsneeze/sshop

https://redd.it/1l8qino
@r_devops

Built a tool to stop wasting hours debugging Kubernetes config issues

Spent way too many late nights debugging "mysterious" K8s issues that turned out to be:
- Typos in resource references
- Missing ConfigMaps/Secrets
- Broken service selectors
- Security misconfigurations
- Docker images that don't exist or have wrong architecture

Built Kogaro to catch these before they cause incidents. It's like a linter for your running cluster.

Key insight: Most validation tools focus on policy compliance. Kogaro focuses on operational reality - what actually breaks in production.

Features:
- 60+ validation types for common failure patterns
- Docker image validation (registry existence, architecture compatibility, version)
- Structured error codes (KOGARO-XXX-YYY) for automated handling
- Prometheus metrics for monitoring trends
- Production-ready (HA, leader election, etc.)

Takes 5 minutes to deploy, immediately starts catching issues.

Latest release v0.4.2: https://github.com/topiaruss/kogaro
Demo: https://kogaro.dev

What's your most annoying "silent failure" pattern in K8s?

https://redd.it/1l8qwyq
@r_devops

Anyone else learning Python just to stop copy-pasting random shell commands?

When i started working with cloud stuff, i kept running into long shell commands and YAML configs I didn’t fully understand.

At some point I realized: if I learned Python properly, I could actually automate half of it ...... and understand what i was doing instead of blindly copy-pasting scripts from Stack Overflow.

So I’ve been focusing more on Python scripting for small cloud tasks:
→ launching test servers
→ formatting JSON from AWS CLI
→ even writing little cleanup bots for unused resources

Still super early in the journey, but honestly, using Python this way feels way more rewarding than just “finishing tutorials.”

Anyone else taking this path — learning Python because of cloud/infra work?
Curious how you’re applying it in real projects.

https://redd.it/1l8uhvk
@r_devops

8 YOE all at the same company Is my resume senior-worthy at a tech company?

Hey all,

I’ve been working full-time for over 8 years at the same Fortune 500 non-tech company (and interned at a different one prior to that), but I’m finally ready to look elsewhere because of being what I perceive as underpaid relative to the value I can create. Here’s my anonymized resume:

https://imgur.com/a/nd3T1MA

I’ve been in 4 different organizations within the company, but I can’t tell whether I am actually going to get looks at FAANG-adjacent companies or if I’m wasting my time by going through the application process. The bar is so low to meet expectations at my current company that I worry it’s made me soft/lazy/unattractive to more prestigious employers. I don’t want to get into a senior or staff interview and make an ass out of myself. What are your thoughts?

Thank you!

https://redd.it/1l8yyie
@r_devops

Change Log Creation

I added a step to my build process to generate a Changlog by using the commit messages by date before the last tag. Now facing an interesting decisión and want to get some suggestions. I can call the change log build task when I generate the release (on GitHub) and only make it part of the release. That’s option 1. Option 2, generate the change log on build and commit it back to the repository as part of the build process. I am not thrilled with either option but I want to make this as easy as possible, but it Alfredo’s dirty to commit as part of the build. I can do this as a pre-commit hook as well, not sure if that’s better but it will require some setup on the dev machine. What are you folks doing in a similar scenario? This is part of a generic build agent/pipline, I think I posted it on here already.

https://redd.it/1l8z1q2
@r_devops

Airflow: how to reload webserver_config.py without restarting the webserver?

I tried making edits to the config file but that doesn’t get picked up. Using airflow 2. Surely there must be a way to reload without restarting the pod?

https://redd.it/1l8yl06
@r_devops

Cloud DevOps mentorship/tutoring needed

Background
I am a msc it security student in Germany and btech computer science graduate from india, with multiple internship experience with full stack web dev. I have completed some course on docker and AWS cloud practitioner.

Expectations
I will complete my first year of msc in 3 more months after which I need to land job with a company to do my master thesis along with the company. I want to do it specifically in the intersection of cloud DevOps and security.

Requirement
I am looking for experienced cloud DevOps engineer (at least 1 years), who can get me interview ready to land a job for such roles. I only have 3 months to land a job so the duration of the contract will also be 3 months. I specifically want to learn in depth about Kubernetes, observability and infrastructure as code (terraform).

Bonus
If someone also can teach me potential security aspects of cloud DevOps and a potential master thesis in this field that would very beneficial for me.

Pay: up to 12 euro per hour

https://redd.it/1l93ej2
@r_devops

What do you use to automate self-healing scripts?

Hey everyone! just asking this to see if I'm missing something or the hereditary blindness already got me.
The thing is, I've been a DevOps engineer for about 5–6 years in two different companies, and in both of them, my main task was creating auto-remediation/self-healing scripts that run automatically when a monitoring tool detects something, like a spike in CPU, swap usage, low disk space, and so.

For that whole pipeline, I've been using a mix of Python/Go/Shell (sensible scripts), orchestrated by Rundeck/Jenkins/n8n/Tower as the executors, and Grafana/Datadog or similar tools for monitoring.

So my question is: is there anything dedicated to this? I mean, a tool that, when a monitoring metric hits a threshold, can automatically trigger something on a machine or group of machines?

https://redd.it/1l956jb
@r_devops

Developer cheat sheet

I created this free cheat sheet for cli commands.

I tend to prefer to invoke commands in my IDE vs GUI.

This is free.

If there is anything you want me to add please let me know.


Https://devcheatsheet.io

https://redd.it/1l95236
@r_devops

Automate adding vCluster to Argo CD using External Secrets Operator - GitOps

A blog post about how to automate provisioning virtual clusters (vCluster) using External Secrets Operator. Basically, when vCluster is created, it will be added automatically to Argo CD using External Secrets PushSecret and ClusterSecretStore.

Automate adding vCluster to Argo CD using External Secrets Operator

Enjoy :-)

https://redd.it/1l973i6
@r_devops

Best way to structure a new Azure DevOps pipeline for Playwright tests?

Hi everyone, I could use some help structuring a test pipeline in Azure DevOps using Playwright. My team used to work with Cypress, but we’re currently migrating to Playwright. The thing is, we never had a dedicated pipeline for automated tests, only build and deploy pipelines for the dev team, which were recently moved to another Azure DevOps project.

Now we want to create a separate pipeline specifically for testing, and I’m unsure of the best approach: should I create a brand-new YAML file just for the Playwright tests? Or try to reuse the old pipeline structure (even though it’s from another project and wasn’t built for testing in the first place)?

I’m looking for advice on what would be the best practice here, especially in terms of long-term organization and maintainability. If anyone has been through a similar migration, I’d really appreciate your insights. Thanks!

*E2E tests

https://redd.it/1l984wd
@r_devops

Secure s3 dashboard/website

Hi everyone. I am loosing my mind over what seems to be a simple problem.

So basically, I created internal dashboard (website stored in private s3). I have internal route53 record to use with it if needed, and internal ALB.
What i can't figure out is how to restrict access to it to only users behind the VPN. I tried CloudFront but the problem is that VPN uses split tunnel and public IP doesn't change, so WAF, lambdas, etc do not work.

What are my options to control access to this dashboard to selected users (preferably ones behind VPN without extra layers to login)

https://redd.it/1l9cd6d
@r_devops

Ode to the sysAdmin

Did the world forget that Systems Administrators existed before heirachical power structures?

- Customer support
- Engineer
- Architect

The architect’s role is to understand the shape of the bridge the customer needs, and the engineer builds the bridge.

If an Architect is expected to play Engineer, asked to build the bridge, whilst others were sabotaging the structure, who’s at fault?

The Architect?
The Engineer?
The 400 other people between,
Or the customer, which isn’t one, but many.

Please, think about that for a second.

A Domain Admin can never be asked to unsee what’s been seen.

We make sure others hold the same responsibility with the same honor, hoping that somewhere along the chain takes up enough of the slack to keep it together.

Systems Engineering isn’t easy.
Complex-Systems Architecture isn’t hard.

Meet me in the middle; or help me build the bridge.

https://redd.it/1l9f98n
@r_devops

Need a config management solution for structured per-item folders

I’m building a Python service that monitors various IoT devices (e.g., industrial motors, cold storage units).
Each monitored device has its own folder with all of its configuration inside:

A `.config` file with runtime parameters
A schema.json file describing the expected sensor input
A `description.txt` file that explains what this device does and how it's monitored

Here is the simplified folder strucure:

`project/`

`├──` [`main.py`](https://main.py)

`├──` [`loader.py`](https://loader.py)

`├── devices/`

`│ ├── fridge_a/`

`│ │ ├── config.config`

`│ │ ├── schema.json`

`│ │ └── description.txt`

`│ ├── motor_5/`

`│ │ ├── config.config`

`│ │ ├── schema.json`

`│ │ └── description.txt`

`│ └── ...`


What I’m Looking For:


A web interface to create/edit/delete these device folders
Ability to store and manage `.config`, `schema.json`, and `description.txt`
A backend (self-hosted or cloud) my Python service can query to fetch this config at runtime



https://redd.it/1l9hr4a
@r_devops

How to trigger AWS CodeBuild only once after multiple S3 uploads (instead of per file)?

I'm trying to achieve the same functionality as discussed in this AWS Re:Post thread:
https://repost.aws/questions/QUgL-q5oT2TFOlY6tJJr4nSQ/multiple-uploads-to-s3-trigger-the-lambda-multiple-times

However, the article referenced in that thread either no longer works or doesn't provide enough detail to implement a working solution. Does anyone know of a good article, AWS blog, or official documentation that explains how to handle this scenario properly?

P.S. Here's my exact use case:

I'm working on a project where an AWS CodeBuild project scans files in an S3 bucket using ClamAV. If an infected file is detected, it's removed from the source bucket and moved to a quarantine bucket.

The problem I'm facing is this:
When multiple files (say, 10 files) are uploaded at once to the S3 bucket, I don’t want to trigger the scanning process (via CodeBuild) 10 separate times—just once when all the files are fully uploaded.

As far as I understand, S3 does not directly trigger CodeBuild. So the plan is:

S3 triggers a Lambda function (possibly via SQS),
Lambda then triggers the CodeBuild project after determining that all required files are uploaded.

But I’d love suggestions or working patterns that others have implemented successfully in production for similar "batch upload detection" problems.

https://redd.it/1l9j2gg
@r_devops

Projects for resume

Hi folks.
I have 2 yoe in IT and I want to proceed in devops. Now I have theory and a little hands on on devops tools like jenkins, ansible, docker, k8s. I have also taken some random codes from chatgpt and built their docker images using jenkins and applied k8s deployment in them.
So now I wanted to know if I can add these in my project or not?
Also if I want to contribute in open source then how to search regarding same?
Would also love to know if you can help me to know about some other project ideas.

https://redd.it/1l9ke1a
@r_devops

Anyone switch from Python to Golang for most of their day-to-day tasks?

I'm in a situation where there's a lot of teams that each use different Linux distributions and dealing with Python dependencies, venvs, etc... is becoming a royal PITA.

https://redd.it/1l9lqdm
@r_devops