The cost of services is on everybody’s mind right now, with interest rates rising, economic growth slowing, and organizational budgets increasingly feeling the pinch. But I hear a special edge in people’s voices when it comes to their observability bill, and I don’t think it’s just about the cost of goods sold. I think it’s because people are beginning to correctly intuit that the value they get out of their tooling has become radically decoupled from the price they are paying.

In the happiest cases, the price you pay for your tools is “merely” rising at a rate several times faster than the value you get out of them. But that’s actually the best case scenario. For an alarming number of people, the value they get actually decreases as their bill goes up.

And the reverse: you should take control of what you are responsible for

Did you ask yourself what is the good threshold for your alert setup?

I have worked on alerting system for more than 10 years in e-commerce or healthcare system. Setting good threshold(s) for an alert is very difficult and contentious.

There are countless challenges around incident investigations and reports. Aside from sensitive situations revolving around blame and corrections, tricky problems come up when having discussions with multiple stakeholders. The problems I’ll explore in this blog—from the SRE perspective—are about time pressures (when to ship the investigation) and the type of report people expect.

Prometheus OpenTelemetry support

Honestly, pull requests sound like a pretty sweet tool for collaborating on a shared code base. They are a huge success in the open source space, and looking at that success alone it’s not surprising that a lot of teams use a pull request-based process for themselves. On the other hand, there are a lot of voices out there highlighting how using pull requests as the default mechanism for collaboration can slow down your team and prevent you from getting changes into the hands of your users quickly and reliably. Patterns that worked well for low-trust open source communities, they say, didn’t translate well to teams where you know and trust all of your collaborators. Critics of pull requests often suggest alternative workflows that predate pull requests and even git and other distributed version control systems.

Tmate is a fork of tmux. It provides an instant pairing solution.

Ingestr is a command-line application that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.

Set up a development environment on any infrastructure, with a single command.

Measuring developer productivity is a difficult challenge. Conventional metrics focused on development cycle time and throughput are limited, and there aren't obvious answers for where else to turn. Qualitative metrics offer a powerful way to measure and understand developer productivity using data derived from developers themselves. Organizations should prioritize measuring developer productivity using data from humans, rather than data from systems.

Grafana Mimir is our open source, horizontally scalable, multi-tenant time series database, which allows us to ingest beyond 1 billion active series. Mimir ingesters use consistent hashing, a distributed hashing technique for data replication. This technique guarantees a minimal number of relocation of time series between available ingesters when some ingesters are added or removed from the system.

Unfortunately, we noticed that the consistent hashing algorithm previously used by Mimir ingesters caused an uneven distribution of time series between ingesters, with load distribution differences going up to 25%. As a consequence, some ingesters were overwhelmed, while the others were underused. In order to solve this problem, we came up with a novel algorithm, called spread-minimizing token generation strategy, that allows us to benefit from the consistent hashing on one side and from an almost perfect load distribution on the other side.

Uniform load balancing optimizes network performance and reduces latency as the demand is equally distributed among ingesters. This allows for better usage of compute resources, which leads to more consistent performance. In this blog post, we introduce our new algorithm and show how it improved ingesters load balancing in some of our production clusters for Grafana Cloud Metrics (which is powered by Mimir) to the degree that it’s now almost perfect.

This blog post describes Uber’s journey towards utilizing hardware efficiently via better load balancing. The work described here lasted over a year, involved engineers across multiple teams, and delivered significant efficiency savings. The article covers the technical solutions and our discovery process to get to them–in many ways, the journey was harder than the destination.

OpenStatus is open-source synthetic monitoring platform with beautiful status page and incident management. We are building it publicly for everyone to see our progress. We believe great softwares are built this way.

jnv is designed for navigating JSON, offering an interactive JSON viewer and jq filter editor.

Retina is a cloud-agnostic, open-source Kubernetes network observability platform that provides a centralized hub for monitoring application health, network health, and security. It provides actionable insights to cluster network administrators, cluster security administrators, and DevOps engineers navigating DevOps, SecOps, and compliance use cases.

Note: This blog post demonstrates how to create a lightweight Internal Developer Platform without relying on Backstage, while still empowering you and your developers with a self-service approach. By utilizing GitOps with Argo CD and leveraging Kubernetes labels, this method offers a streamlined and efficient solution for managing and deploying your infrastructure.