Observability that pays for itself — structured logging that helps on-call, picking between metrics / logs / traces, SLO-driven alerting, and the trace-id discipline that ties everything together.
6 articles · updated regularly
Someone added a single label to a counter to make a dashboard a little more useful. A week later the Prometheus server was using 40GB of RAM, queries timed out, and during the next incident the monitoring was down right when we needed it most. The label was user_id, and it had quietly created millions of time series.
A load test reported a 12ms p99 while real users were timing out. The numbers weren't faked. The benchmark just stopped sending requests whenever the system got slow, so it never measured the slowness. That's coordinated omission, and it quietly poisons most latency stats.
A trace is a tree of spans stitched together by a context that has to ride along every hop. The traces that break almost always break at a boundary where nobody propagated the headers. Here is the model.
Alerting on CPU and disk pages people for things users never notice. SLOs flip it around: define what 'working' means to a user, then spend an error budget against it.
The three pillars overlap, but they answer different questions. Putting the right signal against the right question is half the win.
Free-text logs are a write-only medium. Structured logs are searchable, correlatable, and worth half your debugging time back.