Observability stack on a budget: what to keep and what to drop

A full metrics + logs + traces stack can easily cost more than the application it observes. The lever is not “turn observability off”; it is deciding which signals are high-value and which are expensive noise, then applying sampling, retention and cheaper backends accordingly.

The minimum viable set I defend

• Metrics: the four golden signals (latency, traffic, errors, saturation) plus a small number of business metrics. Cardinality kept under control.
• Logs: 100 % of errors and audit events; everything else sampled or filtered at the source. Short hot retention.
• Traces: tail-based or probabilistic sampling that keeps the interesting requests (errors, slow ones) and discards the rest.

Everything beyond that is optional and must justify its cost.

What I routinely drop or demote

• High-cardinality labels that explode time-series count
• Debug-level logs in production
• Full request/response bodies except for a sampled subset
• Traces for every successful health-check or static-asset request
• Long retention of high-volume, low-value streams

Cheaper backends for the bulk

High-volume logs and metrics can live in cheaper stores (object storage + Athena/BigQuery style query, or a lightweight TSDB) while only the high-value subset stays in the expensive observability platform. The same principle as the logging article applies: keep the signal, move the noise.

Decision rule

If a signal has not been used in a real incident or debugging session in the last 90 days, its retention or sampling rate is a candidate for reduction. The monthly bill review includes the top observability cost lines and asks whether each still earns its keep.

Observability is not optional. Unlimited, unfiltered, long-retention observability is. A deliberately small, high-signal stack costs a fraction of the “collect everything” default and still lets the team find and fix problems.

I revisit the sampling rates and retention whenever a new service is added or when the observability bill grows faster than the application traffic. The goal is a stack that stays useful and stays affordable.