SRE in Practice: Error Budget Math, Burn Rates and Blameless Postmortems

"Five nines" sounds like ambition. In practice it's a number of minutes, and most teams have never written that number down. My SRE Enterprise Handbook — a 20-chapter interactive knowledge base plus a 20-lab practical workbook — starts with that arithmetic, because every other SRE practice depends on it.

SLI, SLO, SLA — in one sentence each

• SLI (indicator): a measurement. "The proportion of HTTP requests that returned 2xx/3xx in under 300 ms."
• SLO (objective): a target for that measurement over a window. "99.9% over 30 days."
• SLA (agreement): a promise to a customer with consequences if broken — always looser than the SLO.

The gap between the SLO and 100% is the error budget: the amount of unreliability you are allowed to spend.

The arithmetic

A 30-day month has 43,200 minutes; using the average month (30.44 days, 43,830 minutes):

error budget (minutes) = (1 − SLO) × minutes in window
99.9%  → 0.001  × 43,830 ≈ 43.8 min
99.99% → 0.0001 × 43,830 ≈ 4.38 min

That last row is why I built the handbook's interactive calculators. At 99.99%, a single manual deployment that takes five minutes to roll back has already blown the month. The SLO doesn't just describe reliability; it dictates process. At 99.9% a careful human can deploy. At 99.99% you need automated canaries and automatic rollback, because humans are too slow.

Burn rate: how fast you're spending

Burn rate is the ratio of the actual error rate to the rate that would exactly exhaust the budget over the window.

burn rate = observed error ratio / (1 − SLO)

SLO 99.9%  → budget ratio 0.001
observed   → 1.44% errors over the last hour
burn rate  → 0.0144 / 0.001 = 14.4×

A burn rate of 14.4 sustained for one hour consumes 2% of a 30-day budget — the classic threshold for paging a human. A burn rate of 6 over six hours (5% of budget) is another page; 1 over three days is a ticket. Multi-window alerts like these page for real fires and stay quiet for noise.

Error budget policy

Numbers only matter if they change behaviour. The handbook's policy matrix:

The four golden signals

1. Latency — distinguish successful from failed requests; a fast 500 is not good latency.
2. Traffic — requests per second, or the domain equivalent (renders per minute for CinePrompt).
3. Errors — explicit (5xx), implicit (200 with wrong content), and by policy (slower than SLO).
4. Saturation — how full the most constrained resource is: CPU, memory, a queue, a single SQLite writer.

Blameless postmortems

The handbook encodes a template rather than a lecture:

• Summary and impact — who was affected, for how long, how much budget burned.
• Timeline — timestamps from detection to resolution, reconstructed from logs, not memory.
• Root cause — via 5 Whys, stopping at a system property ("deploys had no automated rollback"), never at a person.
• What went well / what went badly / where we got lucky.
• Action items — each with an owner, a ticket and a due date.

The rule: if a human could make the mistake, the system allowed it. Fix the system.

Incident command

Triage playbooks define roles — Incident Commander (decides), Operations (hands on keyboard), Communications (status updates), Scribe (timeline) — so the person debugging isn't also writing Slack updates.

Applying it to my own projects

My deployed projects are small, but the vocabulary still helps:

• Nora Labs has an implicit availability SLO protected by design: every agent endpoint falls back to a deterministic engine, so an AI outage burns quality, not availability.
• CryptoEdge keeps min-instances=1 because its scheduler's SLI is "alerts evaluated every 5 minutes" — scale-to-zero would break it silently.
• Saturation is why Nora pins --max-instances=1: the constrained resource is SQLite's single writer.

See the SRE Handbook case study → · Back to the DevOps series