SLOs, Error Budgets, and Alerts That Don't Cry Wolf
100% is the wrong target — what to promise, what to wake up for, and the error budget as the truce between shipping and stability.
🎯 By the end you'll understand
The SLI → SLO → SLA chain and how to pick metrics that actually measure user pain. Why targeting 100% availability is irrational — and how an error budget turns that insight into an actual team policy. Why symptom-based alerting (burning your budget too fast) beats cause-based alerting (CPU > 80%), and how alert fatigue is itself an outage multiplier: every non-actionable page trains engineers to ignore pages. You'll define real SLOs for TraceLab, wire up alerts, and simulate 90 days of traffic to see both the budget math and the over-alerting trap in action.
KEY INSIGHTThe goal is not 100% — it's a promise you can keep
Chasing 100% uptime doesn't make your system more reliable. It makes you lie about what you can deliver, hoard your error budget by refusing to ship, and wake people up at 3 a.m. for things users don't notice.
Every production system fails sometimes. Hardware dies, deploys go wrong, the network hiccups. The engineering question is not "how do we never fail?" — it's "how much failure is acceptable, what kind, how fast do we detect it, and who gets woken up?" SLOs (service level objectives) are the rigorous answer to those questions. They are not metrics you look at in a dashboard; they are decisions made in advance about what acceptable looks like, so that the answers to "should we ship this?" and "is this page worth waking someone up for?" are policy, not gut feel.
An SLI is a measurement of user-visible behavior (latency, success rate, correctness). An SLO is the target you set on that measurement ("95th-percentile latency < 400 ms over a rolling 30-day window"). An SLA is an SLO wrapped in a contract with teeth — financial penalties if you miss it. Most teams only need SLOs. The error budget is the inverse of the SLO: if you promise 99.9% success, you have 0.1% failures to spend. Spend it wisely or lose it to incidents.
CORE CONCEPTSLI → SLO → SLA: the precision hierarchy
The three terms nest. Each one adds precision and stakes.
SLI — what you measure
A service level indicator is a quantitative measurement of a user-visible property. The canonical categories (from the Google SRE book, which coined this vocabulary) are: availability (fraction of requests that succeed), latency (how fast good responses come back), quality (were the results correct or degraded?), and throughput. The discipline starts here: pick SLIs that measure what users actually feel, not what's easy to instrument. CPU utilization is not an SLI. "Fraction of /api/runs requests that return 2xx within 500 ms" is.
SLO — what you target
An SLO is a target on an SLI: "that fraction will be ≥ 99.5% over a rolling 28-day window." Four things define a complete SLO: the SLI it targets, the threshold, the measurement window, and the population (all requests? only non-health-check traffic?). The window matters enormously — a point-in-time 99.9% is vacuous; a rolling 28-day window means a 1-hour outage at a 99% SLO consumes roughly 15% of your monthly budget (1 hr ÷ 6.72 hr budget) and is tracked against the promise you made last month.
SLA — what you contract
An SLA adds external obligation. If you miss the SLO by more than a specified margin, the customer gets a credit, a refund, or you face a breach-of-contract claim. AWS, GCP, and Azure all publish SLAs — their actual internal SLOs are set significantly higher (typically 1–2 nines above the SLA) so that normal variance doesn't trigger payouts. Many engineering teams never write a formal SLA; SLOs are the internal tool, SLAs are the commercial commitment on top.
Reliability is often quoted in nines of availability: 99% = 2 nines = 87.6 hr/year downtime; 99.9% = 3 nines = 8.76 hr/year; 99.99% = 4 nines = 52 min/year; 99.999% = 5 nines = 5.3 min/year. The jump from 3 to 4 nines costs roughly 10× in operational complexity and architecture spend. Committing to 5 nines for a TraceLab instance that one team uses would be absurd — and that absurdity is the point: the right SLO depends on user expectations, not engineering pride.
Choosing the right SLIs
The failure mode here is optimizing for what's easy to measure. Teams with good Prometheus infrastructure start writing SLOs on CPU, memory, and disk — but none of those directly measure whether users are happy. A request/response service should lead with availability (fraction of non-5xx responses) and latency (P95 or P99 response time). A data pipeline like TraceLab's ingestion path should add freshness (how old is the most recent ingested row?) and correctness (fraction of rows with valid schema). If CPU spikes but latency is fine and errors are within budget, you're not breaking your SLO — you don't need to page anyone.
CORE CONCEPTThe error budget: math that decides arguments
An error budget is the complement of an SLO. If you promise 99.9% availability over 30 days, you are also implicitly promising to use no more than 0.1% of requests as failures. With roughly 2.6 million requests per month (a modest service), that's 2,600 bad requests you're allowed — your "budget" of allowable failures. The budget is spent by incidents, bad deploys, flaky tests reaching production, and degraded-mode serving.
The policy implication is the part that changes engineering culture. If the budget is healthy (mostly unspent), you ship features aggressively. Your users aren't hurting. Spending some of that budget on risky deploys is fine — you have room. If the budget is nearly exhausted, you freeze features and harden. No new deploys until the budget recovers. No experiments. The engineering organization agreed in advance that this is how the decision gets made, so the argument between "we need to ship" and "we can't break anything" resolves without a management escalation. That's the truce.
Google's SRE Workbook introduced burn rate: how fast are you consuming your budget relative to the pace that would exactly exhaust it over the window? A burn rate of 1 means you'll just barely hit the SLO at the end of the period. A burn rate of 10 means you're consuming 10× the sustainable rate — you'll exhaust the budget in 1/10th the window. Multi-window burn-rate alerts (e.g., burn rate > 14.4 over 1 hour AND burn rate > 6 over 6 hours) are the modern standard because they fire on real emergencies and almost nothing else.
Error budgets also give you a principled answer to a question teams usually answer with vibes: how much reliability engineering should we do this quarter? If you've never come close to exhausting the budget, you might be over-engineering. If you're exhausting it every sprint, reliability work is load-bearing feature work — not optional hardening. The budget makes the tradeoff visible.
| SLO | Budget / 30 days | Budget / 30 days (at 100k req/hr) | What that means |
|---|---|---|---|
| 99.0% | 1.0% = 432 min (7.2 hr) of outage | 720,000 bad requests | A 40-min deploy incident barely dents it |
| 99.5% | 0.5% = 216 min (3.6 hr) of outage | 360,000 bad requests | One bad deploy burns half your month |
| 99.9% | 0.1% = 43.2 min of outage | 72,000 bad requests | A 30-min incident wipes the budget |
| 99.99% | 0.01% = 4.4 min of outage | 7,200 bad requests | One bad deploy = month's budget gone |
A common mistake: setting SLOs by ratcheting up until something impressive-sounding. The right process is bottom-up — ask "what level of degradation would cause a user to churn or complain?" and set the SLO just above the point of pain. Then check whether your architecture can actually deliver it. If it can't, either fix the architecture or lower the promise — but never paper over the gap with an aspirational number.
SEE ITError-budget simulator
TraceLab is running in production. You've chosen an SLO for the ingestion API. Below, 90 simulated days of traffic play out — incidents are pre-seeded including the over-alerting trap. Each week you make a decision: ship features (spends budget faster via risky deploys) or harden (reduces future incident probability). Your score is incidents caught vs. 3 a.m. pages wasted.
The canonical disaster this widget reproduces: a 2× burn-rate alert configured on a service with normal daily traffic spikes. The alert fires every morning at 08:00 when batch jobs kick off. Engineers silence it. When a real incident arrives at 3 a.m. in week 7, the alert fires — and nobody responds for 45 minutes, because the alert has trained them to expect a false positive. The incident burns three times the budget it would have if caught in the first 15 minutes. The fix is not a louder alert. It's a calibrated threshold that only fires on genuine sustained burns.
IN PRACTICESymptom-based alerting: wake up for users, not symptoms
There are two fundamentally different philosophies for production alerting. The first — cause-based — fires when a metric crosses a threshold thought to predict failure: CPU > 80%, memory > 70%, disk > 90%. The second — symptom-based — fires when user-visible behavior is degrading: error rate rising, latency spiking, budget burning too fast. Symptom-based is almost always better for on-call alerting because it answers the real question: "is a user hurting right now?"
| Alert type | Fires when | What it means for users | Worth waking for? |
|---|---|---|---|
| CPU > 80% | Batch job kicked off | Often nothing — if latency is fine | Rarely |
| Disk > 85% | Log rotation backlogged | Nothing yet — but maybe soon | Ticket, not page |
| Error rate > 0.5% | Bad deploy, DB timeout | Users are seeing 5xx errors | Yes |
| P99 latency > 2s | Slow query, GC pause | Users are waiting | Yes |
| Budget burn rate > 14.4× | Sustained severe incident | SLO breach in ~2 days at this rate | Yes — immediately |
| Budget burn rate > 6× (6 hr) | Gradual degradation | SLO breach in ~4 days at this rate | Yes — within the hour |
The multi-window burn-rate approach (Google SRE Workbook, Chapter 5) is worth understanding in detail. A single-window threshold fires when the current rate is high — but a brief spike resolves quickly and the page was noise. Combining a short window (1 hr) with a long window (6 hr) means the alert only fires when the burn is both severe and sustained. The 14.4× threshold means the budget would be exhausted in 30 ÷ 14.4 ≈ 2 days if the rate held — severe enough to page immediately and act before the window closes. The numbers are tractable arithmetic, not magic.
Alert fatigue is not a morale problem — it is an outage multiplier. Each non-actionable page erodes the Pavlovian link between "alert" and "real problem." Teams with noisy alert logs develop three coping behaviors, all dangerous: faster acknowledgment without investigation ("ack it and go back to sleep"), blanket silencing of entire alert categories, and slower response to novel alerts because they expect false positives. The DORA research on DevOps performance shows high-performing teams have dramatically lower change failure rates and dramatically fewer on-call alerts — the correlation is real.
Runbooks: alerts without runbooks are just noise
An alert that fires without telling the responder what to do is an alert that will be silenced. Every SLO-based alert should link to a runbook — a short document answering: what does this alert mean, what has caused it before, what are the first three diagnostic steps, what can you do in the middle of the night vs. what can wait until morning, and who to escalate to. Runbooks don't need to be elegant; a bulleted checklist in a shared wiki is fine. What matters is that the on-call engineer doesn't have to reason from first principles at 3 a.m.
# Fires if TraceLab is burning error budget at 14.4× rate for 1 hr # AND at 6× rate for 6 hrs — sustained AND severe groups: - name: tracelab.slo rules: - alert: TraceLab_SLO_Fast_Burn expr: | ( job:tracelab_errors:rate1h / on() group_left tracelab_slo_error_budget_rate > 14.4 ) and ( job:tracelab_errors:rate6h / on() group_left tracelab_slo_error_budget_rate > 6 ) for: 2m labels: severity: page annotations: summary: "TraceLab error budget burning fast — SLO breach within ~2 days at current rate" runbook_url: "https://wiki/runbooks/tracelab-slo-burn"
One more thing the alert config above makes explicit: the for: 2m. Prometheus requires a condition to be true for a sustained duration before firing. This is your first line of defense against transient spikes — a 30-second error burst at deploy time should not page anyone; a 2-minute sustained burn at 14× rate is real. Tune this duration per alert based on how long a genuine incident of that type needs to establish itself.
TRACELABDefining TraceLab's SLOs — v4 production commitments
TraceLab v4 is containerized, CI-built, canary-deployed, and instrumented (Lessons 5.1–5.7). Now it needs explicit promises. TraceLab's users are ML researchers and robotics engineers running experiments — the data they ingest is valuable, but they're not running payment systems. The right SLOs reflect that: meaningful commitments, not over-engineered ones.
Three SLOs cover the important user journeys:
What's not an SLO: TraceLab's CPU utilization, its container restart count, or its Postgres connection pool saturation. Those are symptoms to investigate after an SLO alert fires, not independent promises to users. This is the discipline: keep the SLO surface small and user-visible.
from prometheus_client import Counter, Histogram # SLO 1 — ingestion availability ingest_total = Counter( "tracelab_ingest_requests_total", "All POST /api/runs requests", ["status"] # label: "success" | "error" ) # SLO 2 — query latency query_latency = Histogram( "tracelab_query_latency_seconds", "GET /api/runs/:id end-to-end latency", buckets=[.05, .1, .2, .4, .8, 1.6, 4.0] # P95 <= 0.4s target; 0.4 bucket is the SLO boundary ) # SLO 3 — freshness (probed by a synthetic canary job) freshness_check = Counter( "tracelab_freshness_checks_total", "1-minute freshness probe results", ["result"] # label: "fresh" | "stale" ) # Error budget computation (PromQL, shown as comment): # availability = 1 - (rate(ingest_requests_total{status="error"}[28d]) # / rate(ingest_requests_total[28d])) # budget_remaining = (availability - 0.995) / (1 - 0.995)
The v4 milestone is complete when these three SLIs are live in Prometheus, the multi-window burn-rate alerts are configured against them, and the runbook wiki page exists. Then — deliberately — the Stage 5 final exercise breaks TraceLab's ingestion path (the scripted outage from Lesson 5.9's lab), and the learner must use these alerts and the observability stack from Lesson 5.7 to detect, mitigate, and postmortem the incident. The SLOs are not decoration; they are the nervous system of the scripted outage.
TraceLab v0's logging was fire-and-forget: no acknowledgment of whether writes succeeded, no way to know if the file was corrupt. The SLO machinery in v4 closes that debt explicitly — the ingestion availability SLO is the formal promise that "POST /api/runs succeeds," backed by measurement. The v0 debt table said "no history — edits and deletes are forever" (repaid in Stage 2) and "no locking" (repaid in Stage 3). The implicit debt of "no observable reliability promise" is repaid right here.
WEAKNESSWhere this goes wrong
The most common failure: SLOs defined by an SRE team in a document nobody reads, based on historical percentiles ("we were at 99.7% last year, so let's target 99.7%"). Historical-percentile SLOs don't ask "what do users actually need?" — they institutionalize whatever the system happened to deliver. If your service was unreliable last year, your SLO is now formally unreliable. Start from user expectations, not historical stats. The second most common failure: setting SLOs on things you don't measure (and discovering this only when you try to report on them).
Teams inherit alert configs from whoever set them up three years ago. Over time, each incident spawns a new alert to "make sure this never happens again" — without retiring the alerts that didn't catch it. The result is an alert log with 200 rules, most of which fire daily on meaningless variation, trained the team to check the log once a week, and miss the one alert that actually matters. A hard rule that works: if an alert fires more than once a week without action, it should be deleted or converted to a ticket. Alerts that don't cause action don't need to exist.
The error budget only changes culture if leadership agrees to honor the policy. A common failure mode: the budget is exhausted, engineering proposes a feature freeze, product overrides it because a deadline is approaching, an incident happens, the freeze-override is never revisited. The error budget is a policy, not a ceiling you approach and then ignore. If leadership won't honor it, you have a governance problem that a technical tool cannot solve — and it's worth naming that explicitly rather than pretending the SLO framework is working.
KEY INSIGHTHow this connects
- Observability (5.7) — the logs, metrics, and traces that feed your SLIs. SLOs without instrumentation are aspirational fiction; the Prometheus counters and histograms defined in this lesson are the bridge between raw signals and reliability promises.
- Mathematics of Failure (4-6) — availability arithmetic, MTBF/MTTR, and the nines composition math that sits under every SLO target. The 99.9% promise you just made is only credible if your architecture's composite availability (series components multiplying failure probabilities) can deliver it — and 4-6 is where you learn to check.
- Orchestration (5.9) — next lesson. Kubernetes introduces its own reliability surface (pod restarts, readiness probes, rolling deploys). The SLOs you just defined will behave differently under orchestration; 5-9 explains the mechanisms and concludes with the honest decision of whether TraceLab should use it at all.
SLOs are the connective tissue between the observability work of Stage 5 and the judgment work of Stage 6. Without explicit promises, "the system is healthy" is a feeling. With SLOs, it's a measurable claim — and that's the foundation of everything that comes after.
GLOSSARYKey terms introduced
- SLI (service level indicator)
- A quantitative measurement of user-visible service behavior — availability rate, latency percentile, freshness check pass rate. The raw material from which SLOs are made.
- SLO (service level objective)
- A target on an SLI, with a threshold, measurement window, and population. The internal promise your team makes and holds itself to.
- SLA (service level agreement)
- An SLO with external contractual consequences for breach. SLAs should be set safely below internal SLOs to provide a buffer.
- Error budget
- The complement of an SLO — the allowed fraction of failures. If the SLO is 99.5%, the error budget is 0.5% of requests. Spending it accelerates by incidents and risky deploys; the budget resets with the measurement window.
- Burn rate
- How fast the error budget is being consumed relative to the sustainable rate. A burn rate of 1 exactly exhausts the budget by end of window. A burn rate of 14.4 exhausts it in 1/14.4th of the window.
- Symptom-based alerting
- Firing alerts on user-visible symptoms (error rate, latency, budget burn) rather than internal causes (CPU, memory). Dramatically reduces false positives and alert fatigue.
- Alert fatigue
- The degraded response to alerts caused by a high ratio of non-actionable pages to real incidents. Trains responders to treat alerts as background noise — a direct outage multiplier.
- Runbook
- A short, actionable document linked from an alert: what the alert means, past causes, diagnostic steps, and escalation path. The difference between an alert that gets actioned and one that gets silenced.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What's the difference between an SLI, an SLO, and an SLA — and which one do most engineering teams actually need to define?›
An SLI is a measurement (e.g., fraction of requests that succeed). An SLO is a target on that measurement ("≥ 99.5% over 28 days"). An SLA is the SLO wrapped in an external contract with financial teeth. Most engineering teams need SLOs — the internal policy that drives daily decisions. SLAs are a commercial layer on top, typically set 1–2 nines below the internal SLO to provide a buffer. SLIs are meaningless without SLOs; SLOs are unmeasurable without SLIs.
2TraceLab's SLO is 99.5% availability over 28 days. The service receives ~50,000 requests per day. How many bad requests can it afford per day before the monthly SLO is at risk?›
Total monthly requests: 50,000 × 28 = 1.4 million. Error budget: 0.5% of 1.4 million = 7,000 bad requests per month. Daily share: 7,000 / 28 = 250 bad requests per day. A deploy that throws 5xx for 10 minutes at 50 req/min would consume 500 bad requests — two days of budget at once. That's the kind of arithmetic the error budget forces you to do before deploying.
3Explain why "CPU > 80% for 5 minutes" is usually a poor on-call alert, and what a better alert would look like for the same service.›
CPU saturation is a cause, not a symptom — it might not translate to any user-visible degradation (many services run at 80% CPU with normal latency). It creates false positives every time a batch job runs. A better alert is symptom-based: "error rate > 1% sustained for 2 minutes" or "P95 latency > 400 ms sustained for 2 minutes." These fire only when users are actually affected. CPU spike as a cause gets investigated after the symptom alert fires — it's a diagnostic signal, not an alerting signal.
4What is the error-budget policy decision that makes SLOs transformative — not just descriptive?›
The policy: if the error budget is healthy, ship features aggressively; if the budget is exhausted, freeze features and harden. This converts the abstract question "should we deploy this risky change?" into a decision made by policy rather than by whoever is most persuasive in the meeting. Product wants to ship; reliability wants to harden. The error budget is the agreed-upon referee. When leadership overrides the freeze, that's a governance failure — the policy only works if the organization commits to honoring it.