The Mathematics of Failure
Five 99.9% dependencies cap you at just above 99.5% — availability arithmetic, Little's Law, and the numbers that separate designing from decorating.
🎯 By the end you'll understand
How availability compounds in series and improves (at a cost) in parallel; what MTBF, MTTR, and an SLA's "99.9%" actually promise in minutes of downtime per month; why five 99.9% components in series give you only 99.5% composite availability; and Little's Law — the one queueing formula to internalize before capacity planning in lesson 6-2 builds on it. The failure-mode angle: engineers who can't do this arithmetic promise availability their architecture cannot deliver, and discover it in front of customers.
KEY INSIGHTAvailability compounds, and not in your favor
When you chain five components each promising 99.9% uptime, you have not built a 99.9% system. You have built a 99.5% system. That is four times more downtime than your slide deck claimed.
Every distributed system is a product of its parts' availability — literally. The probability that a chain of components is simultaneously operational is the product of their individual probabilities. This is not pessimism; it is basic probability. And the engineers who haven't internalized it are the ones making SLA promises to customers that their architecture cannot keep.
For n independent components in series, composite availability is their product: A_total = A₁ × A₂ × … × Aₙ. Two 99%s give you 98.01%. Five 99.9%s give you 99.5%. For components in parallel (active redundancy), failure requires all paths to fail: A_total = 1 − (1−A₁) × (1−A₂) × … . Two 99%s give you 99.99%. The formulas are not complicated. The discipline is using them before you commit to an SLA.
HOW IT WORKSComponents in series multiply failure probability
Consider TraceLab v2's data path for a single logged run: the robot agent calls the HTTP ingest endpoint, which writes through the queue, which is consumed by the writer service, which persists to the database, which is read by the dashboard. Five hops. If every component is independently 99.9% available — three nines, a respectable number — what is the composite availability?
# Five components, each 99.9% available components = [0.999] * 5 # Series: ALL must be up simultaneously composite = 1.0 for a in components: composite *= a print(f"Composite: {composite:.5f}") # 0.99501 print(f"Downtime/yr: {(1-composite)*8760:.1f} hrs") # 43.7 hrs
The result is 99.5% — not 99.9%. Your five-nines-per-component system delivers 99.5% end-to-end. That is roughly 43.7 hours of downtime per year, or 3.6 hours per month. Three nines would promise 8.7 hours per year. The gap between what you believed and what the math shows is where outage post-mortems are written.
The product formula assumes components fail independently. In real systems, they often don't: a flaky shared database degrades the ingest endpoint and the dashboard simultaneously; a network partition takes out the queue and the writer. Correlated failures make the math optimistic. The formula is a floor, not a ceiling.
The series law has a corollary that surprises engineers repeatedly: the weakest link dominates. If four components are 99.99% and one is 95%, composite availability is roughly 95%. No amount of heroic reliability work on the other four moves the needle materially. Your reliability budget is always limited by the floor of your dependency chain.
CORE CONCEPTRedundancy in parallel buys nines — at a price
If series multiplies failure probability, the answer is obvious: run components in parallel so that all paths must fail simultaneously for the system to go down. Two instances of the ingest API in active-active configuration, load-balanced, each at 99.9%: composite availability is 1 − (1 − 0.999)² = 1 − 0.000001 = 99.9999% — six nines, from two three-nines nodes.
Two nodes cost twice the compute. Three nodes cost three times. A three-nines component at 1× cost becomes, in theory, a nine-nines component if you run three redundant copies (1 − (0.001)³ = 0.999999999) — but in practice correlated failures mean you won't see nine nines, and you are still paying 3× for compute, maintaining three deployment targets, handling split-brain edge cases, and designing your load balancer not to be the single point of failure you just removed from the application tier. Redundancy is a cost, not a free upgrade.
The parallel formula also reveals the concept of a single point of failure (SPOF): any component with no redundant path. Introduce even one SPOF in an otherwise parallel system and you have lost the benefit of everything else. Five 99.99% parallel nodes mean nothing if they all share a single 99% database. The system's availability is then at best 99% — the SPOF's ceiling — not 99.99%.
def series(availabilities): result = 1.0 for a in availabilities: result *= a return result def parallel(availabilities): # fails only if ALL fail fail_prob = 1.0 for a in availabilities: fail_prob *= (1 - a) return 1 - fail_prob # Two 99.9% nodes in parallel print(parallel([0.999, 0.999])) # 0.999999 (six nines) # But if they share a 99.9% database (in series with the pair)... print(series([parallel([0.999, 0.999]), 0.999])) # 0.998999 (~99.9%) # Six nines of compute, three nines total. The DB is the SPOF.
This arithmetic is why real SRE work is mostly identifying SPOFs, not adding servers. Adding a 50th application server instance when your database is a single replica does not improve reliability at all. The formula tells you exactly where to spend the money.
DEFINITIONWhat 99.9% actually promises in minutes per month
SLA numbers are expressed as fractions of uptime, but engineers think in downtime. The translation is simple: downtime = (1 − availability) × period. Here is what the common nines actually promise:
| Nines | Availability | Downtime / year | Downtime / month | Downtime / week |
|---|---|---|---|---|
| One nine | 90% | 36.5 days | 72 hrs | 16.8 hrs |
| Two nines | 99% | 3.65 days | 7.2 hrs | 1.68 hrs |
| Three nines | 99.9% | 8.76 hrs | 43.8 min | 10.1 min |
| Four nines | 99.99% | 52.6 min | 4.38 min | 1.01 min |
| Five nines | 99.999% | 5.26 min | 26.3 sec | 6.05 sec |
| Six nines | 99.9999% | 31.5 sec | 2.63 sec | 0.6 sec |
Three nines — the most commonly sold SLA — means 43.8 minutes of downtime per month. That sounds acceptable until you realize you also need to count your deployment window, your database maintenance window, and your dependency's maintenance window against that budget. If your cloud provider's managed database is itself three nines, and you deploy once a month with a 15-minute rolling restart, you have spent 15 of your 43.8 minutes before writing a single line of buggy code.
MTBF (Mean Time Between Failures) measures how often a component fails. MTTR (Mean Time To Recovery) measures how quickly you restore service when it does. Availability in steady state is MTBF / (MTBF + MTTR). To achieve three nines on a system that fails once a day (MTBF = 1440 min), you need MTTR ≤ 1.44 minutes. To achieve four nines with the same failure rate, you need MTTR ≤ 8.6 seconds. This is why operational tooling — deployment pipelines, runbooks, on-call escalation, automated rollback — is an architectural concern, not an operational afterthought. Your MTTR is as much an architectural property as your MTBF.
An SLI (Service Level Indicator) is the measured thing: request success rate, p99 latency, error rate. An SLO (Service Level Objective) is your internal target: "99.9% of requests succeed, measured over 30 days." An SLA (Service Level Agreement) is the contract with a customer, typically with a penalty clause. Most teams fail by writing SLAs without first confirming their architecture can hit the corresponding SLO. The math in this lesson belongs at the SLO-definition stage, before any customer sees the number.
SEE ITAvailability calculator — chain TraceLab's components
Build TraceLab's architecture one component at a time. Each component has a tunable availability expressed in nines (from one to five). Arrange them in series, add redundancy with parallel copies, and watch the composite availability and expected downtime per year update live. Try to hit 99.95% and discover what it actually takes — and what it costs.
Two things the widget will teach you if you play with it honestly. First: hitting 99.99% on a five-component chain requires each component to be around 99.998% — which usually means redundancy at every tier, not just the frontend. Second: every redundant instance you add also appears in the cost counter, which is the real constraint. Availability is a purchasing decision as much as an engineering one.
CORE CONCEPTLittle's Law — the one queueing formula to internalize
Availability arithmetic tells you whether a system is up or down. Little's Law tells you how a system behaves when it is up but under load — and failing gracefully under load is what actually separates production systems from demos.
L = λ × W
Where L is the average number of items in a system (queue depth + items being processed), λ (lambda) is the average arrival rate (items per second), and W is the average time an item spends in the system (wait time + service time). If any two are known, the third follows. The law applies to any stable system in equilibrium — a database connection pool, a task queue, a load balancer's request buffer, even a ticket queue at a conference. It requires no assumptions about the probability distribution of arrivals or service times.
Here is the version that matters for capacity planning. TraceLab's ingest queue receives robot telemetry at 200 events/second. Each event spends an average of 50 milliseconds in the system (5 ms queue wait + 45 ms processing). Little's Law says the steady-state queue depth is:
# TraceLab ingest queue under steady load lam = 200 # events/second (arrival rate) W = 0.050 # seconds per event (avg time in system) L = lam * W # Little's Law: L = λW print(f"Avg items in system: {L:.0f}") # 10 # If a batch job doubles arrival rate temporarily: lam_peak = 400 L_peak = lam_peak * W print(f"Peak queue depth: {L_peak:.0f}") # 20 — manageable # If W spikes to 500 ms (slow consumer, GC pause, etc.): W_slow = 0.500 L_slow = lam * W_slow print(f"Slow consumer queue: {L_slow:.0f}") # 100 — growing fast
The insight is that queue depth grows linearly with both arrival rate and service time. When a consumer slows down (a GC pause, a slow query, a saturated downstream), W rises, which means L rises, which means memory pressure rises, which can slow the consumer further. This feedback loop is how well-designed systems cascade into overload. The engineering response is to bound the queue length — when it hits the bound, apply back-pressure or shed load, rather than let memory grow unbounded.
Lesson 6-2 (capacity planning) builds directly on this. Before you can reason about how many database connections to provision, how large to make a work queue's buffer, or when to start a second consumer, you need a model that links load, latency, and queue depth. Little's Law is that model. The equation is simple; using it correctly — accounting for the difference between steady-state and transient behavior, and for the fact that W includes both wait time and service time — takes practice. Start now.
The one thing Little's Law cannot tell you: what happens when the system is not stable — when λ exceeds service capacity. In that regime the queue grows without bound, W grows with it (because wait time grows), which makes L grow faster, which is the definition of overload. The correct architectural response is not to provision for the absolute peak (prohibitively expensive) but to design explicit back-pressure mechanisms that reject or shed load gracefully before the system tips into this regime. This is what lesson 4-4's queue design work prepared for.
TRACELABWhat these numbers mean for TraceLab v3
TraceLab v2 has five real components in its critical write path: the HTTP ingest endpoint, the RabbitMQ queue, the writer consumer, the PostgreSQL database, and the dashboard read path. Before drawing the v3 target architecture, let's run the availability budget honestly:
# TraceLab v2 actual availability profile (conservative estimates) components = { "ingest_api": 0.999, # single Flask process, no HA "rabbitmq": 0.999, # single broker, no clustering "writer": 0.998, # single consumer, restarts slowly "postgres": 0.9995, # managed DB with automatic failover "dashboard": 0.999, # single web server } composite = 1.0 for name, av in components.items(): composite *= av print(f"After {name}: {composite:.5f}") downtime_hrs = (1 - composite) * 8760 print(f"\nComposite: {composite:.5f} = {downtime_hrs:.1f} hrs downtime/yr") # Composite: 0.99451 = 48.1 hrs downtime/yr # v2 can only promise ~99.4% — not the 99.9% we'd like to advertise
The v3 milestone addresses this in a concrete, cost-justified way. Rather than adding redundancy everywhere (expensive, complex), we target the two weakest links: the writer consumer (lowest individual availability at 99.8%, and the longest MTTR because restarts require manual intervention) and the ingest API (99.9%, but a SPOF with no graceful degradation path).
The v3 changes: run two writer consumers with idempotency (lesson 4-4's work) so a single-consumer restart doesn't drop events; put a simple health-check-guarded restart on the ingest API so MTTR drops from minutes to seconds. With these changes:
# After v3 reliability changes components_v3 = { "ingest_api": 0.9995, # auto-restart: MTTR drops from 5m to <30s "rabbitmq": 0.999, # unchanged — acceptable for our SLO "writer": 0.9997, # 2 consumers: parallel([0.999, 0.999]) ≈ 0.999999 # but shared queue limits to ~0.9997 effective "postgres": 0.9995, # unchanged (already managed HA) "dashboard": 0.9995, # cache added (lesson 4-3) absorbs DB unavailability } composite_v3 = 1.0 for a in components_v3.values(): composite_v3 *= a print(f"v3 composite: {composite_v3:.5f}") # 0.99720 ≈ 99.72% print(f"Downtime/yr: {(1-composite_v3)*8760:.1f} hrs") # ~24.5 hrs # Still not 99.9% — the queue is the next weakest link
The queue is now the next weakest link. Clustering RabbitMQ is expensive and operationally complex. The real v3 decision is to accept 99.72% availability (~24.5 hrs/yr downtime) and document it explicitly in the design doc, rather than silently promise 99.9% and let customers discover the gap. That is what "recorded decisions" means.
The original v0 CSV logger (lesson 1-1) had no durability guarantees: a crash mid-write left a corrupt row, and concurrent writers sheared rows apart. These are availability debts the architecture is still paying. The expand-contract migration in this stage (the v0 CSV debt to PostgreSQL, paid in lesson 4-5's migration work) is an availability improvement as much as a schema improvement: a database with WAL and transactions offers materially higher write durability than an unguarded CSV append.
WEAKNESSWhere this math gets ignored — and what it costs
The most common failure is writing "99.9% uptime" in a contract because it sounds like the right number, without ever running the series formula. The team then discovers — during a customer escalation — that their five-component chain can only deliver 99.5% by construction. The fix is not engineering; it is architecture. You cannot reliably hit 99.9% end-to-end by tuning individual components that are themselves 99.9%. You need fewer hops, redundancy, or a lower SLA commitment. Every SLA number should trace back to an availability calculation that your architecture can actually sustain.
Adding N instances in parallel improves the availability formula, but it also: doubles operational complexity (you now must handle split-brain, partial failures, and distributed state), requires a load balancer (which is now a new SPOF unless it is also redundant), and multiplies cost. Many teams add a second application server instance and declare the system "highly available" without noticing that the database is still a single point of failure and the load balancer is now the single bottleneck. The math shows you exactly where redundancy helps; it does not tell you it is free or sufficient.
Little's Law uses average arrival rate and average service time. Real systems have high-variance arrival patterns (burst traffic, cron-job spikes) and high-variance service times (slow queries, GC pauses). When you plan capacity based on averages and ignore variance, you are dimensioning for steady state but operating in peaks. A queue that is comfortable at average load can saturate in seconds during a burst. The correct response is to measure and bound the 99th percentile of W, not just the mean, and to provision against the realistic peak, not the average. Lesson 6-2 does this calculation explicitly.
KEY INSIGHTHow this connects
- Cache design (4.3) — a cache is a parallel read path that improves availability for the read side: even if the database is temporarily unavailable, cached data can be served. The cache itself becomes part of the series formula for the uncached path.
- Queue design (4.4) — the ingest queue buffers arrivals against service time spikes; Little's Law explains exactly how queue depth grows under load, and why you need to bound it with back-pressure.
- Distributed systems reality (4.5) — the CAP theorem and failure modes of distributed components feed directly into the availability calculations here. When a component is "unavailable" in a distributed sense (partitioned, returning inconsistent data), its effective availability is lower than its uptime metric suggests.
- Security is a design property (4.7) — next: the same discipline of "treating non-functional properties as first-class architectural constraints" applied to security.
- Lesson 6.2 (Capacity planning) — Little's Law as introduced here is a prerequisite. Capacity planning operationalizes the queueing model with real traffic data and dimensioning decisions.
The theme of Stage 4 is that distribution is a cost, not a promotion. This lesson makes that cost legible: every hop in a series degrades availability, every redundant copy adds operational complexity, and every queue depth has a mathematical relationship to arrival rate and service time. Once these numbers are visible, architectural decisions stop being aesthetic and start being accountable.
GLOSSARYKey terms
- Availability
- The fraction of time a system or component is operational and able to serve requests. Expressed as a decimal (0.999) or percentage (99.9%).
- Series availability
- For components that all must be up simultaneously, composite availability is the product of individual availabilities. Each added component lowers the total.
- Parallel availability (redundancy)
- For components where the system works if any path is up, composite failure probability is the product of individual failure probabilities:
1 − ∏(1−Aᵢ). - MTBF
- Mean Time Between Failures. How often, on average, a component fails. Longer MTBF means more reliable hardware or software.
- MTTR
- Mean Time To Recovery. How quickly the system is restored after a failure. Availability = MTBF / (MTBF + MTTR). Improving MTTR (faster deployments, automated restarts) is often more tractable than improving MTBF.
- SLI / SLO / SLA
- Service Level Indicator (the measurement), Objective (your internal target), Agreement (the contract). SLAs should be derived from SLOs which should be derived from SLIs — not the other way around.
- Little's Law (L = λW)
- In any stable system, the average number of items in the system equals the average arrival rate times the average time each item spends in the system. Applies to queues, connection pools, service handlers, and any system where items arrive, wait, are served, and depart.
- Single point of failure (SPOF)
- Any component with no redundant path whose failure causes total system failure. A single unredundant component limits the entire system's availability to at most that component's availability, regardless of how resilient everything else is.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Five components, each 99.9% available, are in series. What is the composite availability, and approximately how many hours of downtime per year does that represent?›
0.999⁵ = 0.99501, or about 99.5%. Expected downtime per year: (1 − 0.99501) × 8760 ≈ 43.7 hours. Compare to the 8.76 hours that a single three-nines component would promise. The chain multiplies the downtime probability at each hop.
2You add a second instance of your ingest API in active-active configuration (both handle traffic, either can fail without service impact). Each is 99.9% available. What is the composite availability of the pair — and what new risk did you just introduce?›
The pair fails only if both fail simultaneously: 1 − (0.001 × 0.001) = 99.9999% — six nines. But you have introduced a load balancer as a new component in series, which is now a single point of failure if not itself redundant. You may also have introduced distributed state problems (session affinity, split-brain on shared caches). Each redundant tier adds series components around it.
3TraceLab's ingest queue receives events at 500/second. The average time an event spends in the system (wait + processing) is 80 milliseconds. What is the steady-state queue depth? What happens to queue depth if a slow consumer doubles the average processing time?›
L = λW = 500 × 0.08 = 40 events in the system on average. If processing time doubles, W rises to 160 ms, so L = 500 × 0.16 = 80 events. Queue depth scales linearly with service time. If the consumer slows enough that the system becomes unstable (arrivals exceed capacity), the queue grows without bound — Little's Law no longer applies because the system is not in steady state.
4A system's MTBF is 720 hours (fails once a month on average). What MTTR is required to achieve 99.9% availability? What does that imply operationally?›
Availability = MTBF / (MTBF + MTTR) → 0.999 = 720 / (720 + MTTR) → MTTR = 720 × 0.001 / 0.999 ≈ 0.72 hours = ~43 minutes. Operationally: you need to detect the failure, page the on-call engineer, and complete a full recovery — all within 43 minutes. If your deploy pipeline takes 20 minutes and your on-call response time is 15 minutes, you have 8 minutes for the actual fix. Three nines sounds generous until you do this calculation.