ProjekinAja. Software Engineering Stage 6 · Judgment Lesson 6.3
● Stage 6 · Lesson 6.3

Reliability Patterns: Retries, Circuit Breakers, and Load Shedding

Every retry is load you chose to add — taming metastable failure with backoff, jitter, breakers, bulkheads, and graceful degradation.

⏱ ~24 min read ⚡ Reliability Engineering 🔬 TraceLab v5

🎯 By the end you'll understand

Why a naively retrying system can self-perpetuate an outage long after its original cause has resolved — the signature of metastable failure. You will understand exponential backoff with full jitter as a load-spreading mechanism (not just politeness), retry budgets as the enforcement mechanism, circuit breakers as fast-fail gates that let a dependency breathe, bulkheads as blast-radius limiters, and load shedding plus graceful degradation as the last line before total failure. The widget lets you watch each defense claw the system back from collapse — and shows why the wrong order of enablement can leave you worse off than no defenses at all.

KEY INSIGHTA retry is load you chose to add

The overload that caused the failure is still there. You just added more requests on top of it and called it recovery.

Lesson 6.2 showed you how to find where capacity runs out. This lesson is about what happens when it does — specifically the failure mode that senior engineers lose sleep over, because it is both common and deeply unintuitive: the system breaks, the transient cause goes away, and the system stays broken because the recovery attempts are now the cause.

🔑 Hold onto this

A retry is not free. It is a new request, indistinguishable from a genuine one, arriving at a dependency that is already struggling. If 10,000 clients each retry three times — with no coordination, no delay — a momentary spike that would have passed in two seconds instead sees 30,000 requests land in the same two seconds. The dependency never gets a chance to drain its queue. The system tips into a state where it remains overloaded indefinitely, not because the original problem persists, but because the retry behavior continuously recreates the overload. That is metastable failure, and it is the reason every pattern in this lesson exists.

CORE CONCEPTMetastable failure: the overload that outlasts its cause

The term comes from a 2022 OSDI paper by Bronson et al. ("Metastable Failures in the Wild") that catalogued a class of production incidents at Facebook and others where incidents were not caused by insufficient capacity — the system had adequate resources — but persisted because a feedback loop kept the load artificially elevated. Understanding the anatomy saves you from treating symptoms instead of causes.

1

Trigger

A transient event spikes latency or error rates: a brief network partition, a GC pause, a single hot shard, a downstream deployment taking 10 seconds to restart. The trigger is small and temporary.

2

Amplification

Clients timeout and retry. Retries add to the in-flight load. The dependency's queue fills. Latency spikes further. Timeouts become more common, which trigger more retries. The amplification loop runs faster than the dependency can drain.

3

Sustained overload

The original trigger has long since resolved. But the system is now running at 300% of capacity, all of it synthetic retry traffic. The "real" requests cannot get through because the retries are consuming all available threads, connections, and CPU.

4

The trap

Attempts to fix the trigger — rolling the service, adding capacity — don't help, because capacity is not the problem anymore. The retry behavior is the problem. Until clients back off, the system cannot recover regardless of how much hardware you add.

📖 What makes it "metastable"?

In physics, a metastable state is one that is locally stable but not globally optimal — it persists without external energy input, but a small perturbation can dislodge it. A system in metastable failure is "stable" in the sense that it isn't immediately crashing, but every component is stuck in an unhealthy state: high latency, high error rates, high retry volume. It takes deliberate intervention (circuit breakers opening, load shedding, client-side backoff engaging) to "dislodge" it back to healthy equilibrium.

The defenses in this lesson attack different parts of this loop. Backoff + jitter reduce the amplification rate. Circuit breakers prevent clients from contributing to the problem while letting the dependency breathe. Bulkheads limit how far the overload can spread. Load shedding actively reduces what the server must process. Graceful degradation keeps the most critical functions running even when everything else is failing. Each layer, enabled in the right order, claws back a piece of the system.

HOW IT WORKSExponential backoff with full jitter

The simplest partial defense against retry storms: clients wait longer after each successive failure, and they spread out those waits randomly. "Exponential" means the wait ceiling doubles with each attempt. "Full jitter" means each wait is drawn uniformly from [0, min(cap, base * 2^attempt)] — not from [base * 2^attempt / 2, base * 2^attempt] (that is "decorrelated" but not full jitter). AWS's Marc Brooker wrote the canonical analysis in 2015; his finding: full jitter outperforms all other jitter strategies at reducing contention on a shared resource.

backoff.pyfull jitter, deterministic cap
import random, time

def backoff_delay(attempt: int, base: float = 0.1, cap: float = 30.0) -> float:
    """Full jitter: uniform[0, min(cap, base * 2^attempt)]."""
    ceiling = min(cap, base * (2 ** attempt))
    return random.uniform(0, ceiling)

def retry_with_backoff(fn, max_attempts: int = 5, budget_token=None):
    for attempt in range(max_attempts):
        if budget_token and not budget_token.acquire():
            raise RuntimeError("retry budget exhausted")   # budget check first
        try:
            return fn()
        except Exception as e:
            if attempt == max_attempts - 1:
                raise
            time.sleep(backoff_delay(attempt))
            # attempt 0 → uniform[0, 0.1 s]
            # attempt 1 → uniform[0, 0.2 s]
            # attempt 4 → uniform[0, 1.6 s]  (cap kicks in at ~10 s)

Backoff alone is not enough. You also need a retry budget: a service-level cap on the fraction of traffic that can be retries at any moment. Google's SRE book describes retry budgets as the correct mechanism to prevent retry amplification at the per-service level — if retries exceed (say) 10% of requests, the client starts failing fast instead of retrying. This caps the maximum synthetic load a retry storm can add, independently of how many clients are misbehaving.

💡 Non-idempotent operations and retries

Backoff and retry budgets assume the operation is safe to repeat. A GET is fine; a POST /charge may not be. Use idempotency keys — a client-generated UUID sent with the request that the server uses to deduplicate. Stripe, Stripe's competitors, and every payment processor you have heard of require them. TraceLab's ingestion endpoint should as well, for the same reason: a robot that retries a flush can otherwise log the same batch twice.

CORE CONCEPTCircuit breakers: let the dependency breathe

Backoff slows clients down; a circuit breaker stops them entirely — temporarily — when a dependency is clearly failing. The pattern is named after the electrical component: a breaker trips when current exceeds a threshold, protecting the circuit. The software version has three states:

CLOSED traffic passes through OPEN fast-fail, no traffic HALF-OPEN probe: one request through per window ≥N failures in window sleep timeout elapsed probe succeeded → reset probe failed
The circuit breaker state machine. CLOSED is the healthy default. After N failures in a rolling window the breaker trips OPEN — all calls fast-fail immediately (no network round trip). After a sleep timeout one probe request is allowed through (HALF-OPEN). Success resets; failure re-opens.

The fast-fail in the OPEN state is the key contribution. Instead of each client waiting for a TCP timeout (often 30+ seconds), they receive an immediate error. This does three things: it stops adding load to the struggling dependency, it unblocks the calling service's threads immediately (preventing thread-pool exhaustion cascades), and it gives the dependency time to drain its queue and recover without a continuous rain of new requests on top.

🧪 Hystrix, Resilience4j, and "just implement it yourself"

Netflix's Hystrix was the canonical Java implementation; it has been in maintenance mode since 2018. Resilience4j is the active successor. For Python, circuitbreaker (PyPI) covers the basics. For TraceLab — a mostly internal system — a 50-line implementation that tracks the last N calls per dependency and flips a flag is entirely appropriate; the pattern is simple enough that the dependency is not worth it until you need shared-state breakers across processes (at which point the state lives in Redis and the calculus changes). The principle is more important than the library.

Two parameters tune a breaker: the failure threshold (how many failures in what window trips the breaker) and the sleep window (how long the breaker stays open before probing). These should be set empirically from your dependency's actual P99 latency and SLA, not copied from a blog post. A 30-second sleep window on a dependency that normally recovers in 2 seconds keeps you down longer than necessary. A 500ms sleep window on a dependency that needs a full GC cycle to recover will never stay open long enough to help.

CORE CONCEPTBulkheads: containing the blast

A bulkhead in a ship separates the hull into watertight compartments so a single breach cannot sink the vessel. In distributed systems the same idea limits the resource pool that a failing dependency can exhaust.

The failure mode bulkheads prevent is resource exhaustion cascade. A typical service has one shared thread pool. A slow dependency starts consuming threads waiting for responses. Eventually all threads are stuck waiting on that one dependency. Now every other dependency — even fast, healthy ones — cannot be served because there are no threads left to handle their responses. The entire service is down, and the root cause is one slow endpoint that only 10% of traffic exercises.

bulkhead concept (Python semaphore)blast-radius limit
from threading import Semaphore
from contextlib import contextmanager
import functools

# Each downstream gets its own semaphore — its own bulkhead.
# If robot_db is slow, at most 20 threads pile up there.
# The alert_service bulkhead is completely unaffected.
_BULKHEADS = {
    "robot_db":      Semaphore(20),
    "alert_service": Semaphore(5),
    "metrics_store": Semaphore(10),
}

@contextmanager
def bulkhead(name: str):
    sem = _BULKHEADS[name]
    acquired = sem.acquire(blocking=False)
    if not acquired:
        raise BulkheadFullError(f"{name} at capacity — shedding")
    try:
        yield
    finally:
        sem.release()

The semaphore limit is itself a design decision with tradeoffs: too low and you shed legitimate traffic; too high and you have not contained the blast. The right number comes from the dependency's throughput capacity and your acceptable latency budget, not from intuition. alert_service gets 5 slots not because 5 is magic, but because alert-level traffic should be rare (high-severity events only) and a bulkhead of 5 is sufficient to absorb normal burst while preventing a broken alert service from eating the request budget of robot telemetry ingestion.

CORE CONCEPTLoad shedding and graceful degradation

Backoff, breakers, and bulkheads all work on the caller side. Load shedding is the server side defense: actively refusing work when you know you cannot do it safely. The principle is counterintuitive but correct — it is better to serve 60% of requests well than to serve 100% of requests badly (high latency, high error rate, state corruption from partial writes). A service that predictably rejects excess load with an HTTP 503 + Retry-After header is a better citizen of its ecosystem than one that silently queues everything until it falls over.

Shedding strategyWhat gets droppedBest forTraceLab application
Random 1-in-NAny request equallyStateless services with homogeneous trafficNot appropriate — high-priority alerts should survive
Priority-awareLowest-priority work firstMixed-priority workloadsShed detailed telemetry before summary metrics; shed summary metrics before alerts
Cost-awareMost expensive requests firstAPIs with wildly varying request costDrop bulk replay ingestion before real-time robot flushes
Age-awareOldest queued requests firstTime-sensitive pipelinesA stale telemetry flush from 5 minutes ago is less valuable than the current one

Graceful degradation is the companion concept: when you must shed, shed the least valuable work and keep the most critical functions operating, even at reduced fidelity. For TraceLab this means there are explicit priority tiers of telemetry:

1

Tier 1 — Alerts

Robot out-of-bounds, safety stops, hardware faults. Never shed. These have direct safety implications. Reserve capacity explicitly for this class regardless of system load.

2

Tier 2 — Summary metrics

Per-run loss, reward, episode duration. Acceptable to drop at >90% load, but log the drop count so the gap is visible in dashboards.

3

Tier 3 — Detailed telemetry

Per-step sensor traces, per-frame images, full action logs. Shed first. Storage-intensive; losing this data during a load event is an acceptable tradeoff.

This hierarchy is a taste decision — the curriculum brief calls it explicitly. There is no universal answer. A financial trading system inverts this entirely: the audit log is Tier 1, not the alerts. Making the hierarchy explicit before the incident is the exercise. The engineers who do it rarely regret it; the ones who leave it implicit spend the incident arguing about what to sacrifice.

SEE ITRetry-storm recovery simulator

A downstream dependency slows down. TraceLab's 10 client nodes, retrying naively, tip the system into metastable collapse. Enable defenses one at a time and watch the system claw back. Then use the order picker to see why enabling them in the wrong sequence leaves you worse off than none at all.

⚙ Interactive · Retry-storm recovery simulator

A dependency slows. Naive retries create a storm. Enable defenses one at a time — each click steps the simulation forward one phase. Watch queue depth, error rate, and throughput react.

① Backoff+Jitter → ② Circuit Breaker → ③ Load Shedding

Click chips to toggle alternate order — Breaker first, then Shedding, then Backoff — and replay.

Queue Depth
—
requests waiting
Error Rate
—
% of traffic
Good Throughput
—
req/s actually served
Retry Traffic
—
% of total req/s
Dependency load
0%
Retry amplification
0%
Shed / fast-failed
0%
Press "Run storm" to begin.

Click "Run storm" to trigger the dependency slowdown and watch the retry cascade build.

Three observations worth making explicit. First, enabling backoff alone reduces — but does not eliminate — the storm; the amplification factor drops from ~3× to ~1.4×, which is not enough when the dependency is already at 80% capacity. Second, the circuit breaker does the heavy lifting: by fast-failing requests after the failure threshold is crossed, it removes synthetic load from the dependency almost immediately, allowing the queue to drain. Third, the wrong order (breaker first with no backoff) can create a saw-tooth pattern where the breaker oscillates between OPEN and HALF-OPEN without ever stabilizing, because each probe still lands in a burst with all the other probes from other service instances — the "thundering herd" on probe recovery.

TRACELABTraceLab sheds telemetry before it drops alerts

TraceLab v5 — the operated-and-outgrown milestone — is the first version that faces a 10,000-robot fleet. The ingestion tier, which in v4 simply wrote every flush to Postgres, will see bursts. The priority decision must become code, not comment.

tracelab/ingestion/shed.pyv5 — priority shedding
from enum import IntEnum
from threading import Semaphore
import time

class Tier(IntEnum):
    ALERT   = 1   # safety stop, hardware fault — never shed
    SUMMARY = 2   # per-run loss, reward, duration
    DETAIL  = 3   # per-step traces, raw sensor logs

# Hard capacity per tier — ALERT is small but never full-refused.
_CAPS = { Tier.ALERT: Semaphore(200), Tier.SUMMARY: Semaphore(500), Tier.DETAIL: Semaphore(300) }
_SHED_COUNTS = { t: 0 for t in Tier }

def ingest(payload: dict, tier: Tier) -> bool:
    """Returns True if accepted, False if shed.
    Callers MUST log shed events — gaps must be visible in dashboards."""
    sem = _CAPS[tier]
    acquired = sem.acquire(blocking=False)
    if not acquired:
        _SHED_COUNTS[tier] += 1
        if tier == Tier.ALERT:
            # Force-wake the on-call rather than silently drop.
            _SHED_COUNTS[tier] = 0          # reset — we will page regardless
            raise RuntimeError("ALERT tier capacity full — paging on-call")
        return False                          # SUMMARY / DETAIL: shed silently
    try:
        _write_to_store(payload)
        return True
    finally:
        sem.release()

def shed_metrics() -> dict:
    return {t.name: _SHED_COUNTS[t] for t in Tier}

Notice the explicit contract: ALERT tier overflows are never silently dropped — the code raises, which the caller catches and pages on. This is the taste decision made visible. You cannot accidentally leave it implicit by forgetting a comment. The shed_metrics() function is wired into TraceLab's /metrics endpoint so Prometheus can scrape it — shed rate is a leading indicator of approaching overload, not an afterthought.

The other v5 addition is the circuit breaker around the Postgres write. At 10,000 robots with 1 Hz telemetry flushes, a Postgres GC pause or a replication lag spike will cause exactly the metastable failure pattern described above if clients retry without a breaker. The breaker opens on 5 consecutive write failures, sleeps for 10 seconds, then probes — deliberately slow, because Postgres replication lag typically resolves in that window. During the OPEN period, DETAIL and SUMMARY telemetry is buffered in-process (bounded buffer, overflow shed) while ALERT writes bypass the breaker and go to a separate, dedicated write path.

⚠ The debt this lesson introduces

TraceLab v5's in-process buffer during OPEN breaker state is bounded — it holds at most 60 seconds of DETAIL+SUMMARY data at fleet scale. If the breaker stays open longer than 60 seconds, that data is lost. This is deliberate and documented, but it creates a new debt: the restore-and-durability story for that gap is unresolved until Lesson 6.4 addresses replication and backup windows.

WEAKNESSWhere these patterns bite

✗ Retry + circuit breaker with no jitter on probe recovery

A circuit breaker that sleeps for 10 seconds and then allows one probe is correct in isolation. But if 50 service instances all opened their breaker at the same moment — because they all see the same dependency failure — they all probe at t+10 seconds. The probe burst is exactly proportional to the number of instances. If the dependency is just barely recovering, 50 simultaneous probes re-trips the breaker on every instance. This is the thundering herd problem applied to circuit breaker recovery. Fix it with jitter on the sleep window: uniform(sleep * 0.8, sleep * 1.2) is enough to spread the cohort.

✗ Setting breaker parameters from the library defaults

Resilience4j's defaults are 50% failure rate over 100 calls with a 60-second open window. These are reasonable for a generic microservice in a conference demo. For TraceLab's Postgres write path, where a GC pause might cause 3–5 consecutive timeouts on a perfectly healthy instance, a 50% threshold over 100 calls means the breaker never trips — you need to see consistent failures across a short window, not random noise across a long one. Calibrate against your P99 latency and your dependency's actual failure modes. Do not copy-paste configuration from a README.

✗ Shedding without making the gap visible

Priority shedding under load is correct and necessary. But shedding is a silent operation unless you instrument it. A system that sheds 40% of DETAIL telemetry during a busy hour and does not surface that fact in its dashboards gives operators a false picture of fleet health — the telemetry gaps look like robots that weren't running. Export shed counts as labeled Prometheus counters, alert when the rate exceeds a threshold for sustained periods, and document the expected steady-state shed rate (it should be near zero). The gap must be visible, not merely acceptable.

KEY INSIGHTHow this connects

  • Capacity Planning (6.2) — that lesson found where the system saturates; this one builds the defenses that activate when saturation happens. The tail latency models from 6.2 determine the right breaker thresholds.
  • Data at Scale (6.4) — the in-process buffer that backs TraceLab's OPEN-state write path creates a durability gap. Lesson 6.4's replication and backup discussion closes that gap.
  • Observability (5.3) — circuit breaker state transitions, shed counts, and retry-budget consumption are exactly the signals you need your metrics and tracing infrastructure to export. None of these patterns work in the dark.
  • Distributed Systems Fundamentals (4.3) — the metastable failure pattern is one manifestation of the feedback loop dynamics that make distributed systems qualitatively harder than single-process ones.

The theme of Stage 6 is taste — not just knowledge. The reliability patterns in this lesson are straightforward to implement. The hard part is the priority decision: which traffic is Tier 1, what does graceful degradation look like for your system, what is an acceptable shed rate. Those decisions do not come from a paper; they come from understanding your users, your data, and your failure modes well enough to make an explicit, documented, defensible call. That is the judgment this stage is training.

GLOSSARYKey terms introduced

Metastable failure
A failure state that persists after its original trigger resolves, because a feedback loop (typically retry traffic) continuously recreates the overload condition that triggered it.
Exponential backoff with full jitter
A retry delay strategy where each attempt's wait is drawn uniformly from [0, min(cap, base × 2^attempt)], spreading clients out in time to reduce contention on a shared resource.
Retry budget
A per-service cap on the fraction of requests that may be retries at any moment; prevents retry amplification from exceeding a bounded multiple of real traffic.
Circuit breaker
A stateful proxy with CLOSED / OPEN / HALF-OPEN states; trips OPEN after N failures, fast-failing all calls until a probe succeeds — giving a struggling dependency time to recover.
Bulkhead
A resource partition (thread pool, connection pool, semaphore) per downstream dependency; limits the blast radius of a failing dependency to its own partition rather than the shared pool.
Load shedding
Actively refusing excess work at the server, typically with 503 + Retry-After, to keep accepted-work latency within SLA rather than degrading all traffic equally.
Graceful degradation
Shedding in priority order so that the highest-value traffic survives at full fidelity while lower-value traffic is dropped first; requires an explicit, documented priority hierarchy.
Thundering herd
A synchronized burst of clients making simultaneous requests — on connection recovery, cache miss, or breaker probe — that overwhelms the target just as it is trying to recover.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What makes a metastable failure different from a normal overload? Why can't you fix it by just adding more capacity?›

In a normal overload, load exceeds capacity; adding capacity resolves it. In a metastable failure, the original cause has already resolved — the excess load is synthetic retry traffic generated by clients trying to recover. Adding capacity absorbs more retries, which (if clients see success) reduces retry volume, which reduces load — but the system is stuck in a stable high-load state and small perturbations don't dislodge it. The fix must come from the client side (backoff, breakers) reducing the retry load, not from the server side absorbing it.

2Why is full jitter better than a fixed exponential delay for retry storms? What does "full" mean specifically?›

"Full" jitter means drawing the delay from uniform(0, ceiling) — the entire range including near-zero. This maximally spreads clients in time, reducing the probability that multiple clients retry simultaneously. A fixed exponential delay keeps all clients synchronized — they all backed off for 0.8 seconds, so they all retry at the same moment. "Equal jitter" (uniform over the upper half of the range) is intermediate. Marc Brooker's 2015 AWS analysis showed full jitter produces the lowest mean call completion time under retries against a shared resource.

3What does a circuit breaker's HALF-OPEN state do, and why can it create a thundering herd if you have many service instances?›

HALF-OPEN allows one probe request through after the sleep window expires, to test whether the dependency has recovered. If successful, the breaker resets to CLOSED. The thundering herd problem: if N service instances all opened their breakers simultaneously (same trigger), they all enter HALF-OPEN at the same time and each sends a probe — creating a burst of N simultaneous requests, potentially re-tripping the breaker before it can reset. Fix: add jitter to the sleep window so instances probe at different times.

4What is the priority decision TraceLab v5 must make explicit, and why does the curriculum call this a "taste" decision rather than a technical one?›

TraceLab must define which telemetry is shed first under load: DETAIL traces, then SUMMARY metrics, then ALERT events (which are never shed). This is a taste decision because there is no universally correct answer — a financial audit system might make the log unshedable rather than alerts. The right hierarchy depends on understanding your users (safety-critical robotics vs. batch ML research), your failure modes, and what data loss is actually acceptable. Making it explicit — in code, not comments — is itself the engineering act: it prevents the on-call engineer from improvising the decision mid-incident.

Next: Replication, backups, and what "durable" actually means at scale →