ProjekinAja. Software Engineering Stage 4 · Architecture Lesson 4.5
● Stage 4 · Lesson 4.5

Distributed Systems Reality

Partial failure is qualitatively different, not just bigger — timeouts, retries that amplify outages, untrustworthy clocks, and CAP without the triangle.

⏱ ~22 min read 📡 Distributed systems 🔧 TraceLab v3

🎯 By the end you'll understand

Why partial failure is a new category of problem, not a bigger version of local failure. How naive retries turn one slow dependency into a self-inflicted outage of a healthy service. Why clocks in a distributed system are advisory at best. What CAP actually says — and why it is a choice about behavior during partition, not a geometric tradeoff. And what "eventually consistent" really means when your robot telemetry arrives out of order.

KEY INSIGHTDistribution is not scale — it is a different failure model

A local program either runs or it doesn't. A distributed system can be simultaneously running, dead, and somewhere in between — on the same request, to different callers, right now.

Here's the thing you were probably not told: distributing a system does not make it faster or more reliable in any simple sense. It introduces a failure mode that does not exist on a single machine — partial failure. Some of the system is working. Some isn't. You cannot tell which. Callers cannot tell either. And unlike a crash, partial failure can last hours while looking suspiciously like "a bit slow."

🔑 The principle that changes everything

In a local function call, the outcome is binary: you get a return value or you get an exception. In a network call, there is a third outcome — silence. You sent a request. It may have arrived. The service may have processed it. The response may have been lost. You will never know. Every retry you write is a guess about which of these happened. Write enough of them the wrong way and you can kill a healthy service with your own traffic.

CORE CONCEPTThe eight fallacies of distributed computing

In the 1990s, engineers at Sun Microsystems — L. Peter Deutsch, James Gosling, and others — catalogued the assumptions that developers new to networking invariably make and invariably regret. They called them the fallacies of distributed computing. They still bite, thirty years later, because they're so plausible.

FallacyWhy it's wrongThe real consequence
The network is reliablePackets are dropped, NICs fail, cables are cut by backhoesYou need timeouts, retries, and idempotency — everywhere
Latency is zeroA round-trip between AWS us-east-1 and us-west-2 is ~70 ms minimumSynchronous chatty APIs that feel fine on localhost die in prod
Bandwidth is infiniteShared bandwidth is a finite, contested resourceYour "fast" service becomes slow when a neighbor bursty-reads its cache
The network is secureTLS only at the perimeter means internal traffic is in the clearOne compromised service sees all inter-service traffic
Topology doesn't changeNodes restart, scale-down kills IPs, DNS TTLs lagHard-coded IPs, non-refreshing connection pools
There is one administratorMicroservices means multiple teams, multiple deploy cadencesBreaking API changes shipped without coordination
Transport cost is zeroSerialization, deserialization, and compression are real CPUgRPC overhead surprises teams that measured only network time
The network is homogeneousOne service talks JSON over HTTP/2, another XML over AMQPInterop bugs that are invisible in tests, explosive in prod

Each of these is not a mistake — it is the natural mental model of someone who learned to program on a single machine. The only way to exit is to experience the fallacy biting you, which is why this lesson is structured around a simulator rather than a reading list.

CORE CONCEPTPartial failure: the mode that doesn't exist locally

Your robot's telemetry pipeline calls an ingest service. The ingest service calls a storage service. The storage service's disk is 90% full and writes are taking 8 seconds instead of 30 milliseconds. What does the ingest service see? Not an error. A response that eventually arrives — if the timeout is generous enough. What do callers of the ingest service see? The ingest service becoming "slow." What do their callers see? Them becoming slow too. The degradation propagates upstream, service by service, until the entire call graph is underwater — caused by one disk on one machine.

1

Slow ≠ dead, and that's worse

A dead service fails fast. Callers accumulate errors, circuit breakers trip, retries stop. A slow service holds connections open. Thread pools fill with waiting calls. The service becomes unresponsive to other callers. Cascading slowness is the dangerous mode.

2

Retries as amplifiers

A caller with a generous timeout waits, gets nothing, and retries. Three retries per call, ten callers — the slow service now receives 30× the intended load at the worst possible moment, the moment it's already struggling. This is the retry storm: the disease and its treatment are the same pathogen.

3

Cascading vs correlated failure

Cascading: A causes B causes C. Correlated: A, B, and C all fail because they share a dependency (a database, a load balancer, a shared config service) that none of them modeled as a dependency. Correlated failures look like coincidence until you draw the graph.

Robot caller API Gateway threads filling up Ingest Svc waiting on storage Storage Svc disk 90% full → req → req → hangs 8s thread pools exhausted — other callers time out too one disk problem cascades through two healthy services
Partial failure in practice: a single slow dependency holds threads open in every upstream service. The callers are healthy — but their thread pools fill with waiting connections until they stop accepting new work.

The countermeasures are well-understood: timeouts (cut the thread loose), circuit breakers (stop sending requests to a failing service until it recovers), bulkheads (separate thread pools per downstream, so one slow dependency can't exhaust the shared pool). But they each require explicit design. You don't get them by default. And they each introduce their own failure modes — an overly aggressive circuit breaker that trips on normal variance, a timeout that's shorter than a legitimate slow operation.

CORE CONCEPTClocks you cannot trust

On a single machine, time.time() is not the ground truth — it is a value synchronized by NTP that can jump forward or backward — but at least all code on the machine sees the same value. In a distributed system, every node has its own clock, and those clocks drift. Two services that agree on "now" to within 50 ms are doing well. Google's TrueTime (the clock underlying Spanner) explicitly quantifies clock uncertainty — it returns a time interval, not a point — because that's the honest answer.

⚠ The dangerous consequences of trusting wall time

Using wall-clock timestamps to order events across machines is one of the most common distributed systems bugs. Service A records event at t=100, service B records a causally-later event at t=99. You sort by timestamp and the causal order inverts. The fix is logical clocks (Lamport timestamps, vector clocks), which track causality rather than wall time. Kafka's "log-append time" vs "event time" distinction is exactly this problem in production at scale.

For TraceLab, this matters concretely. A robot telemetry event at timestamp T on the robot's clock and a server-side processing event at timestamp T+Δ should be ordered by causality, not by their clock values. If the robot's clock is 200 ms ahead, the processing appears to happen before the telemetry — which produces nonsense timelines. The correct design: record the server-receipt timestamp alongside the event timestamp, and use the sequence number (or a monotonic logical clock) for strict ordering. Never trust the sender's clock for global ordering.

📖 Lamport timestamps in one sentence

Every process keeps a counter; it increments before each event; when sending a message it includes the counter; the receiver takes max(local, received) + 1. This guarantees: if A → B (A happened before B), then ts(A) < ts(B). The converse is not guaranteed — you can't infer causality from Lamport timestamps alone, which is why vector clocks exist.

SEE ITRetry storm simulator

The widget below simulates a 5-service TraceLab call graph: Robot → API Gateway → Ingest → Analytics → Storage. Inject latency or a partition into any service using the presets, then watch what naive retries do to the load. Switch the retry strategy to see circuit breakers and jitter claw back stability.

⚙ Interactive · Distributed call-graph simulator

Inject faults via presets. Watch the load metrics. Switch retry strategy and partition mode to see the behavior change. All numbers are deterministic ratios — they represent relative load, not real ms.

Robot load: 1× API Gateway load: 1× Ingest load: 1× Analytics load: 1× Storage load: 1×
Storage load
1×
Ingest load
1×
Error rate
0%
Write availability
100%
Consistency
strong
Scenario: no fault · retry strategy: naive

Select a fault scenario and step through to see how load propagates.

Three things to observe. With naive retries and "storage slow", the storage load climbs to 30× of the intended load — three retries per call, multiplied by every upstream caller. With the circuit breaker enabled, the load caps and errors fast-fail. When you switch to the partition scenario, the CP vs AP toggle shows the real CAP tradeoff: not a dial, but a discrete choice about what the system should do when it cannot reach its peers.

CORE CONCEPTCAP without the triangle

The CAP theorem (Gilbert and Lynch, 2002, formally proving Brewer's 2000 conjecture) is the most quoted, most misunderstood result in distributed systems. Its actual claim is narrow and precise: in a distributed system, during a network partition, you must choose between consistency (every read sees the most recent write) and availability (every request receives a response — not necessarily consistent). You cannot have both during the partition.

📖 What CAP actually says

CAP does not say "pick two of three." It says: in normal operation you have both consistency and availability. When a partition occurs, you must decide which to sacrifice. Most distributed databases choose a point on this spectrum — and the interesting engineering is in how they handle the partition and the reconciliation after it heals, not in the abstract triangle.

The standard formulation of CP vs AP:

ChoiceDuring partitionAfter partition healsWho uses it
CPRefuse writes (return error). Reads that cannot be confirmed consistent are also refused or return a stale-flag.Resume immediately — no reconciliation needed, no divergence occurred.Google Spanner, etcd, Zookeeper, HBase
APAccept writes on all reachable nodes. Different nodes may diverge.Reconcile diverged states — merge, last-write-wins, CRDTs, or ask the user.Cassandra (tunable), DynamoDB, CouchDB, Riak

For robot telemetry, the question is concrete: if the ingest service cannot reach its storage replicas, does it tell the robot "come back later" (CP), or does it accept the telemetry and promise to write it somewhere eventually (AP)? Neither answer is wrong. The wrong answer is not having made the decision — discovering you are implicitly AP when your operations team expected CP, at 3 AM, during an actual partition.

💡 PACELC: the model beyond CAP

Daniel Abadi's PACELC framework (2012) extends CAP by noting that the interesting tradeoff happens even when there's no partition: Else (during normal operation), you are trading Latency for Consistency. Waiting for all replicas to acknowledge a write is consistent and slow; writing to one and propagating asynchronously is fast and eventually consistent. Most real design decisions live in the ELC leg, not the CAP partition corner.

CORE CONCEPT"Eventually consistent" means occasionally wrong, on purpose

Eventually consistent does not mean "it will be right soon." It means: the system guarantees convergence to a consistent state if writes stop — but offers no bound on when, and makes no promise about intermediate reads. An eventually consistent read can return data that is hours old. Two concurrent reads from different replicas can return different answers. This is not a bug — it is the deliberate tradeoff for availability and partition tolerance.

For TraceLab, this is the difference between "can a researcher trust the loss curve they are looking at right now?" and "will the loss curve eventually settle to the correct values?" An AP ingest system answers the first with "probably yes, hedge on recency" and the second with "yes, within seconds typically." A CP system answers the first with "yes" and the second is trivially satisfied but may temporarily answer the first with a 503.

Consistency modelWhat reads seeCostTraceLab example
Strong (linearizable)The most recent completed write, alwaysCoordination on every read; latency, network roundtripsExperiment metadata ("which config was used?") — must be exact
Read-your-writesYour own writes immediately; others eventuallySession affinity or client-side tokenThe robot that just wrote sees its own data; other readers may lag
Monotonic readNever see older data than you already sawVersion tracking per clientDashboard refresh never shows a metric regress
EventualConvergence guaranteed; no read-time guaranteeCheapest; full availability; reconciliation logic requiredAggregate statistics: "today's total runs" — a second off is fine
⚠ The reconciliation is not free

AP systems that diverge need merge logic. "Last write wins" is simple but loses data. CRDTs (conflict-free replicated data types) give automatic merge for specific data shapes (counters, sets, registers) but require careful upfront design. Custom merge functions are powerful and hard to get right. The cost of AP is not the divergence — it is everything you must build to handle it.

TRACELABTraceLab v3: distribution decisions, written down

TraceLab v2 is a client-server monolith. It works. The question this lesson asks is: should it become distributed, and if so, which parts and why? Stage 4's discipline is that you must be able to answer both halves — here is where distribution helps, here is where it costs.

The architecture decision record (ADR) pattern: a short document with four sections — context, options, decision, consequences. This is what "architecture is recorded decisions" means in practice.

docs/adr-004-ingest-consistency.mdADR
# ADR 004: Consistency model for telemetry ingest
# Status: accepted

## Context
# TraceLab v3 will have a multi-replica ingest service.
# Robots write telemetry at up to 100 Hz during experiments.
# Researchers read dashboards interactively; they accept slight lag.
# Experiment metadata (config, hyperparameters) must be exact and
# auditable — "what config produced this run?" must have one answer.

## Options
# A. Strong consistency everywhere — all writes synchronously replicated.
#    Pros: simple semantics. Cons: write latency proportional to replica
#    count; any replica unavailability blocks ingest.
# B. Eventual consistency everywhere — writes go to any replica, merge later.
#    Pros: highest availability. Cons: metadata can diverge; merge logic for
#    configs is complex (last-write-wins loses data).
# C. Split by data type: strong for metadata, eventual for telemetry.
#    Pros: each decision appropriate to the data. Cons: two consistency
#    domains to understand and test.

## Decision
# Option C. Metadata (experiment configs, run IDs) uses a CP store (etcd).
# Telemetry writes use an AP queue (the ingest queue from lesson 4.4)
# with read-your-writes consistency for the writing robot via sequence
# numbers. Dashboard reads accept eventual consistency explicitly.

## Consequences
# Telemetry ingest stays available during metadata-store partitions.
# Dashboards must display a staleness indicator on real-time metrics.
# The merge strategy for telemetry is last-write-wins by sequence number
# (monotonic, from the robot) — safe because each robot is the sole writer
# of its own telemetry stream. Cross-robot aggregates accept eventual lag.

The key discipline: the decision about retry behavior is also recorded. The robot client uses exponential backoff with jitter (base 100 ms, max 2 s, jitter ±30%) for transient errors (timeout, 503). For 4xx client errors, no retry — retrying a malformed request is guaranteed useless. A retry budget of 2 per request prevents retry amplification from a misbehaving robot. These decisions live in a second ADR, not in a developer's head.

tracelab/client/retry.pyTraceLab v3
import time, random

RETRY_BASE   = 0.1   # seconds
RETRY_MAX    = 2.0
JITTER_FRAC  = 0.3
MAX_RETRIES  = 2      # budget: don't be the storm

def _backoff(attempt):
    """Exponential backoff with decorrelated jitter."""
    base = min(RETRY_BASE * (2 ** attempt), RETRY_MAX)
    jitter = base * JITTER_FRAC * (random.random() * 2 - 1)
    return max(0, base + jitter)

def post_with_retry(fn, *args, **kwargs):
    """Call fn(*args, **kwargs). Retry on transient errors only."""
    for attempt in range(MAX_RETRIES + 1):
        try:
            resp = fn(*args, **kwargs)
            if resp.status_code < 400:
                return resp
            if resp.status_code < 500:
                raise ValueError(f"Client error {resp.status_code} — not retrying")
        except TimeoutError:
            pass   # transient — fall through to backoff
        if attempt < MAX_RETRIES:
            time.sleep(_backoff(attempt))
    raise RuntimeError("Max retries exceeded")

Notice what this does not do: it does not retry indefinitely, it does not retry immediately, and it does not retry client errors. The jitter is important — without it, every robot that hits the same transient error at the same time retries at the same time, creating a synchronized burst. Jitter desynchronizes the retry wave.

WEAKNESSWhere distributed thinking goes wrong

✗ Distributing before the monolith hurts

The single most common architectural mistake of the last decade: splitting a working monolith into microservices before the monolith demonstrably cannot scale. Every service boundary is a network call — latency, failure modes, version mismatches, deployment coordination, distributed tracing, service mesh. You pay all of these costs upfront. The benefit (independent scaling, team autonomy) accrues later, maybe. TraceLab v3 adds distribution only to the ingest path, which has a measured throughput problem. The rest stays a monolith, on purpose.

✗ Retrying without a budget or backoff

A while True: requests.post(...) retry loop is one of the most dangerous things in a distributed codebase. It turns a brief outage into a cascading amplifier. The fix requires both bounded retries and exponential backoff — either alone is insufficient. A service with bounded retries but no backoff can still overwhelm a recovering service with a burst. Backoff without a bound can leave callers retrying forever. Both, together, with jitter, is the minimum viable retry strategy.

✗ CAP as a database-selection heuristic

Choosing a database because it's "CP" or "AP" without specifying what that means for your data and your partition behavior is not architecture — it's label-shopping. Cassandra is tunable; the default quorum level determines where on the consistency-availability spectrum you actually sit. "We use Cassandra so we're AP" is not a consistency model decision; it's a marketing phrase. The actual decision is: what does this specific data item need during a partition, and does the configuration of our specific cluster deliver that?

KEY INSIGHTHow this connects

  • Queues and Async Work (4.4) — the ingest queue from the previous lesson is the first line of defense against retry storms: the queue decouples the robot's write from the storage write, so storage slowness doesn't block the robot's call at all.
  • The Mathematics of Failure (4.6) → — failure math formalizes the intuitions from this lesson: SLOs, error budgets, and availability arithmetic turn "the service is unreliable" into a number you can budget against.
  • HTTP: The Lingua Franca (3.1) — every distributed call in this lesson is an HTTP request; understanding how those requests fail, timeout, and carry status codes is the prerequisite vocabulary for diagnosing retry storms and partition behavior.

The lesson to carry forward: distribution is a cost paid per service boundary, per call path, per consistency decision. Pay it deliberately and document the decision. A distributed system that was built without recording why each boundary exists is a distributed system whose behavior no one can reason about.

GLOSSARYKey terms introduced

Partial failure
A distributed system state where some components are functioning and others are not — distinguishable from the outside only by timeouts and incomplete responses, not by a clean error signal.
Retry storm
The amplification of a service's load caused by multiple upstream callers retrying failed or slow requests simultaneously, often turning a recoverable slowdown into a complete outage.
Circuit breaker
A proxy that tracks failure rates for a downstream service and short-circuits (fast-fails) calls when the failure rate exceeds a threshold, preventing retry amplification and allowing recovery time.
CAP theorem
During a network partition, a distributed system cannot simultaneously guarantee Consistency (every read sees the latest write) and Availability (every request receives a response). One must be sacrificed.
CP system
A system that, during a partition, refuses requests rather than risk returning inconsistent data. Prioritizes correctness over availability.
AP system
A system that, during a partition, continues serving requests, allowing replicas to diverge. Prioritizes availability over consistency; requires reconciliation after healing.
Eventual consistency
A consistency model that guarantees convergence to a consistent state given no new updates, but offers no bound on when that state is reached or what intermediate reads will return.
Lamport timestamp
A logical clock that assigns a monotonically increasing counter to events, guaranteeing that if event A causally precedes B, then ts(A) < ts(B) — without relying on synchronized wall clocks.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Why is a slow service more dangerous than a dead service in a distributed system?›

A dead service fails fast — callers get an immediate error, circuit breakers trip, and retries (if any) exhaust quickly. A slow service holds connections open — callers accumulate waiting threads, thread pools fill, and the caller itself becomes unable to serve new requests. The degradation propagates upstream through every service that depends on the slow one, even though those services are individually healthy. Slow partial failure is the dangerous mode precisely because it mimics high load rather than signaling a fault.

2Why does naive retry logic turn a brief outage into a sustained one?›

If every caller retries three times immediately when a service is slow, the service receives 3× the load at the worst possible moment — when it's already struggling. With multiple callers, the multiplier compounds. Worse, the retries are synchronized: all callers hit the service in waves after the same timeout. The fix requires a retry budget (bound the number of retries), exponential backoff (give the service time to recover), and jitter (desynchronize retry waves so they don't hit simultaneously). Without all three, retries are amplifiers, not safety nets.

3What does the CAP theorem actually say — and what does it not say?›

CAP says: during a network partition, a distributed system must sacrifice either consistency (all reads see the latest write) or availability (all requests receive a response). It does not say "pick any two of three in normal operation" — in normal operation without a partition, you can have both. The theorem is a statement about partition behavior only. The practical engineering work is in deciding what the system does during a partition (CP: refuse writes; AP: accept and diverge) and how it reconciles after healing (trivially for CP; merge logic for AP).

4Why should you not use wall-clock timestamps to order events across services?›

Every machine has an independent clock that drifts. NTP synchronization keeps clocks close but not identical — a 50–200 ms skew is typical; larger skews occur after network disruption. Two causally-ordered events (A happens, then B is caused by A) can record timestamps that invert this order if A's machine is slightly ahead. The correct tool for ordering distributed events is a logical clock (Lamport timestamps track causal precedence; vector clocks detect concurrent events). Wall time is useful for human-readable timestamps and coarse bucketing, not for determining the order of events across service boundaries.

Next: turning failure probabilities into error budgets and SLOs →