ProjekinAja. Software Engineering Stage 5 · Shipping & Running Lesson 5.10
● Stage 5 · Lesson 5.10

TraceLab v4: The Outage You've Been Training For

A scripted ingestion outage hits the freshly shipped system — command it, mitigate first, diagnose later, and write the postmortem that makes it worth it.

⏱ ~22 min read 🚨 Stage 5 milestone & exam 📋 Blameless postmortem

🎯 By the end you'll understand

The full delivery path from commit to live production, what an incident commander actually does (and doesn't do), why mitigate-first-diagnose-later is a principle and not a preference, how retry amplification turns a config change into a queue meltdown, and how to write a blameless postmortem that produces action items people actually complete. The stage-5 exam is not a quiz — it is a scripted outage you must command, scored on time-to-mitigation against the expert path.

KEY INSIGHTProduction is not the finish line — it is the starting line

Code on a laptop is a rumor. Software exists when it runs for someone else — and the moment it does, the clock starts on your first incident.

Stage 5 built the entire delivery apparatus: artifact, container, CI pipeline, canary deployment, declared infrastructure, config discipline, and observability. TraceLab v4 now has all of it. The last act of the stage is not deploying it — you already know how to do that. The last act is surviving the first real outage.

🔑 Hold onto this

An incident has two completely separate jobs: stop the bleeding (mitigation) and understand why it happened (diagnosis). These jobs use different information, different skills, and different timelines. Conflating them is the single most common failure mode in incident response — engineers start debugging root cause while users are still affected, the outage stretches, and everyone is exhausted before the postmortem even begins. The rule is unambiguous: mitigate first, diagnose later. Every decision in the widget below is scored against this principle.

HOW IT WORKSThe full delivery path TraceLab v4 runs

Before the outage fires, let's confirm what the delivery path now looks like. Every commit to main triggers:

1

CI pipeline (Lesson 5.3)

Tests run. The Stage-2 test suite gates the build. A failing test stops the image from being built — the pipeline is the only path to production.

2

Image build with layer discipline (Lesson 5.2)

Multi-stage Dockerfile: build stage installs dependencies, production stage copies only the wheel. The image is ~150 MB, not 1.2 GB. It is tagged by commit SHA — artifacts are immutable.

3

Canary deployment (Lesson 5.5)

The new image is promoted to a canary pod (5% of traffic). Dashboards watch error rate and p99 latency for 10 minutes. Automated rollback fires if error rate exceeds 1%.

4

Full promotion to prod

Canary passes, traffic shifts. Infrastructure declared in code (Lesson 5.6) — no SSH, no manual changes. Config comes from the environment; secrets from the secrets manager (Lesson 5.7).

5

SLO dashboards live (Lessons 5.8–5.9)

Logs structured, metrics scraped, traces sampled. SLO: ingestion error rate < 0.1%, p99 ingest latency < 500 ms, rolling 30-day. Error budget burns are alerted. The system is observable.

Now the outage fires.

CORE CONCEPTAnatomy of the scripted outage

The incident you are about to command is not random. It is constructed to teach three specific things: (1) how a single config change can cascade through retry amplification into a queue meltdown, (2) the difference between the alert telling you something is wrong and the log telling you what is wrong, and (3) why heroics — one engineer silently debugging at 2 AM — are an organizational smell, not a virtue.

📖 What actually happened

T+0:00 — A config change shipped 48 hours ago lowered the ingestion worker's per-message processing timeout from 30 s to 3 s (meant to catch runaway sensors). The change passed canary because canary traffic is low-volume synthetic data — real payloads from field robots are larger.

T+4:32 — Morning peak traffic begins. A subset of large robot payloads now exceeds the 3-second timeout. The ingestion worker logs a timeout error and retries. Retry policy: 3 attempts with exponential backoff. Under load, every failed message retries 3×, tripling the queue depth.

T+4:41 — Queue depth crosses the alert threshold. The SLO error budget burns at 83× normal rate (8.3% error rate ÷ 0.1% SLO budget). The on-call engineer receives a PagerDuty alert: ingestion error rate 8.3%, SLO breach imminent.

message large payload TIMEOUT 3s limit hit retry 1 retry 2 retry 3 queue 3× depth per failure at load queue spiral depth grows faster than workers can drain it root cause: config timeout 30s → 3s canary missed it (low-volume synthetic data)
Retry amplification: a 3-retry policy converts every timeout into 3 queue entries. Under morning load, large-payload failures fan out faster than workers drain the queue.

The key insight about this incident is the gap between the alert and the cause. The alert fires on error rate — a symptom visible in metrics. The cause — a config change 48 hours old — is visible only in logs and git history. An engineer who skips straight to diagnosis risks spending 40 minutes in config history while users are still affected. The correct move is to stop the bleeding first.

💡 Incident severity is a communication contract

Severity levels (SEV1–SEV4, P0–P4) are not grades — they are coordination signals. A SEV1 means: incident commander owns the channel, all hands available, leadership notified, 15-minute check-ins, no side investigations without IC approval. Calling a SEV2 incident "SEV3" because you're optimistic is not humility; it's denying people the coordination they need to help.

SEE ITCommand the incident

The outage unfolds in five timed phases. At each decision point, pick the action you would take. You will be scored on time-to-mitigation: how many decisions delay mitigation vs. advance it. After the incident is resolved, assemble the postmortem timeline and classify contributing factors.

🚨 Interactive · Incident commander step-runner

Each phase reveals new information. Choose your action — some advance mitigation, some delay it. The widget tracks your score against the expert path. No free-text input: all choices are curated to represent the real decision space.

Incident scorecard

Time-to-mitigation
—
Expert alignment
0/0
Phase 1 of 5 — Alert fires

The scoring is designed around one insight: good incident commanders look like they're not doing much. They are not the hero debugging the system — they are the person making sure the right people are talking to each other, that communication is going out on schedule, and that no one is chasing root cause while the queue is still melting down. The engineer who silently debugs for 40 minutes without mitigating is not being thorough — they are making the incident worse.

TRACELABTraceLab v4: shipped and survived

This is the Stage 5 milestone. TraceLab has traveled from a 21-line CSV logger to a system that is containerized, tested, continuously deployed, and instrumented — and has now survived its first real incident. Here is the full debt ledger, closed:

v0 debtResolutionLesson
No encoding; comma injectionCSV writer with csv.writer, UTF-8 encoding declared1.5
Metrics stored as stringsTyped schema with float columns; round-trip safe serialization1.5
O(n) full-file scanB-tree index; query planner via SQLite then Postgres1.6–1.7
No history or versioningGit-tracked; schema migrations via AlembicStage 2, 3.4
No concurrency safetyPostgres row-level locking; connection pooling3.5–3.6
Schema in nobody's headDeclared schema + migration history; ER diagram in docs4.2
No artifact disciplineWheel built by CI; Docker image tagged by SHA5.1–5.2
Manual deployCI/CD pipeline; canary deploy with automatic rollback5.3–5.5
Config in source codeAll config from env; secrets from secrets manager5.7
No observabilityStructured logs, Prometheus metrics, distributed traces, SLOs5.8–5.9
No incident practiceFirst outage commanded; postmortem written; action items tracked5.10 (this lesson)

The v4 artifact is now production-grade by a reasonable definition: it fails safely (circuit breakers, graceful degradation), it is observable (you can tell something is wrong before users call you), it deploys predictably (CI gates, canary, IaC), and it has a team practice for learning from failures. It is not perfect — no production system is — but it is honest about its own state, which is the engineering virtue Stage 5 was training.

tracelab v4 — incident action items (real format)postmortem output
# Action items from the 2024-11-14 ingestion outage postmortem
# Format: [priority] item — owner — due date — status

[P1] Add representative large-payload fixtures to CI integration tests
     # Canary uses synthetic data; this was the enabling condition.
     # Owner: @tracelab-infra  Due: 2024-11-21  Status: IN PROGRESS

[P1] Add payload-size validation at ingestion entrypoint # reject before queue
     # Owner: @tracelab-backend  Due: 2024-11-21  Status: OPEN

[P2] Review all timeout values added in the last 30 days against p99 payload sizes
     # Owner: @tracelab-infra  Due: 2024-11-28  Status: OPEN

[P2] Implement retry budget (max 2 retries, jitter, dead-letter queue)
     # Current 3× retry with no dead-letter is an amplifier waiting to fire again.
     # Owner: @tracelab-backend  Due: 2024-11-28  Status: OPEN

[P3] Canary traffic profile: include a sample of real production payloads
     # Synthetic-only canary has a known blind spot for size/latency regressions.
     # Owner: @tracelab-infra  Due: 2024-12-05  Status: OPEN
💡 The postmortem is where the ROI lives

An outage that produces completed action items is an investment. An outage that produces a document nobody reads is a loss. The difference is almost never the quality of the writing — it is whether action items are assigned to real people with real due dates and tracked in the same tool the team already uses. P1 items from postmortems should appear in the next sprint by default.

IN PRACTICEThe blameless postmortem — what it actually contains

The term "blameless" is widely used and frequently misunderstood. It does not mean "no one did anything wrong." It means the postmortem's purpose is system improvement, not punishment — and these two goals require different information. A blame-oriented debrief produces hiding; a system-oriented one produces honesty about contributing factors, which is the only way to find the real levers.

The structure that works (adapted from the Google SRE book's incident review format, which is widely used but not the only reasonable approach):

1

Timeline — factual, no editorializing

Exact times from logs and monitoring, not from memory. Every significant detection, decision, action, and escalation. If two people remember the timeline differently, the logs are the source of truth. The timeline should be readable by someone who was asleep during the incident.

2

Impact — user-facing and quantified

What percentage of requests were affected, for how long? What was the SLO error budget consumed? "Some ingestion was slow" is not an impact statement — "8.3% of ingestion requests failed over 41 minutes, consuming roughly 8% of the 30-day error budget" is.

3

Contributing factors — trigger, enabler, amplifier

Not "root cause." Complex systems rarely have single root causes — they have conditions that together made the failure possible. The timeout config is the trigger (the direct cause of failures); the synthetic canary traffic is an enabler (let the trigger ship undetected); the 3× retry policy is an amplifier (made the impact larger than the underlying failure rate). Each category points to a different class of fixes.

4

What went well — genuinely

Not a token paragraph to soften the report. Real things: the SLO alert fired within 9 minutes of impact start; the rollback procedure worked on first attempt; communication to stakeholders was clear. These are the practices to reinforce and the playbooks to document.

5

Action items — named, dated, tracked

Each item has an owner (a person, not a team), a due date, and a priority. Items land in the issue tracker before the postmortem meeting ends — not in a document appendix that lives on a drive nobody checks.

⚠ "Root cause" is a trap

The search for "the" root cause satisfies a narrative desire — a single villain to defeat — but complex systems fail through confluence, not single points. Asking "why did ingestion fail?" and stopping at "the timeout was too low" misses the canary blind spot and the retry amplifier. Both will cause future incidents. The contributing-factor taxonomy (trigger/enabler/amplifier) is more annoying to fill out and far more useful.

WEAKNESSWhere incident response bites

✗ Heroics are an organizational smell, not a virtue

The engineer who is always the one to save the system at 2 AM is not a hero — they are a bus factor and a symptom. If incidents require a specific person to resolve, the knowledge is siloed, the runbooks are missing, and the next incident will also require them. Great incident response is boring: someone pages the right people, the runbook is followed, the system recovers without heroics. If it's not boring yet, writing better runbooks and incident simulations (game days) is the engineering work that gets it there.

✗ Debugging before mitigating — the #1 trap

The instinct of a technically strong engineer is to understand before acting. In an incident this instinct kills your SLO. A 40-minute root-cause investigation while 8% of requests fail is 40 minutes of error budget burned that a 5-minute rollback would have stopped. The correct posture: first action is always the fastest path to "users are no longer affected," even if that path is "roll back to yesterday's image and ask questions tomorrow." Understanding is the postmortem's job, not the incident's.

✗ The postmortem action items that never get done

The most common postmortem failure mode: a thorough document with six action items, none assigned to a specific person, none in a tracked system, all marked "OPEN" eight months later. Action items without owners are wishes. Wishes do not reduce future incident rates. If you leave the postmortem review without every P1 item in a sprint and every P2 item in the backlog, the postmortem did not happen — you wrote a document.

KEY INSIGHTHow this connects

  • CD: Releasing Without Fear (5.4) — the canary deployment that missed this incident was built in 5.4; the rollback was a designed capability, not a panic. The lesson is why frequent deployments with automatic rollback are safer than infrequent big-bang releases — and why the canary's synthetic traffic profile was the enabler that let the bug through.
  • Observability: Logs, Metrics, Traces (5.7) — the alert that woke the on-call engineer came from Prometheus metrics; the incident timeline was reconstructed from structured logs with correlation IDs; the trace waterfall showed exactly where the timeout cascaded through queue → worker. Without the instrumentation built in 5.7, the queue spiral would have been invisible until users started calling.
  • SLOs, Error Budgets, and Alerts That Don't Cry Wolf (5.8) — the SLO error budget burning at 83× normal was the quantified signal that made declaring SEV1 unambiguous. The lesson defines what to promise and what to wake up for; this incident is exactly the scenario the error-budget model is designed to price.
  • Scaling Stories: What Breaks at Each Order of Magnitude (6.1) — Stage 6 opens with what happens when the system survives its first incident and then grows. The judgment questions shift from "how do I ship it" to "what do I choose not to build."

Stage 5 ends here. The system is observable, deployed, and incident-tested. The half of software engineering that courses skip — everything after you merge the PR — is no longer foreign territory. Stage 6 is where the judgment questions begin: not "how" but "what, why, and at what cost."

GLOSSARYKey terms introduced

Incident commander (IC)
The single person who owns the incident channel, makes coordination decisions, and drives toward mitigation — not necessarily the person who fixes the system.
Time-to-mitigation (TTM)
The elapsed time from alert firing to users no longer experiencing the impact. The primary incident performance metric — distinct from time-to-resolution or time-to-root-cause.
Retry amplification
A failure mode where a retry policy multiplies queue depth or downstream load: N failures become N×retries, turning a degraded component into a queue spiral under load.
Trigger / Enabler / Amplifier
A three-way contributing-factor taxonomy for blameless postmortems. The trigger directly caused the failure; enablers allowed it to reach production; amplifiers made the impact larger than the underlying failure rate.
Blameless postmortem
A structured post-incident review focused on system improvement rather than individual fault-finding. Produces a timeline, impact quantification, contributing-factor analysis, and tracked action items.
Error budget
The portion of acceptable downtime/errors implied by an SLO. An incident "burns" error budget; a fast mitigation conserves it. Budget depletion triggers a freeze on risky changes.
Dead-letter queue (DLQ)
A queue that receives messages that have exhausted their retry budget, preventing indefinite re-processing and enabling later inspection without blocking the main queue.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Why is "mitigate first, diagnose later" a principle rather than a suggestion — and when, if ever, would you violate it?›

Mitigation stops the user impact; diagnosis finds the cause. These run on different clocks. A 5-minute rollback may consume 5 minutes of error budget; a 40-minute root-cause investigation while users are still affected consumes 40. The principle holds because in almost every incident the fastest mitigation path (rollback, scale out, redirect traffic) is known before the root cause is. The exception is narrow: when the mitigation action itself has a significant risk of making things worse (e.g., a rollback that would also roll back a critical data migration). In that case, a short diagnostic pause to validate the rollback path is warranted — but "short" means minutes, not hours.

2Explain retry amplification. What design choices made it worse in this incident, and what would reduce it?›

Retry amplification: when a component fails and retries, each failure consumes multiple units of capacity (queue slots, worker time) instead of one. In this incident, a 3× retry policy with no jitter meant every timeout produced three queue entries; under morning load, the queue grew faster than workers could drain it. Reducing it requires: (1) a retry budget (max retries with exponential backoff + jitter to prevent synchronized storms), (2) a dead-letter queue so failed messages leave the main queue after exhausting retries, and (3) admission control at the ingestion entrypoint to reject oversized payloads before they enter the queue.

3What is the difference between a "trigger," an "enabler," and an "amplifier" in a postmortem, and why does the distinction matter for action items?›

The trigger is the immediate cause — the config change that lowered the timeout. An enabler is a condition that allowed the trigger to reach production without detection — the synthetic canary data that didn't exercise large payloads. An amplifier made the impact larger than the trigger alone would explain — the 3× retry policy. The distinction matters because each points to a different fix: addressing the trigger might just mean "don't lower timeouts without testing"; addressing the enabler requires improving the canary traffic profile; addressing the amplifier requires redesigning the retry policy. Conflating them into "root cause" would produce a single action item that only prevents this exact incident, not the class of incidents.

4What is the test that distinguishes a real blameless postmortem from a compliance ritual?›

The test: are the action items in a tracked system, assigned to named people, with due dates, before the meeting ends? A blameless postmortem that produces a well-written document with unassigned action items is a compliance ritual with extra steps. The real signal is what happens six weeks later — whether the P1 items were actually completed and whether the system behaved differently the next time a similar condition occurred. Postmortems are only worth writing if the organization is willing to do the work the postmortem identifies.

Stage 5 complete — TraceLab v4 is shipped and survived. Stage 6: Judgment →