TraceLab v3: The Architecture Review
Redesign TraceLab with everything this stage taught, write the design doc, and defend it against three reviewers who don't agree with each other.
🎯 By the end you'll understand
How to consolidate queue, cache, boundaries, and migration decisions into a coherent design — and record them in ADRs that capture context, options considered, tradeoffs, the decision, and reversibility. Why there is no best architecture, only fit-to-context. How to distinguish a defensible tradeoff from a rationalized habit in a live design review. What TraceLab v3 looks like when the God-module debt, the CSV debt, the ingest fragility, and the dashboard latency are all paid simultaneously — and what it still does not do.
KEY INSIGHTArchitecture that isn't written down is folklore
Every system has an architecture. The question is whether it was decided or accumulated — and whether, six months from now, anyone can reconstruct the reasoning.
Stage 4 has been a single long argument: distribution is a cost, not a promotion. Every lesson added one component to TraceLab — an ingest queue, a read cache, a clean boundary, a zero-downtime migration — but only after showing concretely what the pain was that made the addition worth the tax. Now those pieces land together as v3. Before touching more code, you write the design doc. Not to satisfy a process, but because writing the design is how you find out whether it is actually coherent.
An Architecture Decision Record (ADR) forces three disciplines that most "architecture" avoids. First, it requires stating context — the specific constraints that made this decision non-trivial. Second, it requires listing options considered — including the ones you rejected, and why. Third, it requires writing down reversibility — the honest answer to "what would it take to undo this?" Design reviews are the social mechanism that validates whether your ADRs hold up under people who weren't in the room when you made the decision. The reviewer who disagrees is not the enemy of the design. They are the stress test.
CORE CONCEPTWhat v3 actually contains
Here is the inventory. Each item is a decision, not a feature — and each one was earned by a specific failure mode that v2 demonstrated.
Boundary surgery: God-module carved into three modules
The single tracelab.py that grew from v0 through v2 now splits at two seams: ingest.py (write path: validate, enqueue), store.py (persistence: SQLite reads and writes), and query.py (read path: dashboard projections). The rule from Lesson 4.1: a boundary is justified when it prevents a high-frequency change in one place from forcing a change in another. Ingest schema changes happen weekly. Dashboard queries change daily. They should not co-evolve.
Ingest queue: SQLite-backed, idempotent consumers
Adopted from Lesson 4.4: a pending_runs table acts as a durable queue. Writers append rows; a background worker drains them. Each row carries a run_id; the worker's upsert is idempotent — a duplicate delivery produces the same row, not a second one. The cost: one extra DB round-trip per ingest. The benefit: ingest never blocks on slow writes, and a crashed worker loses nothing.
Dashboard cache: TTL-keyed, cache-aside
Adopted from Lesson 4.5: the dashboard endpoint checks an in-process dict keyed by (query_hash, floor(time / TTL)) before hitting SQLite. TTL is configurable; default 30 s. The cost: stale reads within TTL. The benefit: dashboard load under rapid re-ingest no longer scales with run count. This is the correct cache placement because the dashboard is read-heavy and the data changes on write, not spontaneously.
CSV debt paid: expand-contract migration to SQLite
The v0 CSV "schema" lived in nobody's head (flagged in Lesson 1.1's debt table, repayment assigned to Stage 4). Using the expand-contract pattern from Lesson 4.8: Phase 1 writes both CSV and SQLite; Phase 2 reads from SQLite, writes both; Phase 3 drops the CSV write. Each phase is a separate deploy. The migration is zero-downtime because no phase requires the old and new formats to be simultaneously valid — only simultaneously present.
Injection hardening: parameterized queries throughout
Adopted from Lesson 4.7: all SQLite writes use cursor.execute(sql, params), never string interpolation. The SQL injection vector from the old "SELECT … WHERE name = '" + name + "'" pattern is gone. This is not a tradeoff — it is a constraint. The only design question was whether to enforce it by code review or by linting; v3 enforces it with a project-wide ruff rule (S608).
V3 is a single deployable Python process. The queue lives in SQLite, not RabbitMQ. The cache lives in a dict, not Redis. This is the stage's thesis applied: every distributed component only after the monolith demonstrably hurts. The in-process queue is good enough for one lab running a few hundred experiments per day. When TraceLab runs at ten thousand experiments per hour for fifty simultaneous researchers, revisit the queue. Not before.
HOW IT WORKSWriting the design doc: what an ADR actually contains
An Architecture Decision Record is not a design document in the sense of a prose narrative that describes the final state. It is a structured record of a single decision — why it was non-trivial, what alternatives existed, and what the accepting party is committing to tolerate. The canonical format, popularized by Michael Nygard in 2011, is short: title, status, context, decision, consequences. We extend it with two fields that matter for TraceLab's specific situation: options considered (the rejected paths, with honest reasons) and reversibility (because "we can always change this" is the rationalization engineers use to avoid committing, and writing down what "change this" would actually cost forces honesty).
Here are two of v3's ADRs — the queue decision and the cache decision — in full, because seeing the format on real decisions is more useful than reading a description of the format.
cache.py, ~40 lines). Swapping to Redis requires changing the backing store implementation only — no call sites change if the interface is stable.Notice what these ADRs do not contain: no boxes-and-arrows diagrams, no "architecture vision" narrative, no "future roadmap." They contain exactly the information a new engineer needs to either defend the decision or replace it: what problem existed, what was tried, what was chosen, and what it would cost to undo. That is the minimum viable design document.
A design doc answers "what are we building?" An ADR answers "why did we make this specific call?" Real projects need both: the design doc as a snapshot of the system at a point in time, the ADRs as an append-only log of the decisions that shaped it. When they conflict, the ADR log is truth — the design doc is often out of date.
SEE ITDesign review simulator
A real design review has people in it — and people who weren't in the room when you made the decisions. The three reviewers below represent distinct review archetypes: each raises the objection their role demands. For each decision you submitted, select the best defense. The rubric rewards acknowledging real costs over claiming the decision is perfect.
Two patterns separate strong defenses from weak ones in the simulator — and in real reviews. Acknowledging the cost ("yes, TTL means up to 30-second stale data — that's acceptable because the dashboard is diagnostic, not transactional") is always stronger than claiming the cost doesn't exist. And naming the counterfactual clearly ("we'd switch to Redis if and only if we moved to multi-process deployment") gives the reviewer something to test: is the condition real? If it is, the decision is defensible. If it isn't, you've rationalized, not designed.
IN PRACTICEWhat a good defense looks like — and what it doesn't
Design reviews fail in two characteristic ways. The first is the defensive architect: every objection is met with "we considered that and it's not a problem" or "that's handled." This reviewer learns nothing, because no cost is ever acknowledged. Real reviewers stop engaging and start lobbying elsewhere. The second failure is the apologetic architect: every objection triggers a design change, including contradictory ones. After four rounds of "good point," the design satisfies all its reviewers simultaneously, which means it satisfies none of their actual constraints — it's just had every tradeoff averaged out.
| Response pattern | What it signals | Score weight |
|---|---|---|
| Acknowledge cost + state the condition under which you'd change | You modeled the tradeoff consciously; you know when it breaks | +3 |
| Restate the context that makes this the right tradeoff for this system | You understand fit-to-context, not best-in-class | +2 |
| Agree with the objection and propose a mitigation that preserves the core decision | Good judgment under pressure — you can update without capitulating | +2 |
| Claim the objection doesn't apply (without showing why) | You haven't modeled the failure | 0 |
| Change the design to satisfy the reviewer without explaining why | You're optimizing for approval, not correctness | −1 |
| Cite the architecture as "industry best practice" or "what everyone does" | You're using social proof to avoid defending the technical decision | −2 |
"We can always optimize this later" is valid exactly once per component, and only if you have named the specific metric that will trigger the optimization and the specific change you will make. "We'll cache it if it's slow" is not a plan. "We'll add Redis if the dashboard P95 exceeds 500 ms under multi-process deployment" is a plan — because it's falsifiable, and the condition is something you can measure.
The best review question is not "is this the right decision?" but "what does this look like when it fails?" For v3's SQLite queue: the failure mode is a background worker that stops draining while ingest continues — the pending_runs table grows unboundedly. The correct response is not "that won't happen" but "we monitor pending_runs.count and alert at 1,000 pending rows." A design that can describe its failure mode and monitoring is more credible than one that claims to be failure-proof.
TRACELABTraceLab v3 delivered — and its remaining debts
With the design reviewed and ADRs written, here is the actual v3 module structure. This is the concrete artifact, not the diagram:
tracelab/ ├── ingest.py # validate_run(data) → enqueue to pending_runs ├── store.py # SQLite schema, idempotent upsert, migrations ├── query.py # dashboard projections, filter logic ├── cache.py # TTL dict cache, keyed by (query_hash, time_bucket) ├── worker.py # background thread: drain pending_runs → runs table ├── server.py # Flask/HTTP: POST /run, GET /dashboard ├── migrations/ │ ├── 001_create_runs.sql │ ├── 002_add_pending_queue.sql │ └── 003_csv_phase1_dual_write.sql ├── adrs/ │ ├── ADR-001-queue.md │ └── ADR-002-cache.md └── tests/ ├── test_ingest.py ├── test_store.py └── test_worker_idempotent.py
The idempotent upsert is the single most important implementation detail in the worker. Here is the correct pattern — and the wrong one it replaces:
# WRONG — vulnerable to duplicate delivery creating duplicate rows def insert_run_wrong(conn, run): conn.execute( "INSERT INTO runs(run_id, ts, metrics) VALUES (?, ?, ?)", (run['run_id'], run['ts'], run['metrics']) ) # CORRECT — idempotent: re-inserting the same run_id is a no-op def upsert_run(conn, run): conn.execute( "INSERT INTO runs(run_id, ts, metrics) VALUES (?, ?, ?)" "ON CONFLICT(run_id) DO UPDATE SET ts=excluded.ts, metrics=excluded.metrics", (run['run_id'], run['ts'], run['metrics']) )
The ON CONFLICT … DO UPDATE clause (SQL's UPSERT, available in SQLite 3.24+, 2018) is the mechanism that makes idempotency a property of the persistence layer, not the caller. The worker does not need to check "did I already process this row?" The database enforces it.
Now for the honest part. TraceLab v3 is better than v2 in every dimension the Stage 4 curriculum addressed. Here is what it still does not do — the next debt ledger:
| Remaining debt in v3 | The day it bites | Addressed in |
|---|---|---|
| No structured logging — errors write to stderr and disappear | A worker crash at 3 a.m. leaves no trace of what it was processing | Stage 5 (observability) |
| No health endpoint — the worker thread can die silently | Ingest looks fine; pending_runs grows without alert | Stage 5 (shipping & running) |
| SQLite is a single-writer bottleneck above ~500 writes/s | Multi-robot lab with high-frequency telemetry; writes start timing out | Stage 4 note / Stage 5 scale-out |
| No authentication — any localhost caller can ingest or query | Multi-user lab environment; runs from other researchers pollute your dashboard | Stage 5 (configuration & secrets) |
| Migration phase 3 (drop CSV write) not yet executed | Disk fills up writing a format nobody reads | Complete when v3 has been stable for 2 weeks |
TraceLab v3 — architected. The God module is carved into clean boundaries. The ingest queue is durable and idempotent. The dashboard cache prevents dashboard load from scaling with run rate. The v0 CSV debt is paid off with a zero-downtime expand-contract migration. The injection attack surface is closed. A real design doc and two ADRs exist. The architecture was defended against three reviewers. This is what Stage 4 was for.
WEAKNESSWhere this bites
An ADR written to document a decision that has already been implemented and is already in production is not an architecture decision record — it is a retrospective justification. The tells: the "options considered" section lists only one real option (the one chosen) and two obvious strawmen. The "context" describes the world as it currently is, not as it was when the decision was made. Retroactive ADRs are better than nothing, but they don't provide the failure mode you need: the moment of genuine uncertainty, when the options were live and the costs were real. If you write the ADR before writing the code, you occasionally notice that you can't fill in "reversibility" honestly — which means the decision wasn't ready to be made.
When the design review is scheduled after the implementation is already merged, it has become a sign-off ceremony. The reviewer knows their objections won't change anything; the presenter knows they're not expected to change anything; everyone performs thirty minutes of professional theatre and goes home. Real design reviews happen before implementation, with stakes. The TraceLab review simulator above is uncomfortable because the reviewers might be right — that's the point. If your review process is never uncomfortable, it isn't reviewing anything.
A codebase with thorough ADRs but no one who has read them is no better off than one with no ADRs — arguably worse, because the false confidence is higher. ADRs are only useful if the team has a norm of reading them when touching a component, and a norm of updating them when a decision is reversed. Both norms require active enforcement. The first sign that neither norm exists: the ADR says "we chose SQLite; reversibility is easy" while the codebase has SQLite calls in eleven files, none of them going through the store.py abstraction the ADR assumed.
KEY INSIGHTHow this connects
- Boundaries (4.1) — the God-module problem v3 finally solves; the rule for when a boundary is justified.
- Monolith vs services (4.2) — why v3 is still a monolith, and the specific threshold at which that changes.
- Caching: The Two Hard Problems (4.3) — the TTL cache in front of the dashboard and the invalidation strategy that makes staleness a conscious tradeoff, not a bug.
- Queues and Async Work (4.4) — the SQLite-backed ingest queue, at-least-once delivery, and the idempotent consumer that makes duplicate delivery safe.
- Migrations (4.8) — the expand-contract pattern that makes the CSV→SQLite migration zero-downtime; the immediately preceding lesson.
- Next: Environments, Configuration, and Secrets (5.1) — v3 is built; now it needs to run somewhere real. Stage 5 takes TraceLab from working-locally to deployed-and-observable.
The move Stage 4 asked you to make is the hardest in software: stop adding features and start making decisions. Every lesson in this stage was a single decision — where to put the boundary, whether to add the queue, what the cache invalidation policy should be — and each decision had a "when NOT to do this" verdict that mattered as much as the technique itself. V3 is what that discipline produces: a system that is harder to change in some ways (three modules instead of one file) and much easier to change in the ways that matter (the write path and read path can evolve independently). That is the correct outcome.
GLOSSARYKey terms introduced
- Architecture Decision Record (ADR)
- A short structured document recording one architectural decision: its context, the options considered, the decision made, its consequences, and its reversibility. Append-only — decisions are superseded, not deleted.
- Idempotent consumer
- A queue consumer whose operation produces the same result whether the message is processed once or many times. Required for at-least-once delivery queues. Implemented in v3 via SQL UPSERT on
run_id. - Expand-contract migration
- A three-phase zero-downtime schema migration: Phase 1 adds the new column/table while keeping the old; Phase 2 reads from new, writes both; Phase 3 removes the old. No phase requires both formats to be simultaneously valid.
- Cache-aside pattern
- The application code checks the cache before querying the database; on a cache miss, it fetches from the database and writes to the cache. The database is the authority; the cache is a performance optimisation the application maintains explicitly.
- God module
- A single source file or class that owns too many concerns: ingestion, validation, persistence, querying, and output formatting all in one place. Changes to any concern risk breaking the others. Splits at seams where change rates differ.
- Design review
- A structured meeting where a design is presented to reviewers who were not involved in its creation, with the explicit goal of surfacing objections before implementation. Distinguished from a sign-off ceremony by having stakes: the design can be changed.
- Fit-to-context
- The principle that architectural decisions are evaluated against a specific system's constraints (scale, team size, reliability requirements, operational maturity) — not against a universal best practice. An in-process cache is correct for a single-process tool and wrong for a horizontally-scaled service.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What three things does an ADR force you to write down that most informal design discussions skip?›
Context (the specific constraints that made the decision non-trivial), options considered (including the rejected ones, with honest reasons), and reversibility (what it would actually cost to undo the decision). The reversibility field is the one most often omitted — and its omission is usually a sign that the decision hasn't been fully thought through.
2Why is an in-process dict cache correct for v3 but wrong if TraceLab were deployed as a multi-process WSGI application?›
Each WSGI worker process has its own dict. A cache miss in process A writes a fresh entry to A's dict; process B still misses and fetches from the database. The cache provides no cross-process benefit, and each process independently runs the same expensive query. For multi-process deployments, the cache needs to be shared — Redis or Memcached. This is exactly the condition named in ADR-002's reversibility section: "if we move to multi-process deployment, switch to Redis."
3What makes the expand-contract migration "zero-downtime" compared to a big-bang migration?›
In a big-bang migration, there is a moment when neither the old format nor the new format is fully valid — the maintenance window. Any failure during that window leaves the system in an inconsistent state. In expand-contract, no such moment exists: Phase 1 adds the new schema while the old still works; Phase 2 keeps both valid and starts preferring the new; Phase 3 removes the old only after the new has been stable. Each phase can be rolled back independently. The cost is that the migration takes three deployments instead of one.
4What distinguishes a defensible tradeoff from a rationalized habit in a design review?›
A defensible tradeoff names the cost explicitly, states the context that makes the cost acceptable, and specifies the condition under which the decision would change — a condition that is falsifiable and measurable. A rationalized habit claims the cost doesn't exist, or that the alternative wasn't considered, or that "this is how everyone does it." The test: ask "what would have to be true for you to make a different decision?" A defensible tradeoff has a concrete answer; a rationalized habit produces evasions or restates the current decision.