ProjekinAja. Software Engineering Stage 4 · Architecture Lesson 4.9
● Stage 4 · Lesson 4.9

TraceLab v3: The Architecture Review

Redesign TraceLab with everything this stage taught, write the design doc, and defend it against three reviewers who don't agree with each other.

⏱ ~28 min read 🏗 Milestone synthesis 🎓 Stage 4 capstone

🎯 By the end you'll understand

How to consolidate queue, cache, boundaries, and migration decisions into a coherent design — and record them in ADRs that capture context, options considered, tradeoffs, the decision, and reversibility. Why there is no best architecture, only fit-to-context. How to distinguish a defensible tradeoff from a rationalized habit in a live design review. What TraceLab v3 looks like when the God-module debt, the CSV debt, the ingest fragility, and the dashboard latency are all paid simultaneously — and what it still does not do.

KEY INSIGHTArchitecture that isn't written down is folklore

Every system has an architecture. The question is whether it was decided or accumulated — and whether, six months from now, anyone can reconstruct the reasoning.

Stage 4 has been a single long argument: distribution is a cost, not a promotion. Every lesson added one component to TraceLab — an ingest queue, a read cache, a clean boundary, a zero-downtime migration — but only after showing concretely what the pain was that made the addition worth the tax. Now those pieces land together as v3. Before touching more code, you write the design doc. Not to satisfy a process, but because writing the design is how you find out whether it is actually coherent.

🔑 Hold onto this

An Architecture Decision Record (ADR) forces three disciplines that most "architecture" avoids. First, it requires stating context — the specific constraints that made this decision non-trivial. Second, it requires listing options considered — including the ones you rejected, and why. Third, it requires writing down reversibility — the honest answer to "what would it take to undo this?" Design reviews are the social mechanism that validates whether your ADRs hold up under people who weren't in the room when you made the decision. The reviewer who disagrees is not the enemy of the design. They are the stress test.

CORE CONCEPTWhat v3 actually contains

Here is the inventory. Each item is a decision, not a feature — and each one was earned by a specific failure mode that v2 demonstrated.

1

Boundary surgery: God-module carved into three modules

The single tracelab.py that grew from v0 through v2 now splits at two seams: ingest.py (write path: validate, enqueue), store.py (persistence: SQLite reads and writes), and query.py (read path: dashboard projections). The rule from Lesson 4.1: a boundary is justified when it prevents a high-frequency change in one place from forcing a change in another. Ingest schema changes happen weekly. Dashboard queries change daily. They should not co-evolve.

2

Ingest queue: SQLite-backed, idempotent consumers

Adopted from Lesson 4.4: a pending_runs table acts as a durable queue. Writers append rows; a background worker drains them. Each row carries a run_id; the worker's upsert is idempotent — a duplicate delivery produces the same row, not a second one. The cost: one extra DB round-trip per ingest. The benefit: ingest never blocks on slow writes, and a crashed worker loses nothing.

3

Dashboard cache: TTL-keyed, cache-aside

Adopted from Lesson 4.5: the dashboard endpoint checks an in-process dict keyed by (query_hash, floor(time / TTL)) before hitting SQLite. TTL is configurable; default 30 s. The cost: stale reads within TTL. The benefit: dashboard load under rapid re-ingest no longer scales with run count. This is the correct cache placement because the dashboard is read-heavy and the data changes on write, not spontaneously.

4

CSV debt paid: expand-contract migration to SQLite

The v0 CSV "schema" lived in nobody's head (flagged in Lesson 1.1's debt table, repayment assigned to Stage 4). Using the expand-contract pattern from Lesson 4.8: Phase 1 writes both CSV and SQLite; Phase 2 reads from SQLite, writes both; Phase 3 drops the CSV write. Each phase is a separate deploy. The migration is zero-downtime because no phase requires the old and new formats to be simultaneously valid — only simultaneously present.

5

Injection hardening: parameterized queries throughout

Adopted from Lesson 4.7: all SQLite writes use cursor.execute(sql, params), never string interpolation. The SQL injection vector from the old "SELECT … WHERE name = '" + name + "'" pattern is gone. This is not a tradeoff — it is a constraint. The only design question was whether to enforce it by code review or by linting; v3 enforces it with a project-wide ruff rule (S608).

ingest.py validate · enqueue pending_runs durable queue (SQLite table) worker store.py runs table · SQLite cache layer TTL 30 s · in-proc dict query.py projections · filters dashboard HTTP handler write path read path
TraceLab v3. The God-module is gone: ingest, persistence, and query are separate modules. The queue decouples the write path; the cache decouples the read path from the run rate. The distribution tax: one in-process background thread — not a separate service.
💡 Still a monolith — on purpose

V3 is a single deployable Python process. The queue lives in SQLite, not RabbitMQ. The cache lives in a dict, not Redis. This is the stage's thesis applied: every distributed component only after the monolith demonstrably hurts. The in-process queue is good enough for one lab running a few hundred experiments per day. When TraceLab runs at ten thousand experiments per hour for fifty simultaneous researchers, revisit the queue. Not before.

HOW IT WORKSWriting the design doc: what an ADR actually contains

An Architecture Decision Record is not a design document in the sense of a prose narrative that describes the final state. It is a structured record of a single decision — why it was non-trivial, what alternatives existed, and what the accepting party is committing to tolerate. The canonical format, popularized by Michael Nygard in 2011, is short: title, status, context, decision, consequences. We extend it with two fields that matter for TraceLab's specific situation: options considered (the rejected paths, with honest reasons) and reversibility (because "we can always change this" is the rationalization engineers use to avoid committing, and writing down what "change this" would actually cost forces honesty).

Here are two of v3's ADRs — the queue decision and the cache decision — in full, because seeing the format on real decisions is more useful than reading a description of the format.

ADR-001: Ingest Queue Implementation Status: Accepted Context: TraceLab v2 ingest blocks synchronously on every SQLite write. Under concurrent robot telemetry loads (>10 writes/s), writers queue at the OS level and ingest latency spikes. More critically, if the SQLite write fails mid-row (disk full, lock contention), the caller receives an error and must decide whether to retry — creating duplicate-run risk. The write path needs decoupling from the persistence path. Options considered: A. External message broker (RabbitMQ, Redis Streams): decouples fully; adds operational complexity, a new service to deploy and monitor, and a network hop. B. In-process threading queue (queue.Queue): zero ops cost; lost on process crash; no persistence guarantee. C. SQLite-backed queue (pending_runs table): durable across crashes; zero new dependencies; slightly higher write latency than option B; requires idempotent consumer logic. Decision: Option C. TraceLab is a single-process application running on researcher hardware. The persistence guarantee matters (losing a run on crash is unacceptable); the operational complexity of option A does not pay for itself at this scale. Consequences: Ingest write latency increases by one SQLite round-trip (~0.3 ms on NVMe). The background worker must implement idempotent upserts (run_id as primary key). If ingest throughput exceeds ~500 rows/s, SQLite contention will become the bottleneck — at that point, re-evaluate option A. Reversibility: Moderate. Replacing the pending_runs table with an external broker requires changing the worker interface but not the ingest or query paths. Estimated: 1-2 days of work, no data migration.
ADR-002: Dashboard Cache Strategy Status: Accepted Context: Dashboard page load re-runs aggregation queries over the full runs table on every HTTP request. With 10,000 runs, the primary sort + filter query takes ~120 ms on an M2 MacBook. Under rapid re-ingest (30 Hz telemetry from a robot), every page load is a full-table scan. Cache is warranted. Options considered: A. No cache: simple, always fresh, pain above ~5,000 runs. B. In-process dict with TTL key (floor(time/TTL)): zero dependencies; lost on restart; per-process (breaks if multi-process); correct for single-process deployment. C. Redis: persistent across restarts; survives multi-process; requires a Redis deployment. D. HTTP caching (Cache-Control / ETags): browser-level; doesn't reduce DB load. Decision: Option B. Single-process deployment is the current constraint. The 30-second TTL is configurable; researchers can tune it per experiment session. Stale reads within TTL are acceptable — a dashboard that shows 29-second-old data during an active experiment is fine. Consequences: Dashboard data is stale by up to TTL seconds. If TraceLab is ever run as a multi-process WSGI application, the per-process cache will diverge — each worker has its own dict. The cache key must include query parameters; a missing parameter produces a single shared entry for all queries. Reversibility: Easy. The cache is one module (cache.py, ~40 lines). Swapping to Redis requires changing the backing store implementation only — no call sites change if the interface is stable.

Notice what these ADRs do not contain: no boxes-and-arrows diagrams, no "architecture vision" narrative, no "future roadmap." They contain exactly the information a new engineer needs to either defend the decision or replace it: what problem existed, what was tried, what was chosen, and what it would cost to undo. That is the minimum viable design document.

📖 ADR vs. design doc

A design doc answers "what are we building?" An ADR answers "why did we make this specific call?" Real projects need both: the design doc as a snapshot of the system at a point in time, the ADRs as an append-only log of the decisions that shaped it. When they conflict, the ADR log is truth — the design doc is often out of date.

SEE ITDesign review simulator

A real design review has people in it — and people who weren't in the room when you made the decisions. The three reviewers below represent distinct review archetypes: each raises the objection their role demands. For each decision you submitted, select the best defense. The rubric rewards acknowledging real costs over claiming the decision is perfect.

⚙ Interactive · Design review simulator

Configure your v3 design choices, then face three reviewers. For each objection, pick the most defensible response. Scores update as you go.

Your v3 design choices
1. Queue implementation
2. Cache placement
3. CSV migration strategy
4. Module boundary strategy

Two patterns separate strong defenses from weak ones in the simulator — and in real reviews. Acknowledging the cost ("yes, TTL means up to 30-second stale data — that's acceptable because the dashboard is diagnostic, not transactional") is always stronger than claiming the cost doesn't exist. And naming the counterfactual clearly ("we'd switch to Redis if and only if we moved to multi-process deployment") gives the reviewer something to test: is the condition real? If it is, the decision is defensible. If it isn't, you've rationalized, not designed.

IN PRACTICEWhat a good defense looks like — and what it doesn't

Design reviews fail in two characteristic ways. The first is the defensive architect: every objection is met with "we considered that and it's not a problem" or "that's handled." This reviewer learns nothing, because no cost is ever acknowledged. Real reviewers stop engaging and start lobbying elsewhere. The second failure is the apologetic architect: every objection triggers a design change, including contradictory ones. After four rounds of "good point," the design satisfies all its reviewers simultaneously, which means it satisfies none of their actual constraints — it's just had every tradeoff averaged out.

Response patternWhat it signalsScore weight
Acknowledge cost + state the condition under which you'd changeYou modeled the tradeoff consciously; you know when it breaks+3
Restate the context that makes this the right tradeoff for this systemYou understand fit-to-context, not best-in-class+2
Agree with the objection and propose a mitigation that preserves the core decisionGood judgment under pressure — you can update without capitulating+2
Claim the objection doesn't apply (without showing why)You haven't modeled the failure0
Change the design to satisfy the reviewer without explaining whyYou're optimizing for approval, not correctness−1
Cite the architecture as "industry best practice" or "what everyone does"You're using social proof to avoid defending the technical decision−2
⚠ The "we'll optimize later" defense

"We can always optimize this later" is valid exactly once per component, and only if you have named the specific metric that will trigger the optimization and the specific change you will make. "We'll cache it if it's slow" is not a plan. "We'll add Redis if the dashboard P95 exceeds 500 ms under multi-process deployment" is a plan — because it's falsifiable, and the condition is something you can measure.

💡 The reviewer who asks "what's the failure mode?"

The best review question is not "is this the right decision?" but "what does this look like when it fails?" For v3's SQLite queue: the failure mode is a background worker that stops draining while ingest continues — the pending_runs table grows unboundedly. The correct response is not "that won't happen" but "we monitor pending_runs.count and alert at 1,000 pending rows." A design that can describe its failure mode and monitoring is more credible than one that claims to be failure-proof.

TRACELABTraceLab v3 delivered — and its remaining debts

With the design reviewed and ADRs written, here is the actual v3 module structure. This is the concrete artifact, not the diagram:

tracelab/v3 layout
tracelab/
├── ingest.py       # validate_run(data) → enqueue to pending_runs
├── store.py        # SQLite schema, idempotent upsert, migrations
├── query.py        # dashboard projections, filter logic
├── cache.py        # TTL dict cache, keyed by (query_hash, time_bucket)
├── worker.py       # background thread: drain pending_runs → runs table
├── server.py       # Flask/HTTP: POST /run, GET /dashboard
├── migrations/
│   ├── 001_create_runs.sql
│   ├── 002_add_pending_queue.sql
│   └── 003_csv_phase1_dual_write.sql
├── adrs/
│   ├── ADR-001-queue.md
│   └── ADR-002-cache.md
└── tests/
    ├── test_ingest.py
    ├── test_store.py
    └── test_worker_idempotent.py

The idempotent upsert is the single most important implementation detail in the worker. Here is the correct pattern — and the wrong one it replaces:

store.pythe upsert that makes the queue safe
# WRONG — vulnerable to duplicate delivery creating duplicate rows
def insert_run_wrong(conn, run):
    conn.execute(
        "INSERT INTO runs(run_id, ts, metrics) VALUES (?, ?, ?)",
        (run['run_id'], run['ts'], run['metrics'])
    )

# CORRECT — idempotent: re-inserting the same run_id is a no-op
def upsert_run(conn, run):
    conn.execute(
            "INSERT INTO runs(run_id, ts, metrics) VALUES (?, ?, ?)"
            "ON CONFLICT(run_id) DO UPDATE SET ts=excluded.ts, metrics=excluded.metrics",
        (run['run_id'], run['ts'], run['metrics'])
    )

The ON CONFLICT … DO UPDATE clause (SQL's UPSERT, available in SQLite 3.24+, 2018) is the mechanism that makes idempotency a property of the persistence layer, not the caller. The worker does not need to check "did I already process this row?" The database enforces it.

Now for the honest part. TraceLab v3 is better than v2 in every dimension the Stage 4 curriculum addressed. Here is what it still does not do — the next debt ledger:

Remaining debt in v3The day it bitesAddressed in
No structured logging — errors write to stderr and disappearA worker crash at 3 a.m. leaves no trace of what it was processingStage 5 (observability)
No health endpoint — the worker thread can die silentlyIngest looks fine; pending_runs grows without alertStage 5 (shipping & running)
SQLite is a single-writer bottleneck above ~500 writes/sMulti-robot lab with high-frequency telemetry; writes start timing outStage 4 note / Stage 5 scale-out
No authentication — any localhost caller can ingest or queryMulti-user lab environment; runs from other researchers pollute your dashboardStage 5 (configuration & secrets)
Migration phase 3 (drop CSV write) not yet executedDisk fills up writing a format nobody readsComplete when v3 has been stable for 2 weeks
🔑 The milestone

TraceLab v3 — architected. The God module is carved into clean boundaries. The ingest queue is durable and idempotent. The dashboard cache prevents dashboard load from scaling with run rate. The v0 CSV debt is paid off with a zero-downtime expand-contract migration. The injection attack surface is closed. A real design doc and two ADRs exist. The architecture was defended against three reviewers. This is what Stage 4 was for.

WEAKNESSWhere this bites

✗ Writing ADRs after the fact to justify a decision already made

An ADR written to document a decision that has already been implemented and is already in production is not an architecture decision record — it is a retrospective justification. The tells: the "options considered" section lists only one real option (the one chosen) and two obvious strawmen. The "context" describes the world as it currently is, not as it was when the decision was made. Retroactive ADRs are better than nothing, but they don't provide the failure mode you need: the moment of genuine uncertainty, when the options were live and the costs were real. If you write the ADR before writing the code, you occasionally notice that you can't fill in "reversibility" honestly — which means the decision wasn't ready to be made.

✗ Treating the design review as a sign-off ceremony

When the design review is scheduled after the implementation is already merged, it has become a sign-off ceremony. The reviewer knows their objections won't change anything; the presenter knows they're not expected to change anything; everyone performs thirty minutes of professional theatre and goes home. Real design reviews happen before implementation, with stakes. The TraceLab review simulator above is uncomfortable because the reviewers might be right — that's the point. If your review process is never uncomfortable, it isn't reviewing anything.

✗ Mistaking "documented" for "understood"

A codebase with thorough ADRs but no one who has read them is no better off than one with no ADRs — arguably worse, because the false confidence is higher. ADRs are only useful if the team has a norm of reading them when touching a component, and a norm of updating them when a decision is reversed. Both norms require active enforcement. The first sign that neither norm exists: the ADR says "we chose SQLite; reversibility is easy" while the codebase has SQLite calls in eleven files, none of them going through the store.py abstraction the ADR assumed.

KEY INSIGHTHow this connects

The move Stage 4 asked you to make is the hardest in software: stop adding features and start making decisions. Every lesson in this stage was a single decision — where to put the boundary, whether to add the queue, what the cache invalidation policy should be — and each decision had a "when NOT to do this" verdict that mattered as much as the technique itself. V3 is what that discipline produces: a system that is harder to change in some ways (three modules instead of one file) and much easier to change in the ways that matter (the write path and read path can evolve independently). That is the correct outcome.

GLOSSARYKey terms introduced

Architecture Decision Record (ADR)
A short structured document recording one architectural decision: its context, the options considered, the decision made, its consequences, and its reversibility. Append-only — decisions are superseded, not deleted.
Idempotent consumer
A queue consumer whose operation produces the same result whether the message is processed once or many times. Required for at-least-once delivery queues. Implemented in v3 via SQL UPSERT on run_id.
Expand-contract migration
A three-phase zero-downtime schema migration: Phase 1 adds the new column/table while keeping the old; Phase 2 reads from new, writes both; Phase 3 removes the old. No phase requires both formats to be simultaneously valid.
Cache-aside pattern
The application code checks the cache before querying the database; on a cache miss, it fetches from the database and writes to the cache. The database is the authority; the cache is a performance optimisation the application maintains explicitly.
God module
A single source file or class that owns too many concerns: ingestion, validation, persistence, querying, and output formatting all in one place. Changes to any concern risk breaking the others. Splits at seams where change rates differ.
Design review
A structured meeting where a design is presented to reviewers who were not involved in its creation, with the explicit goal of surfacing objections before implementation. Distinguished from a sign-off ceremony by having stakes: the design can be changed.
Fit-to-context
The principle that architectural decisions are evaluated against a specific system's constraints (scale, team size, reliability requirements, operational maturity) — not against a universal best practice. An in-process cache is correct for a single-process tool and wrong for a horizontally-scaled service.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What three things does an ADR force you to write down that most informal design discussions skip?›

Context (the specific constraints that made the decision non-trivial), options considered (including the rejected ones, with honest reasons), and reversibility (what it would actually cost to undo the decision). The reversibility field is the one most often omitted — and its omission is usually a sign that the decision hasn't been fully thought through.

2Why is an in-process dict cache correct for v3 but wrong if TraceLab were deployed as a multi-process WSGI application?›

Each WSGI worker process has its own dict. A cache miss in process A writes a fresh entry to A's dict; process B still misses and fetches from the database. The cache provides no cross-process benefit, and each process independently runs the same expensive query. For multi-process deployments, the cache needs to be shared — Redis or Memcached. This is exactly the condition named in ADR-002's reversibility section: "if we move to multi-process deployment, switch to Redis."

3What makes the expand-contract migration "zero-downtime" compared to a big-bang migration?›

In a big-bang migration, there is a moment when neither the old format nor the new format is fully valid — the maintenance window. Any failure during that window leaves the system in an inconsistent state. In expand-contract, no such moment exists: Phase 1 adds the new schema while the old still works; Phase 2 keeps both valid and starts preferring the new; Phase 3 removes the old only after the new has been stable. Each phase can be rolled back independently. The cost is that the migration takes three deployments instead of one.

4What distinguishes a defensible tradeoff from a rationalized habit in a design review?›

A defensible tradeoff names the cost explicitly, states the context that makes the cost acceptable, and specifies the condition under which the decision would change — a condition that is falsifiable and measurable. A rationalized habit claims the cost doesn't exist, or that the alternative wasn't considered, or that "this is how everyone does it." The test: ask "what would have to be true for you to make a different decision?" A defensible tradeoff has a concrete answer; a rationalized habit produces evasions or restates the current decision.

Stage 4 complete — next: shipping v3 somewhere real →