TraceLab v2: Client–Server, and the First Race
The script becomes a system — API, Postgres, auth, a CLI client, and a genuine race condition you must catch in the wild.
🎯 By the end you'll understand
How TraceLab's v1 package splits into a server and a CLI client; how the HTTP API, Postgres schema, and token auth layer snap together; why two concurrent telemetry writers silently corrupt a run counter even when each one is individually correct; and how to diagnose and permanently fix that lost-update bug using the exact transaction isolation tools from Lesson 3.6. You'll also name the technical debt deliberately left in place — the God module and the synchronous ingest path — because they are Stage 4's opening problem.
KEY INSIGHTA distributed system is not a fast script — it is a new category of failure
The moment two processes share a mutable counter, correctness stops being about logic and starts being about time.
TraceLab began as twenty-one lines of Python that appended to a CSV file. That file was already a shared resource — two robots writing at the same millisecond would shear a row mid-line. We named that debt in Lesson 1.1 and promised to repay it in Stage 3. This lesson is the payment.
The upgrade is real and the architecture is reasonable: an HTTP API hides the database, token auth enforces identity, a CLI client replaces the raw script. But "reasonable architecture" does not prevent races. It just makes them subtler. The lost update you are about to encounter in this lesson is not a contrived demo — it is the exact bug that corrupted experiment counters in real ML tracking systems before idempotency keys became standard practice.
Every layer you added this stage — HTTP, databases, transactions, threads, async — was a mechanism for managing shared mutable state across time. TraceLab v2 is the first time all of those mechanisms must work together. When they don't, you get a race condition. When they do, you understand why.
HOW IT WORKSFrom script to system: what v2 assembles
TraceLab v1 — the cleaned-up package from Lesson 1.9 — was a single Python module with a log_run() function and a flat CSV file. Every piece lived in one process, on one machine. The v2 milestone reassigns those responsibilities:
| Responsibility | v1 (the script) | v2 (the system) |
|---|---|---|
| Storage | CSV on local disk — one file, one writer | Postgres — row-level locking, ACID transactions |
| Schema enforcement | None (values are strings, commas corrupt) | SQL schema from Lesson 3-4: runs, metrics, foreign keys |
| Access control | File permissions, effectively none | Bearer tokens from Lesson 3-9: every write carries an identity |
| Transport | None — direct function call | HTTP POST to /api/runs and /api/metrics |
| Client | The script is the client | A thin CLI (or library) that speaks HTTP |
| Concurrency | None (one writer at a time) | Multiple robots, multiple experiments — concurrent by default |
The architecture is a standard three-tier system: client tier (CLI), service tier (FastAPI or Flask server), data tier (Postgres). Nothing novel — but the transitions between tiers are where the failures live.
What the server looks like
A minimal FastAPI server for v2 has three endpoints — enough for the robot to open a run, append metrics, and for the CLI to query results:
from fastapi import FastAPI, Depends, HTTPException from pydantic import BaseModel from .auth import verify_token # Lesson 3-9: bearer token check from .db import get_conn # Lesson 3-4/3-6: psycopg2 connection pool app = FastAPI() class RunIn(BaseModel): name: str tags: dict = {} idempotency_key: str = "" # ← the fix, added after the race @app.post("/api/runs") def create_run(body: RunIn, user=Depends(verify_token), conn=Depends(get_conn)): with conn.transaction(): # SELECT ... FOR UPDATE grabs a row lock — prevents the lost update counter = conn.execute( "SELECT val FROM counters WHERE name='run_seq' FOR UPDATE" ).scalar() new_id = counter + 1 conn.execute("UPDATE counters SET val=:v WHERE name='run_seq'", {"v": new_id}) conn.execute("INSERT INTO runs(id,name,owner,tags) VALUES(:id,:n,:o,:t)", {"id": new_id, "n": body.name, "o": user.id, "t": body.tags}) return {"run_id": new_id}
The verify_token dependency is the bearer-token check from Lesson 3-9 — it either returns a User object or raises a 401. The get_conn dependency yields a pooled psycopg2 connection. The schema — the runs, metrics, and counters tables — is the one designed in Lesson 3-4.
What the client looks like
The CLI that replaces the v1 script is thin by design — all business logic lives in the server:
import httpx, os BASE = os.getenv("TRACELAB_URL", "http://localhost:8000") TOKEN = os.getenv("TRACELAB_TOKEN", "") def _headers(): return {"Authorization": f"Bearer {TOKEN}"} def create_run(name, tags=None, idempotency_key=""): r = httpx.post(f"{BASE}/api/runs", json={"name": name, "tags": tags or {}, "idempotency_key": idempotency_key}, headers=_headers()) r.raise_for_status() return r.json()["run_id"] def log_metric(run_id, key, value, step=None): r = httpx.post(f"{BASE}/api/metrics", json={"run_id": run_id, "key": key, "value": value, "step": step}, headers=_headers()) r.raise_for_status()
The client never picks its own run ID. The server assigns it after inserting the row — so the database's uniqueness constraint is the single source of truth. If the client generated the ID, two clients could collide with no way to detect it. This is a general principle: IDs are a responsibility of the authority that owns the namespace.
CORE CONCEPTThe write path: six layers, one request
Tracing a single create_run() call end-to-end reveals how the pieces from this stage interlock. Each layer adds a guarantee — and each also adds a failure mode:
CLI → HTTP request
The client serializes the RunIn payload to JSON and fires a POST /api/runs with a Bearer <token> header. From the client's perspective this is fire-and-wait: it blocks until the server responds or times out. The network adds latency, retries, and the possibility of duplicate delivery — if the TCP connection drops after the server commits but before the 200 returns, the client may retry and create a duplicate run.
Server receives → ASGI dispatch
The ASGI worker (uvicorn) accepts the TCP connection, parses HTTP framing, and routes the request to FastAPI's router. This layer does nothing domain-specific, but it's where the worker concurrency model lives: a single-threaded async worker can handle many concurrent I/O-bound requests, but a slow synchronous handler (like a blocking database call) will stall all of them — the God-module problem we're naming and deferring to Stage 4.
Auth middleware
verify_token checks the Bearer token against the api_tokens table. If the token is absent, expired, or belongs to a revoked user: 401 Unauthorized, the request dies here, no database mutation. This is the auth boundary from Lesson 3-9 doing its job.
Route handler
The handler opens a transaction, runs the counter-read-and-increment, inserts the row, commits. The database's atomicity guarantee means both operations either both complete or both roll back — there is no world where the counter increments but the row doesn't exist.
Postgres transaction
Inside the transaction, the default isolation level (READ COMMITTED) does not prevent the lost update — two transactions can both read counter = 42 before either writes 43. The fix is REPEATABLE READ (or SERIALIZABLE), or a pessimistic lock via SELECT … FOR UPDATE. This is the exact lesson from 3-6.
Response returns
The committed run_id travels back up through the handler, FastAPI serializes it to JSON, uvicorn frames it as an HTTP 201, and the client's httpx.post() call returns. The client stores the ID and uses it for subsequent metric posts.
Every layer in that path corresponds to a lesson in this stage. The sequence was not arbitrary — you needed HTTP semantics (3-1, 3-2) to understand the request/response contract; you needed the relational schema (3-4) to understand what the handler writes; you needed transactions (3-6) to understand the isolation failure; you needed auth (3-9) to understand why the server needs to know who is calling. The race is the capstone because it is the first time all of them interact in one bug.
SEE ITEnd-to-end write-path step-runner
Two clients — Client A (robot-1) and Client B (robot-2) — call create_run() at almost the same moment. Watch one request flow through all six layers. Then interleave them and watch the race appear. Finally, apply the fix and watch it collapse.
The lost update manifests as two clients both being assigned run_id = 43. Each client is individually correct — it reads the counter, increments it, writes it back. The bug lives entirely in the interleaving, which you cannot see from either client alone. This is why race conditions are notoriously hard to catch in code review: the code is right; the timing is wrong.
CORE CONCEPTThe first race condition: anatomy of a lost update
The lost-update anomaly is the simplest class of concurrency bug, and TraceLab's run counter is a textbook specimen. Here is the exact interleaving that produces it at READ COMMITTED isolation (Postgres's default):
| Time | Client A (robot-1) | Client B (robot-2) | DB: counters.val |
|---|---|---|---|
| t₀ | counter = 42 | 42 | |
| t₁ | BEGIN; SELECT val → 42 | 42 | |
| t₂ | BEGIN; SELECT val → 42 | 42 | |
| t₃ | UPDATE SET val = 43 | 43 (A's txn) | |
| t₄ | UPDATE SET val = 43 | 43 (B overwrites A) | |
| t₅ | COMMIT → run_id = 43 | 43 | |
| t₆ | COMMIT → run_id = 43 | 43 | |
| Both clients believe they own run 43. One run is silently lost. | 43 (should be 44) | ||
READ COMMITTED prevents dirty reads — client B cannot see A's uncommitted write. But it does not prevent B from re-reading a value it already read within the same transaction (non-repeatable read), and it does not prevent two transactions from performing a concurrent read-modify-write on the same row. Both of those are permitted at READ COMMITTED. You need a stronger guarantee.
This is the most common reaction. Yes, there are transactions — the atomicity guarantee is intact. The problem is that atomicity alone does not prevent the race. Atomicity means "all or nothing within one transaction." Isolation levels govern what one transaction can observe about another. They are different axes. A transaction at READ COMMITTED is atomic AND racy. You need both.
Reproducing it with psql
You do not need the full server to reproduce this bug. Two psql terminals and the schema from Lesson 3-4 are enough:
-- Both terminals: start at the same time BEGIN; SELECT val FROM counters WHERE name = 'run_seq'; -- both see: 42 -- Terminal A: do the update, then terminal B does the same UPDATE counters SET val = val + 1 WHERE name = 'run_seq'; COMMIT; -- counter is now 43 -- Terminal B (update after A committed): B read 42 and writes 43 again UPDATE counters SET val = val + 1 WHERE name = 'run_seq'; COMMIT; -- counter is still 43 — A's increment was lost
Notice the UPDATE SET val = val + 1 form (as opposed to reading into application code and writing back). When you do the arithmetic in the UPDATE itself, Postgres evaluates val + 1 on the row at lock time — not at read time. This form is actually atomic and avoids the race entirely. The problem in TraceLab's original code is that the read happens in application code (Python), a network round-trip occurs, and then the write happens in a separate SQL statement. The gap between those two statements is where the race lives.
IN PRACTICEDiagnosing and fixing: two complementary tools
The right fix depends on the semantics you need. TraceLab's run counter has two separable problems: the counter race and the duplicate-request problem. They need different tools.
Fix 1: Pessimistic locking (SELECT … FOR UPDATE)
If the application must read the value before writing it (e.g., to return it to the caller), use SELECT … FOR UPDATE. This acquires a row-level exclusive lock at read time, so any other transaction trying to read the same row with FOR UPDATE will block until the first transaction commits:
with conn.transaction(): # This SELECT acquires an exclusive lock on the counters row. # Client B's SELECT FOR UPDATE will block here until A commits. counter = conn.execute( "SELECT val FROM counters WHERE name='run_seq' FOR UPDATE" ).scalar() new_id = counter + 1 conn.execute( "UPDATE counters SET val=:v WHERE name='run_seq'", {"v": new_id} ) conn.execute("INSERT INTO runs(id,name,owner) VALUES(:id,:n,:o)", ...) # When A commits, B unblocks, reads 43, writes 44. Correct.
The tradeoff: FOR UPDATE serializes all run creation through the single counters row — it becomes a global mutex for run IDs. Fine for TraceLab's throughput. Not fine if you need thousands of concurrent run creations per second, at which point you'd use Postgres sequences (SERIAL or GENERATED ALWAYS AS IDENTITY) instead, which are designed for exactly this and don't require application-level locking.
Fix 2: Idempotency keys (for the retry problem)
SELECT FOR UPDATE fixes the counter race but does not fix the duplicate-delivery problem: if the network drops after the server commits but before the 200 returns, the client will retry and create a second run with the same name and the next ID. The user sees two identical runs in their dashboard.
The standard fix is an idempotency key — a UUID the client generates and includes in the request. The server stores it in a unique column; a duplicate request with the same key hits a UNIQUE constraint and returns the original run_id instead of creating a new one:
-- Schema: add idempotency column ALTER TABLE runs ADD COLUMN idempotency_key TEXT UNIQUE; -- Python: INSERT OR IGNORE pattern (Postgres ON CONFLICT) conn.execute(""" INSERT INTO runs(id, name, owner, idempotency_key) VALUES (:id, :n, :o, :ik) ON CONFLICT (idempotency_key) DO NOTHING """, {"id": new_id, "n": name, "o": owner, "ik": idempotency_key}) # Client generates the key: import uuid key = str(uuid.uuid4()) # deterministic per logical request, regenerated on each new call create_run("exp-042", idempotency_key=key) # safe to retry with same key
The idempotency key makes the write operation idempotent — calling it twice with the same key produces the same result as calling it once. This is the correct model for any operation that must survive network retries without creating duplicates. Stripe, PayPal, and every payment API uses this pattern for exactly that reason.
Any operation that follows read-modify-write on shared state needs either an atomic SQL operation (UPDATE SET col = col + 1), a pessimistic lock (FOR UPDATE), or an optimistic lock (check version column, fail if changed). Any operation that crosses a network boundary needs idempotency if the client must retry. These are not TraceLab-specific — they are the standard answers to two universal distributed-systems problems.
TRACELABTraceLab v2 — milestone complete, debts named
Here is the full picture of what TraceLab v2 delivers relative to the v0 debt ledger from Lesson 1.1 and the v1 spec from Lesson 1.9:
| v0 Debt | v2 Status | How it was repaid |
|---|---|---|
No encoding; values glued with + | ✓ Closed | Pydantic model validates inputs; JSON serialization handles escaping |
| Metrics stored as strings | ✓ Closed | Postgres NUMERIC column; no float round-trip corruption |
| Full-file scan on every find | ✓ Closed | SQL index on runs.name; B-tree lookup |
| No history | ✓ Closed | Immutable rows; created_at timestamp; soft-delete only |
| No locking — concurrent writers interleave | ✓ Closed | SELECT FOR UPDATE on the counter + idempotency key |
| CSV schema in nobody's head | ✓ Closed | SQL schema with foreign keys + Pydantic API contract |
All six v0 debts are closed. But v2 opens two new ones, deliberately. They are not accidents — they are the feature of Stage 4:
| v2 Debt | The day it bites | Repaid in |
|---|---|---|
God module: server/main.py owns routing, auth, DB, business logic, and the counter all in one file | Adding a second endpoint requires understanding the whole file; onboarding another engineer to one feature requires understanding all of them | Stage 4 · Lesson 4.1 (Modules, Boundaries, Coupling) |
| Synchronous ingest: each HTTP request blocks a worker thread on the DB call; at 100 concurrent robots the server stalls | Load test shows <10 RPS despite a fast Postgres; CPU is idle but workers are saturated waiting on I/O | Stage 4 · Lesson 4.6 (Async internals) and Lesson 4.7 (Message queues) |
This is how real systems work: you close the debts you understand, you name the debts you are deferring, and you design the next refactor to pay them. The v2 God module is Stage 4's raw material. Lesson 4.1 picks it up.
For reference — the final structure of TraceLab v2 as a Python package, without the God module's bloat collapsed (that's Stage 4's job):
tracelab-v2/ ├── server/ │ ├── main.py # routes, handlers, auth deps — the God module (v2 debt) │ ├── db.py # connection pool + migration runner │ ├── auth.py # verify_token() — Lesson 3-9 │ └── schema.sql # runs, metrics, counters, api_tokens — Lesson 3-4 ├── tracelab/ │ └── client.py # thin HTTP client — all business logic in server ├── tests/ │ ├── test_race.py # reproduces the lost-update with threading.Barrier │ └── test_idempotent.py └── pyproject.toml
Use threading.Barrier(2) to synchronize two threads at the point after both have read the counter but before either writes. This forces the exact interleaving that produces the bug. Then assert that the final counter value equals initial + 2 (not initial + 1). The test will fail before the fix and pass after. This is the most reliable way to write a deterministic concurrency test — barriers, not time.sleep().
WEAKNESSWhere v2-style systems bite
PostgreSQL's default isolation level (READ COMMITTED) is the right default for most OLTP workloads — it provides reasonable concurrency without the overhead of SERIALIZABLE. But "default" does not mean "safe for all patterns." Any read-modify-write operation that does the arithmetic in application code (not in the SQL statement itself) needs a stronger guarantee or an explicit lock. The dangerous scenario: you test in a single-process environment where concurrency never occurs, deploy to a multi-worker production server, and discover the bug only when two robots run an experiment at the same second. READ COMMITTED does not bite in development; it bites in production.
TraceLab v2 uses token auth at the middleware layer — every request must carry a valid token. But "authenticated" is not "authorized." If the /api/metrics handler does not check that the token's owner also owns the run_id they're posting to, any authenticated user can write metrics to another user's run. This is a classic IDOR (Insecure Direct Object Reference) vulnerability. The fix is a single extra JOIN in the INSERT or a pre-check SELECT: SELECT id FROM runs WHERE id = :run_id AND owner = :user_id. Omitting it is the most common auth mistake in newly API-ified systems.
The idempotency_key field in TraceLab v2 is optional (it defaults to an empty string). That was a compromise for backward compatibility with the v1 CLI. In production, an optional idempotency key is nearly useless — clients that don't provide one are still vulnerable to duplicate runs on retry. The correct design is to require it: if the client does not provide a key, the server generates a UUID and returns it alongside the run_id, with instructions to use that key on any retry. Making safety features opt-in is a trap; make them the default path.
KEY INSIGHTHow it connects
- API design (3.2) — the
POST /api/runsandPOST /api/metricsendpoints follow the resource-oriented design from that lesson: noun URIs, correct HTTP verbs, 201 on creation, 409 on idempotency conflict. - Relational thinking (3.4) — the
runs,metrics, andcounterstables, their foreign keys, and the B-tree index onruns.nameare the schema designed in that lesson — v2 is the first time it runs in production. - Transactions and isolation (3.6) — the lost-update bug is the exact anomaly that lesson catalogued. The fix (SELECT FOR UPDATE, or raising to REPEATABLE READ) is the exact tool that lesson taught. This lesson is the practical exercise for that theory.
- AuthN and AuthZ (3.9) —
verify_token()is the bearer-token middleware from that lesson; this lesson is the first time it guards a real mutation, and the IDOR warning above is the authZ gap that lesson's framework predicted. - Next: Modules, Boundaries, Coupling (4.1) → — Stage 4 opens by picking up the v2 God module and decomposing it. The debt table above is the exact list of problems Lesson 4.1 will solve first.
TraceLab v2 is the Stage 3 milestone: a real client–server system, with a real database, real auth, and a real concurrency bug that required real tools to fix. The architecture is not clean — that's intentional. Clean architecture is Stage 4's topic, and you can only understand what "clean" means after you've felt the weight of the mess.
GLOSSARYKey terms
- Lost update
- A concurrency anomaly where two transactions both read a value, both compute an update based on it, and one transaction's update overwrites the other's — so one logical increment is silently lost.
- SELECT … FOR UPDATE
- A SQL clause that acquires an exclusive row-level lock at read time. Any other transaction attempting the same lock on that row blocks until the first transaction commits or rolls back.
- READ COMMITTED
- PostgreSQL's default isolation level. It prevents dirty reads (seeing another transaction's uncommitted data) but allows non-repeatable reads and lost updates on concurrent read-modify-write patterns in application code.
- Idempotency key
- A client-generated unique identifier included with a write request. If the server receives a duplicate request carrying the same key, it returns the original result rather than creating a second record — making the operation safe to retry.
- IDOR (Insecure Direct Object Reference)
- A vulnerability where an authenticated user can access or modify another user's resources by simply supplying a different resource ID, because the handler authenticates but does not authorize ownership.
- God module
- A single source file (or class) that accumulates too many responsibilities — routing, auth, business logic, data access — until no change is safe without understanding the whole thing.
- Three-tier architecture
- The standard decomposition of a web application into a client tier (UI/CLI), a service tier (HTTP server with business logic), and a data tier (database). TraceLab v2 is a textbook instance.
- Read-modify-write
- Any operation that reads a value, computes a new value from it in application code, and writes the result back — as opposed to doing the arithmetic in a single atomic SQL statement. Concurrent read-modify-write patterns are the canonical source of lost-update bugs.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Two concurrent transactions both read counter = 42, both write 43. They are both running at READ COMMITTED. Is this possible, and why?›
Yes. READ COMMITTED prevents dirty reads — transaction B cannot see transaction A's uncommitted write. But it does not prevent B from reading the same committed value that A read (42) and then independently updating the counter to 43 after A commits. The gap between A's read and A's write is the window where B's read can occur. Both transactions are internally consistent; the anomaly emerges from their interleaving. You need a stronger isolation level (REPEATABLE READ, SERIALIZABLE) or a pessimistic lock (SELECT FOR UPDATE) to prevent it.
2What does SELECT val FROM counters FOR UPDATE do that a plain SELECT does not?›
It acquires an exclusive row-level lock on the matching row at read time, not just at write time. Any other transaction that subsequently attempts a SELECT … FOR UPDATE on the same row will block until the first transaction commits or rolls back. This serializes the read-modify-write operation: only one transaction can "own" the counter row at a time, so both clients can no longer simultaneously read 42 and write 43.
3What problem does an idempotency key solve that SELECT FOR UPDATE does not?›
SELECT FOR UPDATE fixes the concurrent-writers race. But if the network drops after the server commits but before the 200 response arrives, the client has no way to know whether the write succeeded. If it retries, it creates a second run — a duplicate — even though each individual write is correctly serialized. An idempotency key stored in a UNIQUE column detects the duplicate at the database level: the second INSERT hits the constraint, the server returns the original result, and no duplicate is created. The two fixes address different failure modes: one is about concurrency, the other is about network unreliability.
4TraceLab v2 uses token auth at the middleware layer. What security check does v2 still omit that would allow an authenticated attacker to corrupt another user's runs?›
Authorization checks at the handler level. The middleware confirms "this is a valid user with a valid token," but the POST /api/metrics handler does not verify that the authenticated user owns the run_id they are posting to. An attacker with a valid token can submit metrics to any run_id by simply enumerating integers — this is an IDOR vulnerability. The fix is a single extra condition in the INSERT or a pre-flight SELECT: confirm that the run's owner column matches the token's user ID before accepting the metric.