ProjekinAja. Software Engineering Stage 6 · Judgment Lesson 6.8
● Stage 6 · Lesson 6.8

The Blank-Page Redesign and the Hero's Bar

Design TraceLab from scratch at fleet scale, pass the production-readiness review, and see the whole course as the single connected system it secretly was.

⏱ ~30 min read 🏛 Capstone 🎓 Course finale

🎯 By the end you'll understand

How a principal engineer opens a blank page and designs a system under real constraints — requirements → load math → data model → API → architecture → failure analysis → cost. You'll also carry a 40-item production-readiness checklist that replays every lesson in this course as a single connected gate, and you'll see the same design score differently under three different constraint scenarios, proving there is no context-free right answer.

KEY INSIGHTEverything before this was knowledge; this is taste

A code monkey has answers. A hero has better questions.

You have spent 55 lessons building a mental model that reaches from the CPython pipeline to canary deploys, from IEEE 754 to distributed tracing. The course never gave you recipes — it gave you mechanisms, because mechanisms are good for fifty years and recipes go stale in five. Now the recipes are yours to write.

🔑 The terminal skill

A senior engineer is not someone who knows more patterns than you. A senior engineer is someone who, given a blank page and a constraint set, can ask the right questions in the right order — and knows which answers are load-bearing. The 40-item production-readiness review that closes this lesson is not a checklist to follow slavishly; it is a map of the questions that have killed systems, organized by the lessons that taught you to ask them. Context makes each answer different. That context-dependence is the final lesson — and the one no recipe can give you.

TraceLab has been with you since Lesson 1.1: a 21-line CSV logger, deliberately scruffy, accumulating debt on purpose so you could repay it personally. The v0 script is now a v5 system that has been containerized, canary-deployed, tested for incidents, and survived a postmortem. The fleet is growing. Ten thousand robots means roughly 830 robots per second hammering the ingest endpoint at peak. Let's design that system from a blank page — the way a principal would actually do it.

HOW IT WORKSStage 1: Requirements — what are we actually building?

Every blank-page design starts here, and most engineers skip it too fast. Requirements have two kinds of failure: underspecifying (you build the wrong thing) and overspecifying (you burn runway on constraints nobody had). The checklist is: who are the users, what do they need to do, what can break, and what does "working" mean numerically?

DimensionTraceLab at 10,000-robot fleetDecision rationale
Write pathEach robot posts telemetry every 1.2 s; 50 metrics per post; 5 ms end-to-end budget before TCP ackRobots cannot block waiting for storage — any durability lag lives after ack (Lesson 4-4: queue absorbs spikes)
Read pathLab operators query last-N runs by experiment; dashboard polling at 10 s; alert evaluation at 60 sReads tolerate 5–10 s staleness (Lesson 4-3: cache tier is appropriate)
RetentionRaw telemetry: 90 days rolling. Experiment summaries: forever. Audit log: 7 years (GDPR — pseudonymous robot ID only)Three retention classes → three storage tiers (Lesson 4-4 and 6-4)
Durability SLONo data loss after ack; ≤ 0.001% row loss target over 90 daysMeans write-ahead log and replicated storage are non-negotiable
Availability SLOWrite path: 99.9% (Lesson 4-6: four-nines for fleet ingest is runway-burning for v5). Read path: 99.5%Asymmetric SLOs are the honest answer; read path can degrade gracefully
Non-goals v5Real-time streaming dashboard, ML model versioning, robot firmware OTAExplicit non-goals prevent scope creep and are the first place an interviewer or PM will probe
💡 The non-goal is a design decision

Every experienced reviewer asks "what are you explicitly not building?" before diving into the design. A non-goal list that is too short means you haven't thought about scope. A list that is too long means you're insecure about the design. Both are tells.

CORE CONCEPTStage 2: Load math — before you draw a box, do the arithmetic

This is the discipline that separates a principal from someone who draws cloud boxes. You must derive the numbers before you can reason about capacity. Use back-of-envelope; the goal is order-of-magnitude, not six significant figures.

MetricCalculationResult
Ingest rate (steady)10,000 robots × (1 post / 1.2 s)≈ 8,300 req/s
Ingest rate (peak)× 3 burst factor (herd wakeup)≈ 25,000 req/s
Write throughput8,300 req/s × 50 metrics × 8 bytes≈ 3.3 MB/s sustained
Daily ingest volume3.3 MB/s × 86,400 s≈ 285 GB/day raw
90-day raw retention285 GB × 90 days × 1.2 (overhead)≈ 31 TB
Row count (90 days)8,300 req/s × 86,400 × 90 × 50≈ 3.2 × 1012 rows
Postgres page fits3.2 × 1012 rows at 50 bytes each≈ 160 TB index + data — far past single-node OLTP (Lesson 6-1)

That last row is the most important. At 3.2 × 1012 rows, a single Postgres instance is not a capacity question — it is a category error. You need time-series partitioning (Lesson 3-5), a write-optimized ingest path, and tiered storage. The arithmetic told you before you drew a single box.

⚠ The load math trap

Engineers skip this step and design for their mental image of scale, not their actual numbers. The result is either a massively over-engineered system for a hobby project or a comically under-provisioned one for a hospital fleet. Neither is forgiven in a principal-level review.

From the load math, two architectural requirements fall out automatically: (1) the write path must be decoupled from storage via a queue (Lesson 4-4), because 25,000 req/s peak cannot block on disk writes; (2) raw telemetry lives in a column-store or time-series engine, not a row-oriented relational database.

CORE CONCEPTStage 3: Data model — schema is a contract, not a detail

At fleet scale, schema changes are the slowest operation you will ever attempt. The TraceLab v0 CSV had no schema, which felt flexible right up until the moment it wasn't (Lesson 1-5). The v5 data model must be explicit, versioned, and migration-safe from day one.

tracelab_v5_schema.sqlcore tables — Postgres + TimescaleDB hypertable
-- Experiment metadata: immutable after creation (append-only intent)
CREATE TABLE experiments (
    id         UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
    robot_id   TEXT         NOT NULL,           -- pseudonymous; no PII
    name       TEXT         NOT NULL,
    config     JSONB        NOT NULL DEFAULT '{}',
    schema_ver INT          NOT NULL DEFAULT 5,  -- v0 debt finally paid
    created_at TIMESTAMPTZ  NOT NULL DEFAULT now()
);

-- Telemetry: the hot table; partitioned by time (TimescaleDB hypertable)
CREATE TABLE telemetry (
    ts         TIMESTAMPTZ  NOT NULL,
    exp_id     UUID         NOT NULL REFERENCES experiments(id),
    metric     TEXT         NOT NULL,
    value      DOUBLE PRECISION NOT NULL,      -- IEEE 754, documented (Lesson 1-5)
    CONSTRAINT telemetry_pk PRIMARY KEY (ts, exp_id, metric)
);
SELECT create_hypertable('telemetry', 'ts', chunk_time_interval => INTERVAL '1 day');

-- Schema version table: the migration backstop (Lesson 4-8)
CREATE TABLE schema_migrations (
    version    INT PRIMARY KEY,
    applied_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    checksum   TEXT NOT NULL
);
📖 Hypertable

TimescaleDB's create_hypertable converts a Postgres table into time-partitioned chunks. Each chunk covers a time interval (here, one day). Old chunks are promoted to compressed columnar storage automatically, which cuts storage 10–20× while keeping SQL semantics unchanged. This is the honest answer to "our time-series data is getting too big for Postgres."

Three design decisions embedded in the schema: the robot_id is pseudonymous from row one (GDPR is retroactive pain if you retrofit it — Lesson 5-5); config is JSONB so the schema can evolve without migrations for per-experiment parameters; schema_ver on experiments is the explicit promise that you will run the zero-downtime migration script (Lesson 4-8) rather than a breaking drop-and-recreate.

HOW IT WORKSStage 4: Architecture — draw the boxes, justify each edge

With load math and schema in hand, the architecture diagram is derivation, not decoration. Every box earns its existence by answering a specific number from the load math or a specific requirement from Stage 1.

10k robots 8,300 req/s LB health-check ingest validates + acks < 5ms budget queue Kafka / NATS absorbs spikes TimescaleDB hot 90-day window hypertable chunks object store compressed parquet Redis cache 5-10s TTL API gateway auth + rate-limit versioned routes alert evaluator (60s) solid = write path · dashed = read path
TraceLab v5 architecture. The queue decouples ingest latency from storage throughput. The hypertable chunking handles the row-count math. The cache tier takes staleness budget the requirements gave us. Every box is justified by a number, not an aesthetic.

Three architecture decisions worth defending out loud, because a reviewer will probe them:

  1. Why a queue? Robots cannot tolerate storage latency. The ingest service acks after enqueue, not after write. This decouples the two clocks and lets the consumer process at its own pace (Lesson 4-4). The cost: at-least-once delivery; the consumer must idempotently handle duplicate messages.
  2. Why TimescaleDB and not pure Kafka or ClickHouse? TimescaleDB gives SQL semantics — the team already knows SQL from TraceLab v2. At 10,000 robots we are firmly within TimescaleDB's operational envelope. ClickHouse is a defensible alternative with better analytical throughput; choose it if you have it operationally. Kafka as primary storage is not appropriate here — it is a transport, not a database.
  3. Why not Kubernetes? Lesson 5-9 answered this: K8s is the right answer when you have multiple heterogeneous services with independent scaling envelopes and a platform team to own it. A small robotics lab almost certainly does not. ECS or a three-node Nomad cluster is the honest TraceLab answer at this scale.

WEAKNESSStage 5: Failure analysis — what kills this system?

A principal-level design review always has a failure section. Most engineers skip it or rush it. Here is TraceLab v5's honest failure inventory:

Failure modeBlast radiusMitigationLesson taught this
Queue consumer crashes mid-batchDuplicate rows in TimescaleDBIdempotency key on (ts, exp_id, metric); UPSERT not INSERT4-4
TimescaleDB disk fullWrite path backs up into queue; eventually OOM on KafkaAlerting on disk >70%; chunk compression policy; retention drop policy tested monthly6-4
Robot clock skew > 30 sHypertable chunk routing wrong; data lands in wrong partitionNTP enforcement on robot images; server-side timestamp override for >30 s skew1-4
Thundering herd: 10,000 robots reconnect simultaneouslyIngest service OOM; queue fills faster than consumer drainsJittered reconnect backoff on robot firmware (Lesson 4-6); queue consumer auto-scales within budget6-3
Schema migration under loadTable lock on experiments blocks ingest acknowledgementsZero-downtime pattern (Lesson 4-8): expand–migrate–contract; never ALTER on a live hot table4-8
Alert evaluator read-path lagStale alerts; missed SLO breachAlert query runs against TimescaleDB directly, not through cache; 60 s staleness budget defined in SLO5-8
⚠ The failure analysis no one wants to write

"What happens if your restore fails?" Most engineers list backup procedures in the failure plan but do not actually have restore test evidence. A backup you have never restored is not a backup — it is hope. The failure plan for TraceLab v5 must include a monthly scheduled restore drill with a measured RTO (Lesson 6-4). If you cannot produce the last restore test timestamp, the PRR reviewer will call it out. They should.

IN PRACTICEStage 6: Cost — the number that decides everything

A design that no one can afford is not a design — it is a wish list. Back-of-envelope cost modeling is the last stage of the blank-page review and the one most engineers delay until someone forces them to do it.

ComponentSpecMonthly cost (rough)
Ingest service (3 nodes, HA)c5.xlarge × 3 on-demand≈ $450
Queue (managed Kafka or NATS)3-node MSK or self-hosted≈ $300 – $600
TimescaleDB primary + replicar5.2xlarge × 2, 4 TB gp3 each≈ $900
Object storage (compressed parquet)31 TB at $0.023/GB≈ $715
Redis cache (2-node ElastiCache)cache.r6g.large × 2≈ $180
API gateway + LB + egressALB + 500 GB egress≈ $120
Total≈ $2,700 – $3,100 / mo

For a hospital fleet with a $200k annual infrastructure budget, this is trivially fundable. For a startup with 3 months of runway, it is a conversation stopper. For a hobby project with 20 robots, it is comically over-engineered. The same design, three different verdicts — and you are about to prove it in the widget.

SEE ITCapstone design canvas + production-readiness gate

The widget has two acts. Act 1 is a staged design canvas: you answer deterministic principal-level questions at each of the six stages above, building the design forward. Act 2 is the 40-item production-readiness review — click each item to mark pass or fail, then switch scenarios to see how the same design scores differently under three constraint contexts.

🏛 Interactive · Capstone design review + PRR

Act 1: Walk the six design stages, answer principal-level questions. Act 2: Score your design on the 40-item PRR, then switch constraints.

Act 1 — Staged design canvas
Stage 1 · Requirements
The fleet grows from 100 to 10,000 robots. What is the FIRST question a principal asks before touching the architecture?
How many requests per second does this generate?
What database should we use at this scale?
Should we migrate to Kubernetes now?
Stage 2 · Load math
Load math gives 8,300 req/s sustained and 3.2 × 1012 rows over 90 days. What does this rule out immediately?
A single Postgres instance as the telemetry store
A message queue in the write path
Redis for caching read queries
Stage 3 · Data model
The v0 CSV had no schema version field. What is the cheapest way to ensure v5 schema migrations never require downtime?
Expand-migrate-contract: add columns before removing old ones, never drop live
Run a maintenance window and ALTER TABLE during low traffic
Use a NoSQL store so schema changes are free
Stage 4 · Architecture
Robots post at 1.2 s intervals and cannot block waiting for storage. Which architectural move directly solves this?
Decouple ingest from storage via a queue; ack after enqueue, not after write
Make the database faster with more replicas
Increase the robot posting interval to 5 s
Stage 5 · Failure plan
You list "daily database backups" in your failure plan. A reviewer asks for the last restore test timestamp. You cannot produce one. What should you do?
Schedule a restore drill this sprint and add it to the on-call runbook as a monthly recurring task
Explain that the backup tooling is trusted and a restore test is not necessary
Move the failure plan section out of scope for this review
Stage 6 · Cost
The estimated monthly cost is $2,700–$3,100. A reviewer asks whether this is the right design. What is the most honest answer?
It depends on context: funded hospital fleet — yes. Startup with 3 months runway — too expensive. Hobby project — massively overbuilt.
Yes, it is the right design for a 10,000-robot fleet universally
The cost is a detail; correctness matters more than price
Act 2 — 40-item production-readiness review

Click each item to toggle pass (green) / fail (red). Then switch constraint scenarios — the same toggles, different scoring weights.

PRR score
0 / 40
Verdict
—
Context
Hospital fleet
Blocking fails
0

Pass the blocking items first — they gate the review regardless of scenario.

The most important thing the widget shows is the last act: switch from "Hospital fleet" to "Startup (3 months runway)" and watch items that were optional become blocking, while some heavy requirements downgrade to nice-to-haves. The same design. The same checklist. The context is doing all the work. That is the final lesson of the course — and it is not one you can memorize.

TRACELABTraceLab v5 milestone: the debt is paid

This is the full v0 → v5 debt ledger, closed. Every item that was identified as a future wound in Lesson 1.1 has been repaid.

Debt inherited from v0Repaid inMechanism
open() without encoding; values glued with +Lesson 1.5UTF-8 declared; csv module; floats stored as DOUBLE PRECISION
find_run O(n) full-file scanLesson 1.6–1.7Indexed foreign key on exp_id; hash index on robot_id
No history — edits are foreverStage 2Git object model; experiment table is append-only by design intent
No locking — concurrent writers interleaveStage 3Postgres row-level locking; queue consumer idempotency key
CSV "schema" in nobody's headStage 4 (4-8)Explicit schema migrations table; schema_ver column; expand-migrate-contract pattern
No observability — silent failuresStage 5 (5-7, 5-8)Distributed tracing on ingest path; SLOs on write availability and queue lag
Not scalable past one processStage 6 (this lesson)Queued ingest; TimescaleDB hypertable; horizontal ingest-service scaling

v5 milestone definition: TraceLab is designed on paper for a 10,000-robot fleet. The bottlenecks are found in order (load math first). Replication is tested via restore drill. The oldest module (CSV importer) is migrated via strangler-fig pattern (Lesson 6-5). The blank-page redesign is defended through the principal-level six-stage design review and the 40-item PRR. The debt ledger is closed.

🔑 What you actually built

You did not build a production system in this course. You built something more durable: a mental model that reaches from CPython tokenizing your source text to a 10,000-robot fleet being designed with load arithmetic. TraceLab was never the point — the living with your own decisions was. Every lesson that felt like scope creep was actually a future debt coming due. That is what software engineering feels like from the inside.

WEAKNESSWhere heroes stumble

✗ Skipping the load math and reasoning from aesthetics

The most common failure in principal-level design reviews is a diagram that is beautiful and unjustified. Every box must earn its existence by answering a number. A 10,000-robot fleet designed without back-of-envelope arithmetic is a polished guess — and experienced reviewers will find the number that breaks it in the first five minutes. Do the arithmetic first, draw the boxes second.

✗ Treating the PRR as a compliance form

The 40-item production-readiness review is a map of questions that have killed real systems. Rubber-stamping the checklist without substance is worse than not having one — it creates false confidence. The items that matter most (restore tested? failure modes enumerated? cost modeled?) are exactly the ones that get skipped under deadline pressure. An honest "no, and here's the plan to fix it before launch" is more trustworthy than a dishonest check mark.

✗ Mistaking context-independence for expertise

The scenario switching in the widget is not a trick question — it is the actual content of senior engineering judgment. A junior engineer looks for the right answer. A senior engineer asks "right for whom, at what cost, under what constraints?" The failure mode is having a rigid opinion before you have read the constraints, and defending it when the context clearly demands something different. Scaling restraint — knowing which best practice to skip — is the hardest thing this course teaches, and the one most visible in code review.

KEY INSIGHTHow this connects — the entire course, once more

There is no next lesson. But the course as a single connected system — from a tokenizer eating your source text in Stage 1 to a principal defending a fleet-scale design in Stage 6 — is now visible as the single arc it was designed to be. The TraceLab debt ledger is closed. The questions are yours. Go build something real.

GLOSSARYKey terms

Production-readiness review (PRR)
A structured gate check before a system goes live, covering observability, failure modes, durability, security, and operational runbooks. Valuable as a forcing function, dangerous as a rubber stamp.
Hypertable
TimescaleDB's time-partitioned table abstraction: data is chunked by time interval and old chunks are compressed automatically, extending Postgres to time-series scale.
Idempotency key
A unique identifier on a write operation that allows the server to detect and safely ignore duplicate retries without double-counting. Required in any at-least-once delivery system.
Thundering herd
A failure mode where a large number of clients reconnect or retry simultaneously after an outage, generating a spike that can overload the recovering service. Mitigated with jittered backoff.
Expand-migrate-contract
The zero-downtime schema migration pattern: add the new column (expand), back-fill data and update application code (migrate), remove the old column only after all traffic uses the new one (contract).
Scaling restraint
The discipline of not adding architectural complexity until the load math demands it. The hardest skill in the discipline, because adding complexity always feels like progress.
Restore drill
A scheduled, executed test of the backup restore procedure, producing a measured RTO. Without it, a backup is not a backup — it is untested hope.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What is the first question a principal asks when the fleet grows from 100 to 10,000 robots?›

How many requests per second does this generate — and what does that number rule out? The load math comes before the architecture diagram. At 8,300 req/s sustained, a single Postgres instance as the raw telemetry store is ruled out immediately by row count arithmetic, without needing to draw a single box.

2Why is the queue in the write path a requirement at 10,000 robots, not an optimization?›

Robots cannot block waiting for storage — the ingest SLA is 5 ms to TCP ack. Without a queue, ingest latency is coupled to disk write latency, and any storage hiccup propagates directly to the robot's control loop. The queue decouples the two clocks: ingest acks after enqueue, storage processes at its own pace. This is not a nice-to-have; it is how the requirements are satisfiable at all.

3Why does the same design score differently under three constraint scenarios?›

Because "right" is always relative to context: budget, failure cost, team size, and operational maturity. A $3,100/month design is trivially funded for a hospital fleet with a $200k infrastructure budget and life-safety requirements. The same design is a startup killer with 3 months of runway and a smaller user base. Context-dependence is not a cop-out; it is the actual content of senior engineering judgment.

4What does it mean for a backup to be "not a backup"?›

A backup you have never restored is untested hope. If you cannot produce a recent restore test timestamp — a measured, executed drill with a documented RTO — you do not know that your restore procedure works under real conditions. Backup tooling can silently corrupt, be misconfigured, or produce backups that require a tool version no longer available. The restore drill is the only evidence that matters.

You've reached the end of Software Engineering Mastery — claim your certificate →