Testing II: Doubles, Integration, and the Shape of a Suite
Mocks, stubs, fakes, integration tests, and property-based testing — and the mock-rot spiral where the suite passes while production burns.
🎯 By the end you'll understand
The full test-double taxonomy and when isolation genuinely helps vs. when it just programs the test to parrot the implementation. How to write integration tests against real (temporary) databases and external boundaries. What contract thinking buys you at service seams. How property-based testing finds edge cases no example-based test suite would think to write. And finally: why "test pyramid" is a useful heuristic but a dogma trap — the right suite shape follows system shape, not a picture someone drew in 2012.
KEY INSIGHTA passing suite is not a safe system
A suite that mocks every dependency tests the mocks. The mocks were written by the same person who wrote the code. They agreed by construction.
Lesson 2.6 established what tests are actually for: they model your beliefs about behavior, so you can update them cheaply instead of discovering them expensively. This lesson confronts the sharp edge: the tooling designed to make tests faster and more isolated has a well-worn failure mode where tests become a closed loop — a feedback system that tells you nothing about reality because reality was substituted out.
A developer writes a class, writes a test, mocks the database call, tests pass. The database schema changes. Nothing updates the mock. The test continues to pass. Six months later production fails on a query that has been broken since March. This is mock rot: the slow drift between a mocked dependency and the thing it was meant to model. Every mock is a bet that the mock's behavior matches the dependency's behavior — and that bet has to be maintained.
DEFINITIONThe test-double taxonomy: stubs, mocks, fakes, spies
Gerard Meszaros' 2007 book xUnit Test Patterns gave us the vocabulary. The terms are routinely conflated in conversation — "just mock it out" — but the differences matter because they carry different failure modes.
Dummy
An object passed around but never actually used. Satisfies a type signature. The simplest possible substitute — no behavior whatsoever. Use it when a function demands a collaborator but won't call it in the path under test.
Stub
Returns canned answers to calls. user_repo.get_user("abc") → {"id":"abc","role":"admin"}. No verification of what was called or how many times. Risk: the real repo returns something different; the stub silently locked in a lie on day one.
Fake
A working, lightweight re-implementation. An in-memory database, a local SMTP server that captures mail, a fake file system. Fakes have real behavior — they exercise code paths the stub can't. They also have to be maintained as the real thing's contract evolves. The maintenance cost is the price of the correctness.
Spy
Records calls so you can assert on them afterward. "The email service was called exactly once with the right address." Spies verify behavior, not return values. They're also the most brittle: asserting that a specific internal call happened ties the test to implementation — refactor the internals and the test breaks even when behavior is preserved.
Mock
A spy with pre-programmed expectations set up front. "Expect send_email called once, raise AssertionError if not." In Python, unittest.mock.MagicMock is the standard tool. The term "mock" colloquially covers all five — which is why this taxonomy exists.
Doubles are valuable when the real dependency is: non-deterministic (network, clock, random), slow (real database, third-party API), hard to put into the required state (trigger an error from a payment processor), or doesn't exist yet (working to an interface ahead of implementation). These are good reasons. "I don't want to set up the database" is not.
Consider TraceLab's CSV logger. A stub that returns a fixed parse result tests your downstream logic — useful. But it cannot tell you whether the real CSV round-trips cleanly with Unicode filenames, whether concurrent writes interleave, or whether the schema you're assuming is still what the file actually contains. Those questions require the real thing, or a fake that faithfully replicates it.
from unittest.mock import MagicMock, patch import pytest # ── Stub: canned return, no verification ────────────────────────────────── def test_compare_uses_loaded_run(tmp_path): # Stub the IO layer; exercise the comparison logic in isolation. repo = MagicMock() repo.load_run.return_value = {"loss": 0.31, "reward": 12.7} result = compare_runs(repo, "exp042", "exp043") assert result["winner"] == "exp042" # pure logic, no file IO # ── Spy: verify a side-effect was triggered ─────────────────────────────── def test_log_run_flushes_file(): with patch("tracelab.core.open", create=True) as mock_open: handle = MagicMock() mock_open.return_value.__enter__.return_value = handle log_run({"loss": 0.31}) handle.write.assert_called_once() # spy: was write invoked? # ── Fake: real behavior, lightweight implementation ─────────────────────── class InMemoryRunRepo: def __init__(self): self.runs = {} def save(self, name, data): self.runs[name] = data def load_run(self, name): return self.runs.get(name) def test_compare_with_fake_repo(): repo = InMemoryRunRepo() repo.save("exp042", {"loss": 0.31}) repo.save("exp043", {"loss": 0.45}) result = compare_runs(repo, "exp042", "exp043") assert result["winner"] == "exp042"
HOW IT WORKSIntegration tests: the real thing in a controlled environment
An integration test crosses a real boundary — a real database, a real file system, a real subprocess. The goal is to test the seam itself, not just the code on either side of it. This is where stub-based suites routinely lie: the stub says the database returns rows in a certain format, but the actual query has a typo, or the column was renamed in a migration, or the index was dropped and now the query times out.
The standard Python pattern uses tmp_path (pytest's built-in temporary directory fixture) for file-based integration tests, and a disposable database for heavier storage. SQLite makes local integration tests almost free:
import csv, pathlib import pytest from tracelab.store import CSVRunStore def test_roundtrip_unicode_runname(tmp_path): # Use a real temp file — no stubs. store = CSVRunStore(tmp_path / "runs.csv") store.log("exp-café-01", {"loss": 0.31, "reward": 12.7}) row = store.load("exp-café-01") assert row["loss"] == pytest.approx(0.31) # float round-trip matters assert row["reward"] == pytest.approx(12.7) def test_schema_columns_match_spec(tmp_path): # Contract test: the file format must match the documented schema. store = CSVRunStore(tmp_path / "runs.csv") store.log("r1", {"loss": 0.5}) reader = csv.DictReader((tmp_path / "runs.csv").open(encoding="utf-8")) assert set(reader.fieldnames) >= {"timestamp", "run_id", "metrics"}
The key discipline: integration tests are scoped. They test one boundary at a time, not the whole stack. test_roundtrip_unicode_runname only exercises the store layer — it says nothing about the CLI parser or the comparison logic. Keeping integration tests narrowly scoped lets them fail precisely, pointing at the seam that broke rather than forcing you to bisect a twelve-step stack.
Every boundary has a contract: a set of inputs the producer commits to handling, outputs it commits to producing, and errors it commits to raising. Contract testing makes that agreement explicit and executable. Consumer-driven contract testing (as popularized by the Pact tool) turns the consumer's expectations into a test fixture the producer runs — so the producer cannot accidentally break a consumer without a test catching it. This pattern becomes critical once TraceLab becomes a multi-service system.
CORE CONCEPTProperty-based testing: let the computer find the edge cases
Example-based testing says "given this specific input, expect this specific output." Property-based testing inverts the question: "for any input in this domain, what must always be true?" You specify the property; the framework generates hundreds of random inputs trying to falsify it. When it finds a counterexample, it shrinks it to the minimal failing case.
Hypothesis is the Python library. It is not a toy — Netflix, Dropbox, and the CPython core developers use it in production test suites. The core primitive is @given, which drives a test with generated data:
from hypothesis import given, settings from hypothesis.strategies import floats, text, dictionaries from tracelab.metrics import serialize_metrics, parse_metrics # Property: round-trip — parse(serialize(x)) == x for all valid metric dicts @given(dictionaries( keys=text(min_size=1, max_size=20), values=floats(allow_nan=False, allow_infinity=False), min_size=1, max_size=8 )) def test_metric_roundtrip(metrics): serialized = serialize_metrics(metrics) recovered = parse_metrics(serialized) for k, v in metrics.items(): assert recovered[k] == pytest.approx(v, rel=1e-9) # Property: parsing a malformed string never crashes — it raises ParseError @given(text()) def test_parse_never_raises_unexpectedly(s): try: parse_metrics(s) except ParseError: pass # expected — a well-defined exception is fine except Exception as e: raise AssertionError(f"unexpected exception: {e}") from e
The second property is the one experienced engineers routinely forget. A parser that raises KeyError, IndexError, or an undocumented internal exception for malformed input is broken — the error contract is part of the interface. Property-based testing with text() will find the input that triggers it in seconds, including inputs no human would think to try: null bytes, right-to-left overrides, strings that are valid Python float literals but not your format's float literals.
When Hypothesis finds a counterexample it saves it in a .hypothesis/ directory and replays it on every subsequent run before generating new examples. This means once you've found a bug, it stays caught — even if the generator would statistically almost never produce that input again. This is the database feature, and it changes how you think about intermittent failures.
SEE ITTest portfolio allocator
You have 100 test-budget points. Distribute them across unit, integration, and end-to-end tests for your system. Toggle which layers are mocked vs. real. Watch how escaped-bug rate, suite runtime, and flake rate respond — then hit "change the real implementation" and see which allocations keep lying green.
The key observation: when all boundaries are mocked, "change the real implementation" exposes that your unit tests are a closed system. They test the mock, not the code. Only when integration or e2e tests run against the real boundary do implementation changes surface as failures — which is the only thing that matters in production.
COMPARISONPyramid, trophy, honeycomb — shape follows system shape
The test pyramid (Mike Cohn, 2009) says: many units, some integration, few E2E. It is a good first approximation for CRUD applications where the business logic lives in units. But it was never meant to be a universal law, and treating it as one causes real harm.
| Shape | Canonical distribution | Best fit | Failure if misapplied |
|---|---|---|---|
| Pyramid | many unit · some integration · few E2E | CRUD apps, business logic concentrated in the domain layer, stable external dependencies | Applied to a data pipeline where the value is in the transformation — all unit tests, zero data shape coverage |
| Trophy (Kent C. Dodds) | some unit · many integration · few E2E | Frontend, user-facing flows where integration with the DOM/runtime is the point | Applied to a library with rich algorithmic invariants — slow integration tests miss property violations that a 10-line Hypothesis test would catch |
| Honeycomb (Spotify) | many service-level integration · minimal unit · minimal E2E | Microservice meshes where correctness lives at the boundary, not inside any single service | Applied to a monolith — tests require running N services locally, defeating the speed advantage |
| Ice cream cone (anti-pattern) | many E2E · some manual · few unit | Legacy systems with no internal test infrastructure | Slow, flaky, expensive, provides no isolation on failure — the common end-state of projects that let the suite grow without a shape strategy |
The actual principle behind all of these is: test close to the assertion you care about. If you care that the business rule fires, test the rule. If you care that the database query returns the right rows, test against the database. If you care that the robot doesn't run into a wall, test the controller with real sensor data. The shape emerges from where the value lives, not from a picture.
Unit tests are fast because they use fakes. Integration tests are slower because they hit real I/O. E2E tests are slowest because they exercise the whole stack. This tradeoff is real — but it doesn't mean unit tests are better. A unit suite that runs in 0.3 seconds and catches 40% of production bugs is worse than an integration suite that runs in 30 seconds and catches 80%. The only metric that matters is production bugs per engineering hour — optimize for that, not for suite runtime in isolation.
TRACELABTesting the metric parser with property-based tests
By Stage 2, TraceLab v1 has a typed, structured metric layer. The v0 CSV was a raw string concatenation; v1 has a proper serializer and parser that needs to round-trip correctly across all inputs it will encounter in a robot telemetry stream.
Here is the actual test structure we add in this milestone:
from __future__ import annotations import json, re from typing import Any class ParseError(ValueError): pass def serialize_metrics(metrics: dict[str, float]) -> str: """Stable JSON serialization: sorted keys, no trailing whitespace.""" return json.dumps(metrics, sort_keys=True, separators=(",", ":")) def parse_metrics(raw: str) -> dict[str, float]: """Parse a metrics string. Raises ParseError on any invalid input.""" try: data = json.loads(raw) except json.JSONDecodeError as e: raise ParseError(f"not valid JSON: {e}") from e if not isinstance(data, dict): raise ParseError("expected a JSON object, got " + type(data).__name__) for k, v in data.items(): if not isinstance(v, (int, float)): raise ParseError(f"metric values must be numeric, got {k!r}: {type(v).__name__}") return {k: float(v) for k, v in data.items()}
The property suite verifies two things that an example-based test suite would leave unchecked: that every valid float dict round-trips cleanly (not just the ones we thought of), and that every arbitrary string produces either a valid result or a well-typed ParseError — never an internal crash, never a KeyError, never a silent data corruption.
This debt was originally logged in the TraceLab v0 table in Lesson 1.1: "metrics stored as strings; float round-trips." The Hypothesis suite is the formal repayment — the parser now has a machine-verified contract rather than a handful of examples that happened to occur to the test author.
TraceLab v1's test layout looks like this: unit tests for pure logic (metric comparison, run naming rules); property tests for the serialization boundary (Hypothesis, two properties); integration tests for the CSV store against a real tmp_path; and a small set of CLI smoke tests that run tracelab log and tracelab compare as subprocesses against a temp directory. No mocks of the store layer — because the store layer's correctness is what we most need to verify.
WEAKNESSWhere this bites
The worst form of mock rot: a test that creates a MagicMock, configures it to return a specific value, calls the code-under-test, and then asserts that the mock returned what it was configured to return. This test passes by construction. It has zero information content about the behavior of the system. It will still pass when the real dependency is removed entirely. When you find yourself asserting on a mock's return value rather than the code's output, stop and ask what behavior you are actually trying to verify.
An "integration test" that boots a full application stack, starts a real server, hits it with HTTP requests, and checks the response is an E2E test wearing integration clothing. It will be slow, it will be flaky (port conflicts, startup race conditions, external service availability), and when it fails it will not tell you which layer failed. Keep integration tests scoped to one boundary. If you need to test the full stack, label it as E2E deliberately and accept the tradeoff.
A property test that generates inputs so tightly constrained that it only exercises the happy path is a slow unit test. If your @given strategy filters out 99% of candidates with .filter(), Hypothesis will warn you and eventually give up. The power of property-based testing is in the breadth of the input space — if you constrain it down to a handful of valid values, you are paying the framework overhead without getting the coverage benefit. Design your data structures to be easy to generate, or use hypothesis.strategies.builds() to generate structured objects directly.
KEY INSIGHTHow it connects
- Testing I (2.6) — established what tests are actually for: confidence that behavior is what you believe. This lesson builds the practical toolkit: doubles, integration discipline, properties, and suite shape.
- Debugging as a Science (2.8) — when the test suite tells you something is wrong, the next skill is finding out what and why. The precision of your tests determines the precision of the diagnostic signal — a passing mock-heavy suite gives you no signal at all.
The through-line: test doubles are a tool for controlling what you test, not for avoiding what is hard to test. Used well, they let you isolate and verify behavior precisely. Used badly, they turn a test suite into a mirror that reflects your assumptions back without questioning them. Integration tests and property tests are the mechanisms that keep the mirror honest.
GLOSSARYKey terms introduced
- Test double
- The umbrella term for any object standing in for a real dependency in a test: dummy, stub, fake, spy, or mock.
- Stub
- A double that returns canned values; no verification of how it was called. Risk: the canned value diverges from what the real dependency returns.
- Fake
- A lightweight but working reimplementation of a dependency — an in-memory database, a local SMTP server. Has real behavior; requires maintenance as the real dependency evolves.
- Mock rot
- The drift between a mock's configured behavior and the real dependency's actual behavior, accumulating silently until production fails on something the test suite has been greenlighting for months.
- Integration test
- A test that crosses a real boundary — real file IO, real database, real subprocess — scoped to one seam at a time.
- Property-based testing
- A style where you specify invariants ("for any X, P(X) must hold") and a framework generates hundreds of inputs trying to falsify them. Hypothesis is the Python standard.
- Contract test
- A test that verifies the interface agreement between a producer and a consumer — what the producer commits to returning, and what the consumer commits to accepting. Prevents silent breaking changes at service seams.
- Test pyramid / trophy / honeycomb
- Suite shape heuristics describing the mix of unit, integration, and E2E tests. Each fits certain system shapes well and others poorly — none is universal.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What is the difference between a stub and a fake, and when should you prefer each?›
A stub returns a canned answer — it has no behavior of its own. A fake is a working, lightweight reimplementation that has real logic (an in-memory database, a local queue). Prefer a stub when you only need to control a return value and don't care about the dependency's internal behavior. Prefer a fake when the dependency's behavior — ordering, uniqueness, error conditions — is part of what you're testing, or when multiple tests need to share state through the fake as a real system would.
2What is mock rot, and what is the structural reason it is hard to detect?›
Mock rot is the drift between a mock's configured behavior and what the real dependency actually does. It is hard to detect because the test passes — by construction, the mock returns what you configured it to return, so the assertion holds. The real dependency has changed (schema, API, semantics), but nothing in the test loop is connected to the real dependency, so the feedback is severed. It only surfaces when production code runs against the real system.
3Describe one property you could test for TraceLab's metric parser that an example-based test would almost certainly not cover.›
The "no-crash" property: for any arbitrary string input, the parser must raise either a valid result or a well-typed ParseError — never an unhandled exception. Example-based tests use inputs the author thought of (valid JSON, invalid JSON, empty string). Hypothesis with text() will generate null bytes, multi-byte Unicode edge cases, JSON-looking strings that are not quite JSON, and extremely long strings that might trigger buffer behavior. Human test authors do not think of these systematically.
4A colleague says "we should follow the test pyramid." What is the right question to ask before accepting that advice?›
"Where does the value of this system actually live?" The test pyramid makes sense when business logic is concentrated in a domain layer that can be tested in pure isolation. For a data pipeline, the value is in transformations across real data shapes — integration tests dominate. For a robot controller, the value is at the hardware boundary — property tests on the controller with realistic sensor inputs matter most. The pyramid is a heuristic for a specific system shape, not a law. The correct shape follows system shape.