ProjekinAja. Software Engineering Stage 2 · Craft Lesson 2.6
● Stage 2 · Lesson 2.6

Testing I: What Tests Are Actually For

Tests as executable specification and regression armor — and the uncomfortable proof that 100% coverage can still miss the bug.

⏱ ~20 min read 🧪 Testing theory 🎮 Interactive widget

🎯 By the end you'll understand

What tests actually purchase (cheap change, not bug-free code), the arrange-act-assert skeleton and why it matters, boundary values and equivalence classes, a clear-eyed view of TDD (when it helps, when it is actively counterproductive for research code), and why coverage is the most gamed metric in software — with mutation testing as the honesty check. You will apply all of this to TraceLab's metric-aggregation function.

KEY INSIGHTTests buy cheap change, not bug-free code

A test suite is not a proof of correctness. It is a safety net that lets you move fast without breaking things you already got right.

Here is the test that surprises almost every researcher who first writes professional software: a test suite can be green, coverage can read 100%, and the code can still silently compute the wrong answer in every real scenario. The tests passed because you tested the wrong things. The metric lied because coverage measures execution, not correctness.

🔑 The real purpose of tests

Tests are executable specifications — they say what the code is supposed to do in machine-checkable form. They are also regression armor — they make it safe to refactor, upgrade dependencies, or extend a module, because the moment something previously correct breaks, the suite screams. The value is not "proved correct once" but "confidence every time you change anything." That confidence is what lets software actually evolve.

CORE CONCEPTWhat tests buy you — and what they don't

Think about TraceLab. You have a compute_stats() function that aggregates loss values across runs. Without tests, every time you touch that function — add a new metric, fix a rounding issue, refactor the data model — you are flying blind. You run the full experiment once and squint at the numbers. That process does not scale past two contributors and does not survive six months of changes.

With a good test suite:

What tests don't buy:

Code change refactor / feature Test suite executable spec regression armor ✓ green Deploy with confidence ✗ red → regression caught early No tests → guesswork
A test suite is a gate between your change and production. Without it, the regression detection happens in production — or not at all.

HOW IT WORKSAnatomy of a test: arrange-act-assert

Every test worth reading has three parts — the three A's. Not as arbitrary ceremony, but because each section serves a different cognitive purpose:

A

Arrange — build the world

Construct all the inputs and state the function needs. Isolate from the real world: no live databases, no real network, no wall-clock time. Use real data objects but minimal ones — give the function just enough to exercise one behavior. If arrange is long and tangled, the design is telling you something.

A

Act — call the unit under test

One call. One operation. If you find yourself writing two function calls before an assert, you are probably testing two things — split the test. The act is a single line in well-designed tests.

A

Assert — state what must be true

Compare the actual outcome to the expected one. Assert one logical thing per test (not one assert statement — one concept). A failing test that says "expected 3.14, got 0.0" is a root cause; a failing test that checks five things simultaneously is a riddle.

test_stats.pypytest
from tracelab.stats import compute_stats

def test_mean_of_two_values():
    # Arrange
    losses = [0.4, 0.6]

    # Act
    result = compute_stats(losses)

    # Assert
    assert result["mean"] == pytest.approx(0.5)

def test_empty_list_raises():
    # Arrange
    losses = []

    # Act + Assert (exception testing — one idiom for expected errors)
    with pytest.raises(ValueError, match="at least one"):
        compute_stats(losses)
💡 pytest.approx is not optional

Floating-point equality (result == 0.5) fails for values computed through arithmetic because of IEEE 754 rounding. Use pytest.approx() for all float assertions — it checks within a relative tolerance of 1e-6 by default. Lesson 1.5 gave you the theory; this is the test-writing consequence.

Boundary values and equivalence classes

You cannot test every possible input. The smart shortcut is to identify equivalence classes — groups of inputs the code handles identically — and test one representative from each. Then test the boundary values at the edges between classes, because boundary bugs are the most common kind.

For compute_stats(losses), the equivalence classes are: empty list (error class), single value, two values, many values, values with duplicates, values that include zero or negative numbers. Boundary values: exactly one element (the minimum valid input), lists with repeated values that could affect variance, very large lists where floating-point accumulation matters. A suite testing all equivalence classes and boundaries is far more valuable than a suite achieving 100% line coverage on the happy path.

Input classRepresentativeWhat breaks if untested
Empty[]Division-by-zero, silent NaN
Single element[0.5]Variance formula (n-1 = 0)
Normal range[0.1, 0.5, 0.9]Happy-path logic errors
All identical[0.3, 0.3, 0.3]Zero variance path
Negative values[-1.0, 0.0, 1.0]abs/min/max assumptions
Very large listlist(range(10_000))Float accumulation drift

IN PRACTICEWhat to test — and what not to

The hardest question beginners face is scope. Testing everything is impossible; testing nothing is worthless. Here is the heuristic that holds up:

Test your logic, not your tools. Your job is to verify that your code, given correct inputs, produces correct outputs — not to verify that Python's sum() function works, or that numpy.mean() is correct. Do not write tests for the standard library. Do not write tests that check whether a third-party library behaves according to its documentation.

💡 The productive question

Before writing a test, ask: "If I broke this in the most plausible way — off-by-one, wrong sign, forgot an edge case — would this test catch it?" If yes, write it. If no, write a better test, not a passing one.

What to test:

What not to test (at the unit level):

COMPARISONTDD: where it shines, where it flounders

Test-Driven Development (TDD) — write the failing test, then the minimal code to pass it, then refactor — is one of software engineering's perennial religious debates. Both sides are partly right, because TDD's value is genuinely context-dependent.

ContextTDD verdictWhy
Well-defined business logic (parsing, pricing, validation)Excellent fitThe requirement is a spec; the test IS the spec. Red-green-refactor rhythm keeps scope tight.
API design — "what should calling this feel like?"Good fitWriting the test first forces you to think about the caller's perspective before the implementer's.
Bug fixingStrong fitThe reproduction test is the definition of "fixed." It cannot slip back silently.
Research / exploratory codePoor fitYou don't know the right output yet. Writing tests for unknown behavior is testing your guesses, not your code. Explore first; extract and test the pieces once the shape is clear.
ML model training loopsPoor fitNondeterminism and compute cost make fast-feedback TDD loops infeasible. Test the surrounding infrastructure (data loaders, metric computation, checkpoint saving) instead.
Early prototypesRisky fitIf the design will be thrown away, TDD optimizes the wrong thing. Wait until the structure stabilizes.

The honest read: TDD is a powerful discipline for software whose specification is known in advance. For research code — which is most of what this learner writes — the wiser approach is test after: explore freely, find the right structure, then lock it down with tests before anyone else touches it. The crime is not "failing to do TDD"; it is shipping important shared code with no tests at all.

SEE ITMutation-testing game

Here is where coverage gets exposed. The widget below presents TraceLab's compute_stats() function and a pre-built test suite that achieves 100% line coverage. Your job: pick which tests to include in your suite. Then run the mutation engine — it systematically corrupts the function (flipping operators, off-by-ones, deleted lines) and scores how many mutants your chosen tests kill. A mutant that survives means your suite missed a real class of bugs.

Start with the coverage-only suite (just the first test) and see why 100% can be a lie. Then add tests strategically and watch your mutation score climb.

⚙ Interactive · Mutation-testing game — kill the mutants

Select which tests to include in your suite. Run the mutation engine and see which bugs your tests catch — and which survive. Challenge: can you reach 100% mutant kill with the fewest tests?

Function under test

def compute_stats(losses): """Aggregate a list of loss values.""" if not losses: raise ValueError("need at least one") n = len(losses) mean = sum(losses) / n variance = sum( (x - mean) ** 2 for x in losses ) / (n - 1) if n > 1 else 0.0 return { "mean": mean, "min": min(losses), "max": max(losses), "var": variance, }

Choose your tests

  • T1: mean of [0.4, 0.6] = 0.5
  • T2: mean of [0.0, 1.0, 2.0] = 1.0
  • T3: empty list raises ValueError
  • T4: single element → var = 0
  • T5: min of [0.3, 0.1, 0.9] = 0.1
  • T6: max of [0.3, 0.1, 0.9] = 0.9
  • T7: variance of [1.0, 3.0] = 2.0
  • T8: negative values — mean of [-1, 0, 1] = 0
100%
Line coverage
Always 100% — even with just T1.
This is coverage's lie.
Select tests and click Run.

The opening position (only T1) achieves 100% line coverage — every line of compute_stats executes — but kills fewer than half the mutants. The empty-list path runs; the variance formula runs; the min/max calls run. They just run with wrong answers that T1 doesn't check. Adding T2 through T8 progressively closes the gaps. The mutation score rises to 100% only when your tests check what the code computes, not just which lines it reaches.

This is mutation testing in a nutshell. Real tools — mutmut for Python, PIT for Java — apply hundreds of mutations automatically and report your kill rate. They are slow (each mutant requires a full test run), so you run them in CI on a schedule, not on every commit. But they are the only metric that asks whether your tests actually check anything.

WEAKNESSThe 100% coverage lie

Coverage metrics were invented as a tool to find untested code paths — places you forgot to exercise at all. They were never designed to measure test quality. The problem is that a line being executed is not the same as a line being verified.

Consider this test:

bad_test.pyachieves 100% coverage, tests nothing
def test_runs_without_crashing():
    compute_stats([0.4, 0.6])  # no assert

This single test exercises every branch in compute_stats (assuming the happy path covers them all). Coverage: 100%. Mutants killed: 0. It proves the function does not raise an exception on this input. That's it.

⚠ The metric that gets gamed

Coverage targets ("thou shalt not merge below 80%") create a specific incentive: write tests that touch lines without making claims about correctness. This produces suites full of assertionless tests, or tests with trivially-true assertions, that look excellent in the coverage report and catch nothing in production. Coverage is a necessary but far-from-sufficient measure of suite quality.

The correct use of coverage: find gaps, not confirm quality. If coverage shows you a branch you never exercise, that is useful. If coverage is 100% and you declare yourself done, you are using it wrong. Mutation score is the better quality signal; branch coverage is the better gap-finder (it counts which control-flow paths execute, not just which lines, which catches missed if arms).

Coverage vs. mutation score — the dangerous quadrant Mutation score (quality) Line coverage Missed paths (find with coverage) Good suite (the target) Low effort (early stage, acceptable) Coverage theater high coverage, no assertions "100% covered, nothing caught" High Low Low High
The dangerous quadrant: high coverage, low mutation score. That is exactly where the "100% covered" suite with no real assertions lives.

TRACELABTraceLab v1: testing the stats module

By Stage 2, TraceLab is becoming a real package. The tracelab/stats.py module — which the widget above tested against — needs a proper test suite before it ships. Here is what that looks like with the principles from this lesson applied:

tests/test_stats.pyTraceLab v1
import pytest
from tracelab.stats import compute_stats


# ── equivalence class: empty ────────────────────────────────────────────
def test_empty_raises():
    with pytest.raises(ValueError, match="at least one"):
        compute_stats([])


# ── boundary: single element ────────────────────────────────────────────
def test_single_element():
    result = compute_stats([0.5])
    assert result["mean"] == pytest.approx(0.5)
    assert result["var"]  == pytest.approx(0.0)  # not nan — that's a bug
    assert result["min"] == result["max"] == 0.5


# ── normal range ────────────────────────────────────────────────────────
def test_mean_of_symmetric_list():
    result = compute_stats([0.0, 1.0, 2.0])
    assert result["mean"] == pytest.approx(1.0)


def test_variance_known_value():
    # Computed by hand: mean=2, deviations=[-1, 1], var=(1+1)/1=2.0
    result = compute_stats([1.0, 3.0])
    assert result["var"] == pytest.approx(2.0)


def test_min_max_correct():
    result = compute_stats([0.3, 0.1, 0.9])
    assert result["min"] == pytest.approx(0.1)
    assert result["max"] == pytest.approx(0.9)


# ── negative values ──────────────────────────────────────────────────────
def test_negative_values_mean():
    result = compute_stats([-1.0, 0.0, 1.0])
    assert result["mean"] == pytest.approx(0.0)
    assert result["min"]  == pytest.approx(-1.0)

Run with pytest tests/ -v --tb=short. The -v flag prints each test name and its pass/fail status, which makes the test suite readable as living documentation. Run pytest --cov=tracelab --cov-report=term-missing to see which lines are uncovered — and then remember that 100% coverage here is a floor, not a ceiling.

To actually run mutation testing on this suite:

terminalbash
# install mutmut (pip install mutmut)
mutmut run --paths-to-mutate=tracelab/stats.py
mutmut results  # shows surviving mutants
mutmut show 3   # shows the diff for mutant #3

A surviving mutant is a gift: it tells you exactly what your tests don't assert. Treat each one as a test requirement, not a mutmut bug.

WEAKNESSWhere testing goes wrong

✗ Testing implementation, not behavior

The most common mistake: tests that reach into private state, assert on internal variable values, or check that a specific sub-function was called with specific arguments. These tests break every time you refactor — which is precisely when you want tests to help you. Test the public contract: given these inputs, the function returns this output with these side effects. The implementation is yours to change freely.

✗ One test per function instead of one test per behavior

def test_compute_stats(): is an antipattern. A single function can have five distinct behaviors (normal case, empty, single, negative, overflow). Testing all five in one function means: a single failure hides four others (pytest stops at the first assertion failure by default), the test name tells you nothing about what failed, and the test body reads like a script, not a spec. One test per behavior, descriptively named.

✗ The 100% coverage cargo cult

Teams that mandate 100% line coverage as a merge requirement reliably produce suites that achieve 100% and catch nothing. The mandate is not wrong — incomplete coverage is a real problem — but the metric is the wrong proxy. Better gates: mutation score above a threshold (harder to fake), branch coverage rather than line coverage (harder to achieve with assertionless tests), and mandatory test review as part of code review. A reviewer asking "what does this test actually verify?" catches more than any metric.

KEY INSIGHTHow this connects

  • 2.1 · Code Is Read 10× More Than It's Written — tests are the most-read artifact in a codebase. A test suite is documentation that cannot go out of date without failing. The readability lessons there apply directly: descriptive test names, clear arrange-act-assert structure, one logical concept per test.
  • 2.5 · Types and Invariants: Making Illegal States Unrepresentable — good types and invariants reduce the number of tests you need to write. If the type system already prevents invalid inputs, you don't need to test for them. Types and tests are complements, not substitutes — types catch construction errors; tests verify computation.
  • 2.7 · Testing II: Doubles, Integration, and the Shape of a Suite — this lesson covered unit tests for pure functions, the simplest case. Next lesson expands to test doubles (fakes, stubs, mocks), integration tests, and the broader question of how to organize a suite that stays fast and honest as a project grows.

GLOSSARYKey terms introduced

Arrange-Act-Assert (AAA)
The three-part structure of a readable, focused test: build the world, call the unit, check the result. Separating these sections forces clarity about what is being tested.
Equivalence class
A group of inputs the code handles identically. Testing one representative from each class, rather than every possible input, is the efficient way to choose test cases.
Boundary value
An input at the edge between two equivalence classes — the most common site for off-by-one and edge-case bugs. Always test boundaries explicitly.
Line coverage
The fraction of source lines executed by the test suite. Necessary but not sufficient — a line executing does not mean the output was verified.
Branch coverage
The fraction of control-flow branches (each arm of every if, loop, etc.) exercised. A stronger metric than line coverage, since it catches missed else-arms that line coverage ignores.
Mutation testing
Automated testing of your tests: a tool systematically mutates the code (flips operators, changes constants, deletes lines) and checks whether your suite catches the resulting bugs. Surviving mutants are gaps in test quality.
Mutation score
The fraction of seeded mutants killed by the test suite. A better proxy for suite quality than line or branch coverage because it asks whether tests actually detect logical errors.
TDD (Test-Driven Development)
The practice of writing a failing test before writing the code to make it pass, then refactoring. Effective for well-specified logic; counterproductive for exploratory or research-stage work where the right output is not yet known.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1A colleague says "we're at 100% coverage, we're done testing." What's wrong with that reasoning?›

Coverage measures which lines execute, not whether they produce correct results. A test that calls compute_stats([0.4, 0.6]) with no assertion achieves full coverage and catches nothing. The mutation-testing game in this lesson shows exactly this: 100% line coverage with only T1, yet most mutants survive. Coverage is a gap-finder, not a quality certificate.

2Why is it a problem to test implementation details rather than public behavior?›

Implementation details change during refactoring — that's the point of refactoring. Tests tied to internal state or private functions break every time you restructure the internals, even when the external behavior is preserved. This turns the test suite from a safety net into an obstacle. Good tests assert on public contracts (inputs → outputs → exceptions), leaving the implementation free to change.

3When is TDD genuinely counterproductive for a researcher?›

TDD requires knowing the correct output before writing the code. In exploratory research — a new loss function, an untested algorithm — you don't know the right output yet. Writing tests for a behavior you're still figuring out tests your current guesses, not your code, and creates a brittle scaffold around code that will change significantly. The right discipline is: explore freely, then write tests before sharing or depending on the result.

4What is a mutant in mutation testing, and what does it mean for a mutant to "survive"?›

A mutant is a version of the code with one deliberate fault: a flipped operator (> becomes >=), an off-by-one (n - 1 becomes n), a deleted line. A surviving mutant means the test suite ran against this broken version and all tests still passed — the suite cannot distinguish the correct code from a faulty version. Each surviving mutant identifies a gap in test quality: a real class of bugs your tests don't catch.

Next: test doubles, integration tests, and the shape of a sustainable suite →