ProjekinAja. Software Engineering Stage 2 · Craft Lesson 2.8
● Stage 2 · Lesson 2.8

Debugging as a Science

Reproduce, hypothesize, bisect, instrument — debugging is experiment design under pressure, not staring.

⏱ ~18 min read 🔬 Method 🧪 Interactive widget

🎯 By the end you'll understand

The hypothesis-driven debugging loop — observe, hypothesize, predict, test — and the three concrete techniques that instantiate it: minimal reproduction, binary search (over commits with git bisect and over pipeline stages with probes), and strategic instrumentation. You'll run the widget, watch your observation count pile up against the log₂ optimum, and understand why guessing is worse — by a number you can calculate.

KEY INSIGHTDebugging is experiment design

A good debugger doesn't have better intuition. They make predictions and let the program refute them faster.

Something broke. Your first instinct is to stare at the code — then tweak a line — then stare again. An hour later the bug is fixed or you've "tried everything" and opened a chat window. That loop is not debugging; it's superstition with a keyboard. Debugging done well looks exactly like scientific method: you observe a phenomenon, form a falsifiable hypothesis, design the cheapest experiment that would distinguish it from the alternatives, and repeat. The difference is measurable: binary search through n possibilities requires ⌈log₂ n⌉ tests. Guessing randomly requires n/2. For n = 64 that gap is 6 vs 32 — a 5× difference in how long you sit there.

🔑 The central discipline

Before you touch code, you need two things: a reproducible failure and a hypothesis with a prediction. The reproduction tells you when you're done. The prediction tells you what to look at. Without both, every change is a guess, and you cannot tell whether the bug is gone or merely hiding.

HOW IT WORKSThe hypothesis-driven debugging loop

Good debugging has a tight four-step cycle. It's not a metaphor — it's a procedure you can follow literally, and doing so will make you faster even when intuition fails:

1

Observe — gather evidence without touching code

Read the error message. All of it, including the parts that look like noise. Read the stack trace bottom-up (where execution started) and top-down (where it blew up). Check logs, check inputs, check the environment. No hypothesis yet — this is the observation phase. Cutting it short is how you form hypotheses that contradict the evidence.

2

Hypothesize — commit to a mechanism, not a fix

Write one sentence: "I believe the bug is caused by X." X is a specific mechanism, not "something is wrong with the auth module." A specific mechanism makes a testable prediction: if X is true, then Y should be observable. If you can't write the prediction, the hypothesis is too vague.

3

Predict — what else must be true if your hypothesis holds?

This is the move that distinguishes science from guessing. Your hypothesis says: "the JWT token expires before refresh." That predicts: if I set the expiry to 24h in the test, the error should disappear; if I log the token timestamp, it should be past expiry at the crash site. Without a prediction, you're not testing a hypothesis, you're poking.

4

Test — run the cheapest experiment that distinguishes your hypothesis from alternatives

"Cheapest" means smallest code change, or the one that rules out the most alternatives. If the test supports the hypothesis, you've found the cause — now fix it and verify the fix via your reproduction. If it refutes the hypothesis, you have new evidence, which is faster than confusion.

The loop repeats. Each iteration either converges on a cause or eliminates a class of hypotheses. The key invariant: every test must be able to refute the hypothesis. An experiment that confirms the hypothesis no matter what it shows is not an experiment; it's a comfort blanket.

CORE CONCEPTThe minimal reproduction: why it is the whole game

Before binary search can work, you need a reliable failure signal: does the bug happen? Yes or no, consistently, from a known starting state. That's the minimal reproduction (or "repro"), and it is more valuable than the fix, for three reasons.

First, a repro separates symptoms from causes. Intermittent failures often become deterministic once you strip away the surrounding noise — timings, environment, network calls, database state. If you can't reproduce the bug reliably, you haven't understood it yet. Second, a repro gives you a termination condition: you're done when the repro passes. Without it, you might accidentally fix something unrelated, declare victory, and rediscover the bug six weeks later. Third, a repro is the only honest thing to share in a bug report; "it crashed" is not a bug report, it's a complaint.

💡 The shortest script that triggers the bug

Your goal is a self-contained script, ideally under 30 lines, that: (1) does not require production data or external services, (2) triggers the bug deterministically, and (3) is immediately runnable by someone who just cloned the repo. TraceLab's v0 CSV appender had a row-shearing bug under concurrent writes — the minimal reproduction was two threads appending simultaneously to a test file, nothing else. Everything else was noise that hid the signal.

Building the minimal reproduction is itself diagnostic. You often find the bug while stripping away code that turns out to be load-bearing. When removing a function makes the failure disappear, that function is in the blast radius. You just did binary search by accident — now do it on purpose.

CORE CONCEPTReading the evidence: stack traces and logs

A stack trace is the VM's call stack, printed. It is not error message boilerplate — it is a snapshot of exactly what was executing, at every level of abstraction, at the moment of failure. Most engineers read only the last frame. That is like reading only the last line of a paragraph: you get words but not meaning.

The discipline is to read bottom-up first (where execution entered — your code, your entrypoint), then top-down to understand how it arrived at the crash. The top of the trace is typically deep inside a library; the real cause is often a wrong value passed in your code three frames down. A KeyError at frame 0 inside pandas' groupby is not a pandas bug until you've verified what you handed it.

Traceback (most recent call last): File "tracelab/cli.py", line 42, in main result = compare_runs(a, b) File "tracelab/compare.py", line 18, in compare_runs return metrics[key] KeyError: 'loss' Read here first your code called compare_runs Crash site 'loss' key missing — why? What did you pass in? The bug lives in the caller, not the dict
Reading a traceback: the bottom frame is where execution entered; the top frame is where it crashed. The cause usually lives in between — specifically in the values your code passed to the code that crashed.

For logs, the same principle applies: read before you add. If logs exist, they already contain the failure — the challenge is knowing what to look for. Add timestamps if they are missing; correlate across service boundaries; look for the last normal-looking log before the cascade of errors. The cascade is the symptom; the cause is the quiet line just before it.

📖 The "last normal log" technique

Scan backward from the error until you find the last log entry that describes the system in a good state. Everything between that line and the error is the window in which the bug manifested. Narrow from there. This works in distributed systems too — look for the last successful health check, the last successful request, the last config reload. Causality is usually there.

CORE CONCEPTBinary search over history and space

Binary search is not just an algorithm for sorted arrays. It is a universal debugging strategy: divide the search space in half, test the midpoint, discard the half that cannot contain the bug, repeat. Applied to code, it appears in two distinct forms.

git bisect applies binary search over commit history. Given a known-good commit and a known-bad commit, it finds the first bad commit in ⌈log₂ n⌉ tests — where n is the number of commits in between. For 64 commits, that is 6 tests. The workflow is explicit and mechanical:

terminalgit bisect
# start the bisect session
git bisect start
git bisect bad                         # current HEAD is broken
git bisect good v2.0.0                 # last known-good tag or hash

# git checks out the midpoint commit; run your repro
python -m pytest tests/test_compare.py  # or python minimal_repro.py

# mark the result
git bisect good   # bug not present here
git bisect bad    # bug is present here

# repeat until git names the first bad commit
# the minimal repro is the only reliable test oracle here

git bisect reset                       # restore your working branch

The repro is the linchpin. git bisect can even be automated: git bisect run python minimal_repro.py will exit-code its way to the culprit without human intervention, as long as the repro exits 0 for good and non-zero for bad. That's the payoff for having a repro in the first place.

Binary search over pipeline stages is the same idea applied to a multi-stage data transformation. If data enters a 12-stage pipeline and exits wrong, you probe the midpoint (stage 6) first. Correct output there? The bug is in stages 7–12. Wrong output? Bug is in 1–6. Repeat. In the worst case, you find the faulty stage in ⌈log₂ 12⌉ = 4 probes. An exhaustive left-to-right scan could require 12. Binary search halves the expected cost at every step.

12-stage pipeline wrong output here 1 2 3 4 5 6 ✓ probe 1 7 8 9 ✗ probe 2 10 11 12 fault is in stages 7–8 one more probe isolates it · 3 probes total vs 9 sequential
Binary search over a 12-stage pipeline. Probe 6 first (green = pass), then 9 (red = fail). The fault is isolated to stages 7–8 in two tests. A left-to-right sequential scan might need 9.

SEE ITBug-hunt step-runner

The widget runs in two phases. First, binary-search 64 commits to find which one introduced the bug (exactly as git bisect works). Then, probe a 12-stage pipeline to localize the fault. Your total observation count is tracked against the log₂ optimum — the number the scientific method demands. The widget grades your strategy, not your luck.

🔬 Interactive · Bug-hunt step-runner

Phase 1: binary-search 64 commits. Phase 2: probe a 12-stage pipeline. Both phases are deterministic — the bug is pre-set; your strategy is what varies.

🐛 Symptom report

Loading...

Phase 1 — git bisect (64 commits)
Commit range
…
Bisect state
Good: commit #0
Bad: commit #63
Testing: —
Tests used: 0 / optimum ⌈log₂ 64⌉ = 6
Ready — choose a scenario and start bisecting.

Notice what the grade is measuring: not whether you found the bug (you always will, eventually), but how many observations you needed. An engineer who guesses randomly through 12 pipeline stages uses an average of 6.5 probes — over 60% more than the binary search optimum of 4. For git bisect over 64 commits, guessing averages 32 tests versus 6. The unscientific approach is not slower by a little — it is slower by a factor that compounds badly with pressure.

IN PRACTICEDebugger vs print: both are legitimate, neither is always right

There is a recurring tribal debate: "real engineers use debuggers, not print statements." This is wrong on both sides. They are different tools with different tradeoffs, and confusing the debate with snobbery slows you down.

ToolBest forWeaknesses
Interactive debugger
pdb, ipdb, VS Code debugger
Exploring unknown state; inspecting complex objects (tensors, dicts with unknown keys); conditional breakpoints; stepping through unfamiliar code Hard to use on production systems; awkward with concurrent code (stepping one thread masks others); adds latency per step; can't survive a process restart
Logging / print Production systems; long-running processes; concurrent code; distributed systems; bugs that only appear under load or after warm-up Heisenbug risk (logging changes timing enough to make race conditions disappear); log volumes can obscure signal; requires foresight about what to log
Assertion probes Pipeline debugging (place assertions at stage boundaries); encoding invariants as executable documentation; catching regressions early Can be disabled (python -O); must be removed or gated for production if expensive

The actual decision rule: use an interactive debugger when you don't know what to look for and can reproduce the bug locally. Use structured logging when the bug lives in production, in concurrent code, or across process boundaries. Use assertion probes at pipeline stage boundaries as a first-pass localization strategy before you pick either. pdb.set_trace() and print(f"[DEBUG] {value=}") both have full rights.

💡 f-string value= shorthand (Python 3.8+)

print(f"{metrics=}") prints metrics={'loss': 0.31, 'reward': 12.7} — the variable name and its repr, for free. Faster to type than print("metrics:", metrics) and less likely to be left in after the fix is merged. logging.debug(f"{state=}") does the same with proper log-level gating.

One practice worth adopting immediately: structured logging over string messages. logger.info("run logged", extra={"run_id": run_id, "loss": loss}) produces a log entry you can query and aggregate. logger.info(f"logged run {run_id} loss={loss}") produces a string you have to parse. The first is debuggable at scale; the second is folk art.

TRACELABDebugging TraceLab's first real failure

By Stage 2, TraceLab is being restructured into a real package (v1). The v0 script's debts are being repaid. One of those debts — concurrent writers interleaving — became a real failure during the v1 integration work. Here is how hypothesis-driven debugging applies to a TraceLab-shaped bug.

🧪 Symptom

Running tracelab compare run042 run043 sometimes raises KeyError: 'loss', but only when another process is actively appending. The error is not deterministic — it appears roughly 1 in 5 times on a loaded machine.

Step 1 — Observe. The traceback shows the error inside compare.py:compare_runs, accessing metrics['loss']. The metrics dict is built by parsing a CSV row. The row sometimes has only a timestamp and no columns.

Step 2 — Hypothesize. "A concurrent log_run call is writing a partial row while compare_runs is reading — the reader sees a half-written line."

Step 3 — Predict. If the hypothesis is correct: (a) adding a test that starts two threads — one writing, one reading — should reproduce the empty-row KeyError, and (b) the partial rows will appear in the file as truncated lines when we capture the file at failure time.

Step 4 — Test. The minimal reproduction:

repro_concurrent.pyminimal reproduction
import threading, pathlib, time
from tracelab import log_run, compare_runs

path = pathlib.Path("test_runs.csv")
path.unlink(missing_ok=True)

errors = []

def writer():
    for _ in range(100):
        log_run({"loss": 0.3, "reward": 12.5}, path=path)

def reader():
    for _ in range(100):
        try:
            compare_runs("run042", "run043", path=path)
        except KeyError as e:
            errors.append(e)

t1, t2 = threading.Thread(target=writer), threading.Thread(target=reader)
t1.start(); t2.start(); t1.join(); t2.join()

print(f"Errors: {len(errors)}")   # Errors: 23 — bug reproduced
assert not errors, "concurrent shearing confirmed"

The repro runs in under one second, requires no external services, and exits non-zero when the bug is present — making it a valid oracle for git bisect run. The fix (write atomically to a temp file and rename) is Stage 3 material; the diagnosis is Stage 2's job.

The TraceLab v1 milestone for this stage: the package now has a typed API (tracelab init / log / compare), proper src/ layout, locked dependencies, and one recovered "disaster" commit visible in the reflog. This lesson's contribution is the repro infrastructure — a tests/ directory with the concurrent-write reproduction as a pinned regression test, gated on the stage2 marker so it can be skipped on CI until the fix lands.

WEAKNESSWhere the scientific approach fails or is misused

✗ Guessing is the default failure mode — and it compounds

The standard failure is to skip Step 1 (observe), form a hypothesis from vague intuition, and change code before making a prediction. The problem is not that you waste one attempt; it is that each speculative change re-seeds the problem space. Now you have a new code state, possibly new behavior, and no clean way to tell whether the original bug is still present. You've traded one bug for a fog. The widget's observation count is a proxy for this: every probe beyond the log₂ optimum is an additional step you had to take because you didn't use the available information fully.

✗ Minimal reproduction theater

Some engineers write a "repro" that still requires a database, a running service, seeded data, and a specific OS user — and call it minimal. A minimal reproduction is minimal. If removing a dependency doesn't break the repro, remove it. If the bug appears in 5 lines and your repro is 500, it is not minimal; it is just the production code renamed. The cost of a non-minimal repro: you can't use it as a git bisect oracle, you can't share it in a bug report, and it won't still work in six months when the external service changes.

✗ Binary search without a reliable oracle

git bisect is only as good as your test. If the test is flaky — passes 80% of the time even when the bug is present — bisect will mark good commits as bad and vice versa, producing a wrong answer with high confidence. This is worse than not bisecting: you now "know" the first bad commit is #42 and it isn't. Verify that your repro is deterministic before you automate it. Run it five times from a clean state. If it fails once, make it deterministic before bisecting.

KEY INSIGHTHow this connects

  • 2.3 · Branching, Merging, and Unscrewing Yourself — git bisect requires a clean commit history to work cleanly. Squashed histories can skip context; a history with one "megacommit" per feature makes bisect land on a 3,000-line diff instead of a 10-line one. The git disciplines in 2.3 are the precondition for effective bisection.
  • 2.6 · Testing I: What Tests Are Actually For — the minimal reproduction is structurally a test. A good regression-test suite converts every bug fix into a test that prevents recurrence. If your repro is already in pytest format, you're one decorator away from a permanent regression guard.
  • Next: 2.9 · Dependencies, Packaging, and the Supply Chain → — debugging gets harder when you don't know which version of a library you're running. Locked dependencies and a clean packaging story make the "what actually ran?" question answerable.

Debugging competence is not about knowing every debugger flag or memorizing common error messages. It is about applying the same information-theoretic discipline that makes binary search so much better than linear scan, to the search for causes in complex systems. That discipline is transferable to every tool, every language, and every scale of system you'll work on.

GLOSSARYKey terms

Minimal reproduction (repro)
The smallest, self-contained code or configuration that reliably triggers a bug — no external services, no production data. Acts as the oracle for git bisect and the termination condition for the fix.
Hypothesis-driven debugging
The scientific method applied to software: observe → hypothesize (a specific mechanism) → predict (what else must be true) → test (the cheapest distinguishing experiment). Never skip the prediction step.
git bisect
A git command that performs binary search over commit history. Given a known-good and known-bad commit, it finds the first bad commit in ⌈log₂ n⌉ tests, where n is the number of commits between them.
Binary search (diagnostic)
Dividing a search space in half, testing the midpoint, and discarding the half that cannot contain the bug. Applicable to commit history, pipeline stages, configuration options, and input space.
Heisenbug
A bug that disappears or changes behavior when you attempt to observe it — typically because the observation (adding a log, attaching a debugger) changes timing enough to mask a race condition.
Stack trace
The VM's call stack at the moment of an exception, printed as a sequence of frames from entrypoint to crash site. The cause is usually in the caller frames, not the crash frame itself.
Structured logging
Logging that emits machine-parseable key-value pairs (JSON, logfmt) rather than free-form strings, enabling aggregation, filtering, and correlation without string parsing.
Log₂ optimum
⌈log₂ n⌉ — the minimum number of binary tests needed to locate one item among n candidates. For 64 commits: 6. For 12 pipeline stages: 4. The benchmark against which observation counts are graded.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What is the prediction step in hypothesis-driven debugging, and why does skipping it make you slower?›

The prediction step requires you to state what else must be true if your hypothesis is correct — something observable and testable beyond the symptom itself. Skipping it means your "test" can confirm the hypothesis no matter what it shows, so you learn nothing from a failure. Without a prediction, every change is a guess, and you cannot distinguish "fixed the bug" from "accidentally masked it."

2How many tests does git bisect need to find the first bad commit in a 512-commit range?›

⌈log₂ 512⌉ = 9 tests. Binary search halves the search space each time: 512 → 256 → 128 → 64 → 32 → 16 → 8 → 4 → 2 → identified. A random-guess strategy would average 256 tests for the same range — 28× more. The repro must be deterministic for this guarantee to hold.

3When is a print statement the right debugging tool over an interactive debugger?›

When the bug lives in a production environment you can't attach a debugger to; when the bug involves concurrency (stepping one thread masks races); when the bug only appears under load or after long warm-up; or when the bug is in a remote or containerized service. An interactive debugger requires the ability to reproduce locally and pause execution — when those conditions don't hold, structured logging is the honest tool.

4What makes a minimal reproduction truly minimal, and what does a non-minimal repro cost you?›

Truly minimal means: runnable with no external services or seeded production data, under 30 lines if possible, deterministically reproduces the bug, and any removed dependency doesn't make the bug disappear (if removing it does, you found a clue). A non-minimal repro can't be used as a git bisect oracle, can't be shared cleanly in a bug report, drifts as external services change, and is harder to maintain as a regression test. Minimizing the repro is itself diagnostic — you often find the bug while doing it.

Next: dependencies, packaging, and knowing what you're actually running →