Errors Are Part of the API
The failure path is a feature you design, not an accident you apologize for — exceptions, result values, and the catch-log-and-pray antipattern.
🎯 By the end you'll understand
How to classify errors by who caused them and who can fix them; when to use exceptions vs result types; how to design what callers and logs experience when things break; where error boundaries belong in a call stack; and why swallowing exceptions is lying to your future self — illustrated concretely through TraceLab's ingest pipeline and the three weeks of silent corrupted data that follow the catch-log-ignore antipattern.
KEY INSIGHTThe error path is half the contract
Every function has two APIs: the happy path everyone documents and the failure path everyone ignores — until someone's experiment data is silently corrupted for three weeks.
Here's a question most software authors cannot answer about their own code: if ingest_run() fails, what does the caller see? What does the log say? What state has been mutated already? If those three answers are "whatever Python defaults to," "nothing coherent," and "I'm not sure" — your error path is not designed. It happened.
A function's error behavior is as much a part of its public contract as its return value. When you write code that calls your function, you're implicitly relying on both. Get it wrong and you don't get a crash — you get silent corruption, which is the worst possible outcome: a crash tells you something broke; corruption lets you believe everything is fine.
CORE CONCEPTA taxonomy that actually tells you what to do
Errors split cleanly along two axes: who caused it and who can fix it. Getting this right is the difference between a useful error message and a useless one.
Programmer errors (bugs)
Your code passed a negative index, called a method on None, or violated a precondition. The bug is in the code, not the input. Do not catch these. Let them crash loudly — a crash tells you exactly where your mental model was wrong. Swallowing a NullPointerException or an AttributeError is how you turn a five-minute fix into a three-week investigation. These are not recoverable at runtime; they need a human with an editor.
Operational errors (expected failures)
A file doesn't exist. The network timed out. The CSV row has the wrong number of columns. The user gave an invalid run name. These are expected — any code that touches the outside world will encounter them. Design the response explicitly. Return an error value, raise a typed exception, or propagate with context. The choice depends on whether the caller can recover.
Infrastructure errors (platform failures)
Out of memory. Disk full. Database connection pool exhausted. These sit between the previous two: the calling code didn't cause them, but recovery varies by context. Sometimes you retry; sometimes you degrade gracefully; sometimes you crash intentionally (fail-fast), because partial writes in a corrupted state are worse than no writes at all.
The reason this taxonomy matters practically: what layer catches an error determines what information survives. A database layer that catches OSError and re-raises a generic StorageError("write failed") has lost the filename, the errno code, and the stack trace that would have told you it was a disk-full condition. Wrapping adds context; catching-and-discarding removes it.
HOW IT WORKSThe five things you can do with an error
When an error occurs somewhere in your call stack, every layer between origin and caller has exactly five choices. Understanding which one you're making — and being explicit about it — is the whole game.
| Strategy | What it does | Use when | Never use when |
|---|---|---|---|
| Propagate | Let the exception climb the stack untouched | This layer can't meaningfully handle it; let a smarter layer try | The exception carries no context about what this layer was doing |
| Wrap | Catch, add context, re-raise a higher-level type | You can add information the caller needs ("writing run exp042"); keeps the cause chain intact via raise ... from err | Wrapping into a type that discards the original cause |
| Retry | Catch, wait, try again; propagate after N attempts | Transient failures: network hiccup, lock contention, a GCS bucket that's briefly throttling | Programmer errors or anything that won't change between attempts |
| Catch & return error value | Catch, return None / a sentinel / a Result object | The caller must handle failure as a normal code path (e.g., "run not found") | When the caller will just check if result is None and do nothing — which is the same as swallowing |
| Swallow | Catch, maybe log, return nothing; execution continues | Almost never. The only defensible case: a cleanup in a finally block where you really don't care | Anywhere that state has been mutated. Anywhere a future reader needs to know something went wrong. |
The strategy nobody puts in that table is catch-log-ignore, which looks like "Catch & return error value" but is actually Swallow with a receipt. You feel virtuous because you wrote the log line. You feel less virtuous three weeks later when experiment exp042's loss column is None in twelve rows and nobody can tell why — because the log line fired six times and you learned to tune it out.
The pattern is unmistakable once you know what to look for:
try: row = parse_csv_row(line) store(row) except Exception: logger.warning("Failed to process row") # which row? what field? why? # execution continues # the row is silently dropped # store() was maybe half-called # the caller never finds out
The log line is not documentation; it's a receipt for a crime. The caller asked you to store a row. You didn't. You didn't tell the caller. You told the log. The log is not the caller — it can't do anything about it.
When you catch an exception and raise a different one, use raise NewError("context") from original_err. This preserves the original traceback as __cause__. Without it you strip the "why" from the "what." Every Python exception handler that raises a new exception should use from.
SEE ITError propagation simulator
TraceLab's ingest_run() has a four-layer call stack. A failure originates at the bottom layer — a malformed CSV row. At each layer you can choose how to handle it. The simulator shows exactly what the caller sees, what the log records, and what state is left corrupted by each strategy combination.
Notice the two worst combinations: swallow at the parser + anything above returns None silently, and rows vanish without any signal. And catch-log-ignore at ingest_batch is the catch-log-pray pattern fully assembled — the log fires, the row is gone, and the batch "succeeds." Three weeks later your loss curves have unexplained gaps.
The pattern that survives contact with reality: raise a typed error at the origin, wrap with context at each layer that can add information, and handle at the boundary that can actually do something meaningful — usually the CLI or the job scheduler.
CORE CONCEPTResult types: making the failure path visible in the type
Exceptions are invisible at the call site. When you call ingest_row(row), nothing in the signature tells you it can fail, what it fails with, or whether the caller is expected to handle it. The failure path is a hidden contract.
Result types make failure visible. Languages like Rust, Go, and Haskell build this in. In Python you build it explicitly — and for code that will be called by other humans (or AI agents), the visibility is worth the ceremony:
from dataclasses import dataclass from typing import Generic, TypeVar T = TypeVar("T") @dataclass class Ok(Generic[T]): value: T @dataclass class Err: message: str cause: Exception | None = None Result = Ok[T] | Err def parse_row(line: str) -> Result[dict]: try: fields = dict(pair.split("=", 1) for pair in line.split(",")[1:]) if "loss" not in fields: return Err(f"missing 'loss' field in row: {line[:40]!r}") return Ok(fields) except ValueError as e: return Err(f"malformed row", cause=e) # The caller MUST pattern-match — the failure path is in the type result = parse_row(line) match result: case Ok(value=row): store(row) case Err(message=msg): raise ParseError(msg)
The honest tradeoff: result types force callers to handle failure explicitly — which is the entire point — but they add boilerplate and make Python's existing exception idioms feel awkward. Use them at the boundary between modules, or anywhere the failure path is a normal expected outcome that callers must branch on. Prefer typed exceptions everywhere else.
Fail-fast: crash immediately when an invariant is violated, before any state is mutated further. Preferable when partial writes are worse than no writes (TraceLab's DB commits, for example). Degrade-gracefully: handle the failure, skip or substitute, continue with reduced functionality. Preferable when skipping bad rows is acceptable and the batch is still useful. The choice is a product decision, not a technical one — and it belongs in a spec, not buried in a try/except with no comment.
IN PRACTICEWhere do error boundaries live?
An error boundary is the layer that catches an exception and makes a final decision: present a human-readable message, log with full context, update a status field, or escalate. Get the placement wrong in either direction and you pay:
- Too low: the boundary has no context about what the user was trying to do. It can't give a useful error message. It swallows recoverable errors that a higher layer might have handled.
- Too high: every exception becomes one generic "something went wrong." Structured error information that accumulated during propagation is discarded.
In TraceLab the right boundary is the CLI command handler — it's the only layer that knows both the user's intent and all the context the inner layers wrapped in. Below the CLI, every layer raises or wraps. Above would be the OS, which turns your unhandled exception into an exit code and a traceback to stderr — which is fine for developer runs, but not what you want a robot's watchdog process to see at 3 AM.
At the error boundary, log three things: what the user was doing (the command, the file, the run name), why it failed (the full exception chain via exc_info=True), and what state is consistent (committed rows, partial files written). Logs are not for the exception message; they're for the engineer at midnight who can't reproduce the failure.
TRACELABDesigning TraceLab's error taxonomy
TraceLab v0 has no error taxonomy at all. find_run() returns None whether the run doesn't exist or runs.csv doesn't exist — the caller has no way to know which happened. log_run() will happily continue after a partial write, leaving a truncated row in the file. Repaying these debts in Stage 2 is concrete: TraceLab v1 gets a typed exception hierarchy and every error path gets an explicit decision.
Here's the exception hierarchy we add in the Stage 2 refactor — narrow types, clear ownership:
class TraceLabError(Exception): """Base class — catch this to handle any TraceLab failure.""" class RunNotFound(TraceLabError): """Raised when a queried run id does not exist in the store.""" def __init__(self, run_id: str): super().__init__(f"Run not found: {run_id!r}") self.run_id = run_id class ParseError(TraceLabError): """Raised when a CSV row cannot be parsed; includes the offending row.""" def __init__(self, message: str, row: str = ""): super().__init__(f"Parse error: {message}") self.row = row class StoreError(TraceLabError): """Raised when a storage operation fails — wraps OS/IO causes.""" # usage in log_run(): def log_run(run_id: str, values: dict[str, str]) -> None: row = _build_row(run_id, values) # may raise ParseError try: with open("runs.csv", "a", encoding="utf-8") as f: f.write(row + "\n") # NOT atomic — concurrent writers can interleave; see Stage 3 for the rename-atomicity fix except OSError as e: raise StoreError(f"could not write run {run_id!r}") from e
Three things to notice. First, the base class TraceLabError gives callers a single catch-all without forcing them to import every subtype. Second, RunNotFound carries the run_id as a field — the CLI can say "Run 'exp042' not found" instead of "Run not found." Third, StoreError uses from e to preserve the original OSError cause, including the exact errno code that distinguishes "disk full" from "permission denied."
The v0 debt this repays: the silent return of None from find_run() when the file is missing is now a StoreError with a file path. The truncated-row risk is eliminated because the write is atomic for small payloads, and a failure raises before execution continues. Two of the six debts from the v0 ledger, cleared cleanly.
WEAKNESSWhere error design bites engineers
Every except Exception: pass in production code is a time bomb. The error has already corrupted some state — maybe a half-written file, a database transaction that didn't commit, a metric that has a None where a float should be. The exception was your only chance to know. Once you swallow it, you've destroyed evidence. The bug now manifests three weeks later as a mysterious gap in your experiment logs, and you'll spend three days suspecting your training code, your data pipeline, and your own sanity before anyone looks at that except block.
A broad except Exception at a low layer catches everything: your operational errors and your programmer bugs. Now a KeyError from a typo in your own code looks the same as a ParseError from malformed input. You've turned a five-minute bug-fix into a baffling investigation, because the exception that would have pointed directly at your mistake is now wrapped in a generic storage error. Catch the narrowest type you can name. Catch OSError, not Exception. Let unexpected exceptions propagate to the boundary where they'll get logged with a full traceback.
"An error occurred" is not an error message. "Write failed" is barely better. A good error message answers three questions without requiring a debugger: what was being attempted, what went wrong, and what the caller can do about it. "Could not write run 'exp042' to runs.csv: disk full (errno 28)" is an error message. It names the run, the file, the cause, and implicitly tells you the fix. The extra thirty characters in your format string will be worth it at 2 AM. This also applies to AI-generated code: models routinely produce raise ValueError("invalid input") — review every exception message for caller usefulness before merging.
KEY INSIGHTHow this connects
- 1.3 · The Operating System Contract — syscalls return error codes and set
errno; everyOSErrorin Python is the kernel's refusal translated into an exception. Understanding that layer makesStoreError from OSErrorfeel natural, not ceremonial. - 2.1 · Code Is Read 10× More Than It's Written — error messages and exception types are read more often than they're written; they're documentation that lives in the failure path. The same readability discipline that shapes a variable name shapes an exception class name and its string argument.
- Next: 2.5 · Types and Invariants — the logical continuation: once you've designed what to do when things go wrong, you design types and invariants that make entire classes of errors impossible to construct at all. The error taxonomy you built here is part of what strong typing enforces at the boundary.
Errors are the one part of your code that runs exactly when everything else has already failed. The more carefully you design them, the more control you retain in the moments when you most need it.
GLOSSARYKey terms
- Programmer error
- A bug in your own code — wrong precondition, null dereference, logic flaw. Should crash loudly, not be caught at runtime.
- Operational error
- An expected failure from the outside world: missing file, invalid input, network timeout. Should be handled explicitly with a designed response.
- Exception chaining
- Using
raise NewError("context") from originalto attach the original exception as__cause__, preserving the full cause trace. - Result type
- A type that is explicitly either a success value or an error value (e.g.
Ok(value)orErr(message)), making failure visible in the function signature. - Fail-fast
- Crash immediately when an invariant is violated, before state is further corrupted. Preferred when partial writes are worse than no writes.
- Degrade-gracefully
- Handle a failure by skipping, substituting, or reducing functionality while continuing. Preferred when the partial result is still useful.
- Error boundary
- The layer that makes a final decision about an exception: present a user message, log fully, update status, or escalate. Everything below raises or wraps; the boundary handles.
- Catch-log-ignore
- The antipattern of catching an exception, logging a message, and continuing execution — giving you the false comfort of a log receipt while silently corrupting state.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What is the fundamental difference between a programmer error and an operational error, and what is the correct default response to each?›
A programmer error is a bug in the code itself — wrong precondition, null dereference, logic error. The code needs to be fixed by a human with an editor; there's no meaningful runtime recovery. The correct response is to let it crash loudly so the stack trace points directly at the bug. An operational error is an expected failure from the outside world — missing file, invalid user input, network timeout. These are normal operating conditions and the correct response is to design an explicit handling path: a typed exception, a Result value, or a clear message to the caller. Catching programmer errors at runtime turns a five-minute bug-fix into a months-long investigation.
2What makes catch-log-ignore worse than simply letting an exception propagate unhandled?›
An unhandled exception propagates to a boundary where it terminates execution loudly — a crash with a traceback that tells you exactly what failed, where, and when. The system is in a known failed state. Catch-log-ignore does three things wrong: it consumes the exception so execution continues into a corrupted or undefined state, it gives no signal to the caller (who believes the operation succeeded), and the log line is passive evidence, not action. The result is that state corruption accumulates silently, often for weeks, before anyone notices something is wrong — and by then the evidence (the exception trace, the timestamp, the exact row) is gone. The log receipt is not a substitute for correct behavior.
3If ingest_row() catches an OSError and re-raises a StoreError, what does raise StoreError("msg") from e do that raise StoreError("msg") does not?›
The from e syntax sets the new exception's __cause__ to the original OSError and marks the relationship as explicit chaining. Python's traceback display then shows both exceptions — "The above exception was the direct cause of the following exception" — including the original's type, message, and stack trace. Without from e, the original exception is attached as __context__ (implicit chaining, displayed as "During handling of the above exception, another exception occurred") but can be suppressed with from None. Crucially, the original OSError carries the errno code that distinguishes disk-full from permission-denied — stripping it with a bare raise StoreError destroys diagnostic information that the engineer at midnight will desperately want.
4Where should an error boundary live in a layered application, and what should it do that intermediate layers should not?›
An error boundary lives at the layer that can make a final decision about the error — in TraceLab that's the CLI command handler or the job scheduler that invoked the batch. Intermediate layers (csv_parser, ingest_row, ingest_batch) should raise typed exceptions or wrap with context, but not make final decisions: they don't know whether the user wants to abort the batch, skip bad rows, alert a dashboard, or retry tomorrow. The boundary does: it catches the fully-wrapped exception chain, extracts what the user needs to know, logs the full context with exc_info=True, decides the fate of the operation (exit code, status update), and ensures no ambiguous partial state remains. Too low and the boundary has no context; too high and a generic handler discards all the structured information the inner layers carefully accumulated.