Engineering in the Age of AI
Agents collapse typing and inflate verification — the engineer who can specify, verify, and operate is the one the tools amplify.
🎯 By the end you'll understand
What AI coding agents concretely change (boilerplate, exploration, recall) and what remains stubbornly the engineer's job (ownership, design judgment, verification). Why throughput rising with delegation is real but treacherous — defect-escape probability compounds quietly across features. How to delegate spec-first so the tests become the contract you keep. Where agents quietly fail: context limits, plausible-but-wrong generations, and the security surface they expand. And why "engineer of record" is not a title — it's the non-delegable role every one of these tools assumes a human is filling.
KEY INSIGHTThroughput is cheap; verification isn't
An agent can fill a diff in twelve seconds. The question is who is responsible for that diff being correct, safe, and operable — and that question has only one answer.
Every stage of this course was built around a TraceLab debt. You repaid debts about encoding, algorithms, concurrency, schema, reliability, and scale — each one a consequence of shipping code you didn't fully understand, or understand too late. AI coding agents are the most powerful amplifier this discipline has seen. They are also a new and faster way to accumulate exactly those debts, compounding silently, across every feature delegation you accept without verification.
Agents compress generation time to near-zero. They have not compressed verification time. The shift in your work is therefore not "do less" — it is "generate more, specify better, and review at a different altitude." The engineer who adapts to that shift gets genuine leverage. The engineer who mistakes throughput for quality has simply moved debt accumulation off-screen.
CORE CONCEPTWhat agents actually change
The hype conflates several different improvements. It helps to separate them, because they have different risk profiles and different implications for how you work.
Boilerplate generation
The most reliable category. Scaffolding, serialization code, CRUD endpoints, test fixtures, migration boilerplate — tasks with a clear, verifiable shape and bounded scope. Agent-generated boilerplate is usually correct and the mistakes are usually obvious. High delegation is appropriate. This is where the throughput gain is most real.
Exploration and recall
An agent can synthesize library APIs, suggest algorithm options, and stub a prototype faster than a search-and-read cycle. Treat this output as a first draft for your thinking, not a conclusion. The agent has no stake in whether the approach is right for your constraints — it will suggest a Redis cache for TraceLab's log buffer as confidently as it would for a 10,000-node cluster, because both are "reasonable patterns."
Incremental implementation from specs
Given a tight specification — a function signature, a docstring with preconditions and postconditions, a passing test suite — agents produce correct implementations more often than not. The specification is the work. When engineers complain that an agent "misunderstood," the real finding is usually that the engineer had not articulated the requirement precisely enough for a human junior either. Agents surface underspecification fast.
Diff synthesis and refactoring
Renaming, reshuffling, adapting a pattern across a codebase — agents handle these well when the transformation is mechanical and the scope is bounded. The risk is scope creep inside the diff: agents will helpfully fix adjacent things they notice, silently changing behavior you weren't asking to change.
Ask: if this task took a capable junior one hour, would you review the output in 10 minutes without running it? If yes, agent delegation is probably safe with a quick review. If you would spend an hour reviewing, the agent hasn't saved you anything — you've traded writing time for reading time, which is arguably worse because reading plausible code is harder than writing explicit code.
CORE CONCEPTWhat agents do not change
Every capability the agent lacks falls into the engineer's permanent job description. None of these are going away at current capability levels, and several are structurally non-delegable regardless of future capability.
| Non-delegable function | Why agents can't own it | TraceLab example |
|---|---|---|
| Requirements judgment | No stake in correctness, no knowledge of downstream cost, no context outside the diff window | Deciding that the metrics retention policy should be 90 days, not 30 — because the robotics team reviews quarterly |
| Design tradeoff ownership | Will generate a locally coherent design that violates system-level constraints it was never shown | Choosing a single-writer architecture for the log buffer because the fleet's write pattern is sequential, not random |
| Verification sign-off | Cannot run the system against production load, cannot observe emergent behavior, cannot evaluate its own output against intent | Confirming that the CSV escaping change handles all 14 robot-name character classes that actually appear in the fleet |
| Operational accountability | Is not paged at 3 am; does not feel the consequence of a bad decision | Being on call when TraceLab's retention job runs the wrong date range and deletes six months of logged data |
| Security review | Will generate code that follows the pattern it learned, including the vulnerability that was in the pattern | Auditing that the new REST export endpoint respects the per-experiment access control — the agent doesn't know that ACL exists |
The term engineer of record is the frame that clarifies all of these. It borrows from civil engineering: the licensed engineer who stamps the design is personally responsible for whether the bridge stands. AI tools exist inside a field that has no such licensing — but the accountability structure is the same. When the system fails, the question "who was responsible for this decision?" points at a human. That human is the engineer of record, regardless of who (or what) wrote the code.
IN PRACTICESpec-first delegation: tests as the contract you keep
The workflow that makes agent delegation both productive and defensible is to specify before you generate. You may have learned this in lesson 2-10 as a craft discipline; here it is a correctness argument:
# 1. Write the spec as tests, not prose # The tests define the contract. If you can't write the test, the spec is fuzzy. def test_escape_commas_in_robot_name(): row = format_csv_row({"robot": "Café,3", "loss": 0.31}) assert '"Café,3"' in row # RFC 4180 quoting assert row.count(",") == 1 # one separator, not two def test_float_round_trip_precision(): row = format_csv_row({"loss": 0.31415926535}) v = float(row.split(",")[-1]) assert abs(v - 0.31415926535) < 1e-9 # 2. Hand the tests to the agent: "implement format_csv_row so these pass." # The implementation is the agent's problem. Passing the tests is the bar. # 3. Review the diff for: scope (only format_csv_row changed?), legibility, # and any new import whose semantics you don't know. # 4. Add the tests to CI. The contract now exists in the repo, not in your memory.
This pattern does two things at once: it forces you to articulate what you actually want (the work that used to show up as revision cycles), and it gives the agent an objective success criterion that does not require it to guess your intent. The agent's output is better, your review is cheaper, and the test is a permanent record of the requirement.
With hand-written code, you review for correctness line by line. With agent-generated code, the correctness argument is "this passes the test suite you specified." Your review shifts to completeness: are the tests actually the full contract? Did the agent change anything outside the scope? Are there edge cases the tests don't cover? This is a different, higher-altitude review — harder in some ways, easier in others.
SEE ITDelegation-dial simulator
The following simulator models a TraceLab feature delivery pipeline with three verification gates: code review, test coverage, and operational readiness checks. You control how much of the feature implementation is AI-generated vs. human-written, and independently how thoroughly each gate is applied. Throughput (features per week) rises with delegation. Defect-escape probability compounds across the gates you loosen — and that compounding is the part that sneaks up on teams.
Three things to notice as you play. First, throughput rises almost linearly with delegation — that part is real. Second, defect-escape probability rises faster than linearly when gates are off, because the compounding is multiplicative: if each gate misses a defect 20% of the time, a defect slips past all three gates only 0.2³ ≈ 0.8% of the time, past two gates 4% and past one gate 20%, so every gate you switch off multiplies the escape rate by five. Third, the highest leverage score — throughput per unit of escape risk — is not at 100% delegation. It's at the configuration where you delegate hard and keep the gates. That's the point: gates are cheap relative to defects; the throughput cost of review is not why teams skip it — it's that the cost is certain and visible while the defect is probabilistic and deferred.
WEAKNESSWhere agents quietly fail
The failures that should worry you are not the obvious ones. An agent that generates a syntax error is embarrassing and catches itself. The failures below are plausible and invisible.
An agent generating TraceLab's new multi-robot aggregation endpoint has a context window. If that window does not include the access-control middleware (fifty files away), the agent will generate an endpoint with no ACL check — and the code will be correct against the tests you wrote, which also did not test for ACL. This is not an agent failure. The agent did exactly what was asked. The failure is a specification that was scoped to the happy path because the engineer forgot about the cross-cutting concern. Context limits make this failure mode structural: as systems grow, the fraction of the system that fits in any single context shrinks.
An agent asked to implement efficient lookup over TraceLab's run index may produce a binary search — over an unsorted list. It will look exactly like a binary search (correct variable names, correct loop structure, correct return semantics) and will return wrong results on the 40% of cases where the input happens to be unsorted. The agent is not guessing; it is pattern-matching to the nearest correct thing it has seen. The tell is "confident code that doesn't test its own preconditions." Senior engineers notice when an implementation silently depends on an invariant that isn't enforced upstream. Junior engineers and reviewers under deadline pressure do not. Agent-generated code fails this class of check reliably because agents don't reason about invariants — they match patterns.
Every line an agent generates that you don't read is an unreviewed line. AI-assisted workflows produce substantially more code per engineer per day than unassisted workflows — controlled studies (e.g., Peng et al. 2023, NBER) measured ~55% faster task completion in lab settings; anecdotal production reports range much higher, though figures like "3–5×" depend heavily on task type and remain contested. If review throughput does not scale with generation throughput — and it usually doesn't — the fraction of code that receives genuine adversarial review shrinks. The threat model is not that agents generate insecure code more often than humans (the evidence is mixed). The threat model is that the total attack surface grows faster than your capacity to audit it. On TraceLab, this is how a path traversal bug ends up in the log export endpoint: the feature was legitimate, the agent's implementation was almost right, and the reviewer didn't check the path sanitization because the feature was boilerplate and the diff was large.
TRACELABTraceLab in the AI era
TraceLab v5 now runs a 10,000-robot fleet. The codebase is large enough that no single engineer holds it all in memory — which means no agent does either, and the specification discipline is load-bearing, not optional.
Here is the delegation policy the team has settled on, represented as a short decision table. It is not a rulebook — it is a record of which categories have burned them and which have not:
| Task category | Delegation level | Gate requirement | Rationale |
|---|---|---|---|
| Schema migrations | None | Human-authored + PR review | Irreversible; one wrong migration can destroy six months of fleet data |
| Access control changes | None | Human-authored + security review | Context window can't see all downstream callers; ACL bugs are invisible until exploited |
| Retry and backoff logic | Low | Spec + tests + load review | Agents write retry loops that storm under real failure rates (lesson 6-3) |
| Serialization / encoding | Medium | Spec + tests with edge cases | Agents know RFC 4180; tests catch the Café-3 case if you write them |
| Metric aggregation logic | Medium | Spec + tests + output audit | Plausible-but-wrong statistical code is common; human verification of sample output is fast |
| CRUD endpoints, boilerplate | High | Test pass + scope check | Verifiable shape, low blast radius; most gains here |
| Documentation and comments | High | Human read | Incorrect docs are bad; they are also easy to spot and fix |
The TraceLab debt ledger from v0 is also a useful frame here: every original debt was created by exactly the kind of "this is good enough for now" judgment that agents make by default, because they optimize for plausibility in the local context. The engineer who built v5 knows those debts — because they paid them. The agent does not know them. The engineer's job is to carry that institutional memory into every delegation decision.
# Agent-generated diff review additions (v5 practice) # On any AI-assisted PR, the reviewer runs through: # 1. Scope creep: did the agent change anything outside the specified scope? # git diff --stat | grep -v# 2. Precondition assumptions: does any function silently assume a sorted input, # a non-None value, or an initialized connection the caller doesn't guarantee? # 3. ACL coverage: if the endpoint is new, does it go through the # @require_experiment_access decorator that 14 other endpoints use? # 4. Retry loop shape: does any new retry have a jitter + backoff cap? # (lesson 6-3: plain retry storms are the most common agent-generated prod incident) # 5. Test to impl ratio: is test coverage going down? If delegation rose # and test lines didn't, the spec is getting fuzzier, not sharper.
WEAKNESSWhere this bites
"We shipped 4× as many features this sprint" is a throughput metric, not a quality metric. Defect-escape probability compounds per feature: if each feature has a 15% escape rate and the team ships 4× as many, the expected defects in production quadruple. The signal that this is happening is not a spike in bug reports — it's a slow rise over weeks, because most defects have a delay between shipment and discovery. Teams that catch this early are the ones that track defect rate per feature shipped, not just raw ticket velocity.
An AI-generated diff is 200 lines. A human reviewer glances at it, the tests are green, and they approve in two minutes. This is not review — it is a social ceremony that produces the false confidence of review without its protection. The correct response is not to stop delegating; it is to write specifications tight enough that review can be done at altitude: "these tests pass, the scope is bounded, the preconditions are enforced upstream — approved." That review takes ten minutes and is genuine. The two-minute rubber stamp is the worst of both worlds: low leverage and low quality.
The seductive move is to delegate the implementation and hold the design. But agents increasingly generate plausible architecture documents, design sketches, and ADRs. The risk is not that the agent's design is bad — it often isn't. The risk is that the engineer stops forming a design opinion before reading the agent's output, and confirmation bias makes the review shallow. The discipline of "write your own sketch first, then read the agent's" is not inefficiency. It is the only way to bring genuine judgment to a review rather than editorial cleanup.
KEY INSIGHTHow this connects
- AI-assisted development (2.10) — the craft-level treatment: adversarial review of AI diffs, the rubber-stamping failure mode, and how to review at altitude. This lesson is the judgment-level complement, focused on the economics and the non-delegable role.
- Estimation, Scope, and Saying No (6.6) — delegation changes how long things take but not how to scope them honestly. The throughput gains from agents feed directly back into scope pressure and the "we can do twice as much" trap.
- Next: Capstone — the blank-page redesign (6.8) — the TraceLab v5 capstone applies everything this lesson taught: the design review and production-readiness gate are conducted as engineer of record, not as curator of generated output.
Everything before this lesson was knowledge. This lesson is the last piece of judgment: knowing when to reach for a tool, how much to trust it, and who is responsible when it gets something wrong. The answer to the last question — always — is you.
GLOSSARYKey terms
- Engineer of record
- The human who bears accountability for a design or implementation decision — regardless of what tool generated the artifact. Borrowed from civil engineering: the licensed engineer who stamps the drawing is personally responsible for the structure.
- Defect-escape probability
- The probability that a bug introduced in a feature makes it past all verification gates into production. Compounds multiplicatively across weakly-applied gates.
- Spec-first delegation
- The workflow of writing tests (or a precise specification) before handing a task to an agent, so the agent has an objective success criterion and the reviewer has an audit trail.
- Context window truncation
- The failure mode where an agent generates code that is locally correct but violates a cross-cutting concern (ACL, retry policy, invariant) that exists outside the context it was shown.
- Plausible-but-wrong
- An agent output that matches the surface pattern of a correct implementation while silently failing on a precondition, invariant, or edge case the agent's training did not surface.
- Rubber-stamp review
- A code review that approves a diff without genuinely engaging with its correctness — providing the social appearance of review without its protection.
- Leverage score
- Throughput per unit of defect-escape risk. Maximized not at maximum delegation but at maximum delegation with gates held — the configuration where verification overhead is paid to preserve quality.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Throughput rises with AI delegation — why doesn't that mean quality is improving?›
Throughput is a count of features shipped; quality is a function of defect-escape probability per feature. If delegation rises without proportionally tightening verification gates, each feature has a higher escape probability. With more features, expected defects in production multiply even as the team feels more productive. The signal lags because most defects have a discovery delay. Tracking defect rate per feature shipped rather than raw velocity is the only way to see this coming early.
2What is the "engineer of record" and why is it non-delegable?›
The engineer of record is the human who owns accountability for a decision — design, implementation, or operation. It is non-delegable because accountability is a social and legal construct that attaches to persons, not tools. When a system fails, the post-mortem asks "who was responsible for this decision?" The agent is not paged, not questioned, not liable. A human is. That human must have genuinely exercised judgment — not rubber-stamped generated output — for the role to mean anything.
3A teammate says "I'll write the tests after the agent implements it — that way I know what to test." What's wrong with this workflow?›
Tests written after an implementation tend to test what the implementation does, not what it should do. The spec-first discipline inverts this on purpose: tests written before implementation encode the contract and surface the agent's deviations. Tests written after are likely to mirror the agent's misunderstandings, including its precondition assumptions and edge-case gaps. The pre-written test is an independent oracle; the post-written test is a description of the implementation.
4An agent generates a correct-looking binary search for TraceLab's run index. Under what condition is this implementation silently wrong?›
Binary search requires a sorted input. If the run index is not guaranteed to be sorted at the call site — and agents do not check whether invariants are enforced upstream — the search returns wrong results on unsorted inputs without error. This is the plausible-but-wrong failure mode: the code matches the surface pattern of a correct binary search while silently depending on a precondition the agent didn't verify and the caller doesn't enforce. The fix is either to sort before the call, enforce sortedness as an invariant at insertion, or use a different data structure. The agent would have suggested any of these if the spec had mentioned the sortedness constraint.