ProjekinAja. Software Engineering Stage 5 · Shipping & Running Lesson 5.3
● Stage 5 · Lesson 5.3

CI: Trust Through Automation

You don't deploy code, you deploy a frozen artifact of it — reproducible builds, pipelines that gate, and flaky tests as trust poison.

⏱ ~18 min read 🛡 Reliability 🤖 TraceLab v4

🎯 By the end you'll understand

Why "it built on my machine" is a real bug class and how hermetic builds eliminate it. How CI pipelines are structured as a DAG of jobs ordered for fast feedback. Why flaky tests are not a test problem but a trust problem — they train humans to ignore the immune system. And how to build TraceLab's CI pipeline with the right parallelism, caching, and failure semantics.

KEY INSIGHTCode on a laptop is a rumor

Software exists when it runs for someone else. Everything before that is a hypothesis. CI is the machine that decides whether your hypothesis is worth shipping.

You've had the conversation. "It works on my machine." The other person runs the same command. It doesn't. Both of you are right, and both of you are wrong, because you're running different machines in different states with different implicit assumptions. The source code is not the artifact. The artifact is what you build from that source under controlled, reproducible conditions — and until your pipeline runs clean, you don't have an artifact. You have a feeling.

🔑 The core move

CI flips the burden of proof. Instead of "trust me, it works," every commit must pass a gauntlet of automated checks before it reaches main. The team doesn't debate whether code is safe to integrate — the pipeline decides, in minutes, every time, on every commit, using the same logic. When the pipeline is green, you have evidence. When it's red, you have a pinpoint. That's the immune system metaphor: CI doesn't prevent illness, it detects and quarantines it before it infects the rest of the body.

CORE CONCEPTSource → artifact: the build is not optional

The pipeline in lesson 5.2 containerized TraceLab: source code went in, a Docker image came out. But why is that step necessary? Why not just clone the repo on the server and run it?

Because source code is not software — it is instructions for producing software. Those instructions are underspecified: they assume a Python version, a set of installed system libraries, a particular pip resolver state on that day, whether libffi happens to be installed, and a thousand other ambient facts. Two machines following the same instructions in different environments will produce different artifacts — or one will fail. "It built yesterday" is a real bug class. It has a name in production engineering: environmental drift.

1

Hermetic builds: seal the environment

A hermetic build runs in an environment where every input is declared and pinned. No ambient system libraries, no network calls at build time, no implicit tool versions. Docker gives you a strong approximation: the FROM python:3.12.4-slim in lesson 5.2 pins the interpreter; COPY requirements.txt then RUN pip install --no-cache-dir -r requirements.txt pins the dependencies. The image is a frozen snapshot of your environment at a point in time. Bazel takes this further: it hashes every input and refuses to build if the hash changes.

2

Reproducible builds: same input, same output

Hermetic builds aim for reproducibility: given the same source, the same lockfile, and the same toolchain version, you get a bit-for-bit identical artifact. Python wheels are mostly there (PEP 517/518 + pip-tools lockfiles), but Prior to Python 3.8, .pyc files embedded timestamps, meaning two builds of identical source produced different hashes — a latent supply-chain vulnerability fixed via SOURCE_DATE_EPOCH support in Python 3.8+. reproducible-builds.org documents the scope of the problem across the ecosystem. The signal to watch: if your CI build produces an artifact with a different hash on each run for identical source, you have non-determinism and you don't yet know where it lives.

3

The artifact as unit of deployment

Once you have a reproducible artifact — a Docker image tagged with a content hash like tracelab:sha-4af22c1 — that artifact is what you test, promote, and deploy. Not a tag. Not a branch. The image. This means what QA tested is exactly what goes to production — same bytes. It also means rollback is trivial: point at the previous image hash. And it means the artifact can be scanned for vulnerabilities before it ever runs in production.

⚠ The non-determinism hiding in plain sight

Your build can be non-deterministic in ways that aren't obvious: pip install requests without pinning will pull different versions on different days. A COPY . . that includes .git/ embeds commit metadata that changes every commit. A build step that calls date embeds a timestamp. Any of these makes your artifact non-reproducible. The fix is always the same: make the input explicit and pin everything that matters.

HOW IT WORKSCI as the team's immune system

A CI pipeline is a directed acyclic graph (DAG) of jobs. Each job runs in a clean environment, does one thing, produces a result (pass/fail plus optional artifacts), and gates the jobs downstream of it. The pipeline fires on every push to a branch — usually a pull-request branch — and the result is either green (safe to merge) or red (do not merge).

The canonical setup for TraceLab looks like this:

git push trigger lint ruff / mypy · ~30s test pytest · ~3min build docker build · ~2min scan trivy / pip-audit · ~1min ✓ merge gate all jobs green run in parallel
The TraceLab CI DAG. Lint and test run in parallel (independent), then build waits for both, then scan checks the image. Any failure anywhere blocks the merge.

The critical design decisions are in the ordering and parallelism.

💡 The 10-minute rule

Google's Site Reliability Engineering practice popularized a useful heuristic: a CI pipeline that takes more than 10 minutes to deliver a result trains developers to not wait for it. They push, context-switch to something else, come back 20 minutes later, and now they have a context-restoration cost on top of a fix cost. This is why fast-feedback ordering (lint early, parallelize aggressively, cache heavily) is not just an efficiency preference — it's a behavioral economics decision about whether your team uses CI as a real gate or a checkbox.

IN PRACTICECaching: the only free lunch in CI

A cold Docker build that re-installs every Python dependency from scratch on every push is tolerable for a small project. For TraceLab at any realistic scale, it's agonizing. CI caching is the solution, and understanding it requires understanding Docker layer caching.

From lesson 5.2: Docker builds layers in order, and each layer is a content-addressed cache entry. If the layer's inputs haven't changed, Docker skips it and reuses the cached result. The canonical pattern for Python:

Dockerfile (cache-friendly ordering)
FROM python:3.12.4-slim
WORKDIR /app

# Layer 1: copy just the deps manifest
COPY requirements.txt .
# Layer 2: install deps — only re-runs when requirements.txt changes
RUN pip install --no-cache-dir -r requirements.txt

# Layer 3: copy source — changes on every commit
COPY . .

CMD ["python", "-m", "tracelab"]

When you push a commit that only changed tracelab/ingest.py, Layer 1 and Layer 2 are cache hits — Docker skips two minutes of dependency installation. Only Layer 3 reruns. This transforms a 4-minute build into a 30-second one for typical commit patterns.

CI systems like GitHub Actions have their own cache layer on top: you can cache the pip download cache or the Docker layer cache explicitly between runs. The important mental model is that caching in CI is not an optimization — it's the expected operating mode. An uncached run is the exceptional case (a new runner, a forced cache bust, a dependency change). Design your pipeline so uncached runs are correct and cached runs are fast.

📖 Social contract of a red main

The pipeline is only as useful as the team's commitment to it. The social contract is simple and non-negotiable in teams that actually operate it: a red main branch is an all-hands emergency. Not "someone fix it eventually." Not "I'll get to it." The person who broke it fixes it immediately, or reverts within minutes. The reason is asymmetric: a broken main stops everyone from shipping. Two hours of blocked developers costs more than one commit of lost work. The revert is almost always the right call while the fix is debugged.

SEE ITBuild TraceLab's pipeline

Configure TraceLab's CI pipeline — choose which jobs run in parallel, toggle caching, then push a stream of commits through. One commit carries a hidden non-determinism (an unpinned dependency) that makes two "identical" builds differ until you pin it. A seeded flaky test will also appear; the pipeline will track how many times it has been retried. Watch the trust score — it erodes every time you blind-retry a failure instead of fixing it.

⚙ Interactive · CI Pipeline Runner

Configure the pipeline, then push commits. Fix the non-determinism and quarantine or fix the flaky test before trust collapses.

lint → test → build → scan → merge gate
Est. pipeline time: — · Press "Push commit" to start
— pipeline idle. Configure above and push a commit. —
Commits pushed
0
Pipeline green
0
Pipeline red
0
Blind retries
0
Build mismatch events
0
Team trust in CI: 100%

Configure the pipeline, then start pushing commits.

Three things to notice. With serial lint+test and no caching, the pipeline takes ~6.5 minutes on a warm run — just inside the 10-minute rule but with no headroom. With parallel + caching enabled, it drops to ~3.2 minutes. The non-determinism event shows why pinning is not paranoia: two "identical" commits producing different build hashes is a supply-chain attack surface, not just an inconvenience. And the trust bar is the most important readout: watch what happens if you keep retrying the flaky test instead of fixing or quarantining it.

WEAKNESSFlaky tests: the slow death of the immune system

A flaky test fails intermittently for reasons unrelated to the code under test. Common causes: a test that depends on timing (time.sleep(0.1) is not a synchronization primitive), a test that shares mutable global state with another test, a test that makes real network calls, a test that assumes a fixed file path exists. The test is not wrong about what it checks — it's wrong about its own reliability.

Why is this so dangerous? Because the failure mode is insidious. The first time the test fails, an engineer re-runs the pipeline. It passes. "Must have been a transient thing." They merge. Three weeks later, the same test fails again. Re-run. Passes. And now something has happened that is very hard to undo: the team has learned that a red pipeline is not necessarily a real signal. The immune system has been poisoned. The next time the pipeline goes red, someone will re-run first and think second — and occasionally they'll re-run through a real failure and ship a bug.

⚠ The retry anti-pattern

Some CI configurations add automatic retries for failed tests — "retry 3 times before failing the build." This is well-intentioned and catastrophically misguided. It masks flakiness instead of surfacing it, inflates pipeline times, and causes the team to lose visibility into test reliability. The correct response to a flaky test is a binary: quarantine it (disable it, file a bug, fix it within N days) or fix it now. A quarantined test is still a tracked debt; a retried test is a hidden one.

💡 Measuring test reliability

Google's testing blog has documented their approach to flakiness at scale: they track a flakiness rate per test over a rolling window, and automatically quarantine tests that exceed a threshold. Tools like pytest-flakefinder can detect flaky tests locally by running each test in isolation multiple times. The key metric is not "did the suite pass" but "what is the false-positive rate of each test" — a number that almost no team tracks and almost every team should.

⚠ The non-determinism that isn't a flaky test

Distinguish flaky tests from test suites that reveal real non-determinism in your application. If your ingest pipeline produces different output on two runs with identical input, that's not a flaky test — that's a bug in TraceLab. A test that detects this is doing its job. The failure is in the code, not the test. Flakiness is specifically when the test infrastructure is the source of randomness, not the application.

TRACELABTraceLab gets a real pipeline

TraceLab v4 ships with a CI pipeline. By this stage, we have: the Stage-2 test suite from the craft milestone, the Docker image from lesson 5.2, and a need to make the whole thing trustworthy on every commit. Here is what goes into .github/workflows/ci.yml:

.github/workflows/ci.ymlTraceLab CI
name: CI
on: [push, pull_request]

jobs:
  # ── Stage 1: lint (fast gate, ~30s) ──────────────────────────
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12.4" }
      - run: pip install ruff mypy
      - run: ruff check .
      - run: mypy tracelab/

  # ── Stage 1: test (parallel with lint, ~3min) ─────────────────
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12.4" }
      - uses: actions/cache@v4              # pip cache across runs
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
      - run: pip install -r requirements.txt
      - run: pytest --tb=short -q

  # ── Stage 2: build (waits for lint + test) ────────────────────
  build:
    needs: [lint, test]            # explicit DAG edge
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Build image
        run: |
          docker build \
            --tag tracelab:${{ github.sha }} \
            --cache-from type=gha \   # use GitHub Actions layer cache
            --cache-to   type=gha,mode=max \
            .

  # ── Stage 3: scan (checks the built image) ────────────────────
  scan:
    needs: [build]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Vulnerability scan
        uses: aquasecurity/trivy-action@v0.18.0  # pin to a version tag, never @master
        with:
          image-ref: tracelab:${{ github.sha }}
          exit-code: "1"               # fail the job on HIGH/CRITICAL CVEs
          severity: HIGH,CRITICAL

A few decisions worth calling out explicitly. The needs: [lint, test] declaration is how GitHub Actions encodes the DAG: it won't start the build job until both lint and test have passed. The hashFiles('requirements.txt') cache key means the pip cache is invalidated exactly when dependencies change — no manual busting required. The image is tagged with github.sha (the commit hash), making every image uniquely and immutably traceable to its source. And the Trivy scan fails the build on HIGH or CRITICAL CVEs, which means a newly disclosed vulnerability in a dependency will block your next deployment until patched — an automatic tightening of the supply chain.

💡 The one thing TraceLab's CI doesn't yet do

This pipeline gates on tests and builds the image, but it doesn't deploy it. That's the job of the CD pipeline in lesson 5.4, which picks up the artifact this pipeline produces. The division is intentional: CI is a trust mechanism, CD is a delivery mechanism. They have different failure semantics, different audiences, and different rollback behaviors. Collapsing them into one pipeline is a common mistake that makes each harder to reason about.

WEAKNESSWhere CI bites

✗ The pipeline tests things the production environment won't run

A CI suite that tests with SQLite while production uses PostgreSQL, or that mocks every external call with in-memory fakes, is providing confidence about a system that doesn't exist. The bugs it catches are real, but the bugs it misses — the ones that depend on PostgreSQL's transaction isolation semantics, or on the real behavior of your S3 client under throttling — will show up in production. Docker Compose can run your full dependency stack in CI at near-zero cost. The gap between "tested" and "tested in a realistic environment" is where production outages live.

✗ Treating a slow pipeline as a fixed cost

A pipeline that takes 25 minutes is not a pipeline; it is a merge blocker that people work around. The common workarounds are: push directly to main, stack uncommitted work until the pipeline clears, or ignore failures until end-of-day. All of these defeat the purpose. Pipeline time is not a natural constant — it's an engineering choice, and a 25-minute pipeline almost always has at least one high-impact optimization available (parallelize the slowest job, cache the heaviest dependency installation, split the test suite). Measure and own it.

✗ "It passed CI" as a substitute for judgment

CI is an immune system, not a brain. It catches what the test suite catches, plus what the linter and scanner catch. It does not catch that your new API endpoint is semantically wrong, that the data migration will take 8 hours on production data, or that the feature will confuse users. A green pipeline means "the automated checks passed." A senior engineer's review is the check that CI cannot replace. The risk is that green pipeline → merge becomes reflexive enough that the second step atrophies.

KEY INSIGHTHow this connects

  • Containers (5.2) — the Docker image this pipeline builds is exactly the artifact we discussed there; the layer-caching model is the same model that makes CI caching work.
  • CD: Releasing Without Fear (5.4) — the artifact CI produces is the artifact CD deploys; next lesson covers canary releases, feature flags, and what "releasing without fear" actually requires.
  • Testing discipline (Stage 2) — the test suite CI runs is the one Stage 2 built; flaky tests in CI are a Stage 2 debt with Stage 5 consequences.

CI is the boundary between "software that exists on a laptop" and "software that exists in the world." Every concept in this stage — artifacts, containers, deployments, infrastructure, observability — assumes that boundary is reliable. A broken CI pipeline is not a CI problem; it's a claim that you don't know whether your software works, which makes everything downstream uncertain.

GLOSSARYKey terms introduced

Hermetic build
A build in which every input is declared and pinned, so the same inputs always produce the same output regardless of the surrounding environment.
Reproducible build
A build that produces bit-for-bit identical output given identical inputs; stronger than hermetic — reproducibility is the property, hermeticity is the mechanism.
Pipeline DAG
The directed acyclic graph of CI jobs; edges encode the needs (dependency) relationships that determine which jobs can run in parallel and which must wait.
Merge gate
A branch protection rule that requires all pipeline jobs to pass before a pull request can be merged to main.
Flaky test
A test that fails intermittently for reasons unrelated to the code under test — typically timing, shared state, or real network dependencies — training humans to ignore red pipelines.
Test quarantine
The practice of disabling a flaky test with a tracked bug rather than retrying it; keeps the signal-to-noise ratio of the pipeline high while the fix is prepared.
Environmental drift
The accumulation of differences between two environments over time, causing builds or runs that succeeded in one environment to fail in another.
Layer cache
Docker's mechanism for skipping unchanged build steps by hashing each layer's inputs; the basis for fast incremental CI builds.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Why is "it works on my machine" a real bug class, not just an excuse?›

Because the source code is underspecified: it implicitly depends on a Python version, system library state, pip resolver behavior at a point in time, and dozens of ambient facts that differ between machines. Two machines following the same instructions in different environments can produce different artifacts or fail differently. This is environmental drift. A hermetic build eliminates it by making every input explicit and pinned — the artifact is reproducible because the environment is declared, not ambient.

2Why does lint come before test in the CI pipeline, and why do lint and test run in parallel rather than sequentially?›

Lint comes first because it's cheap (~30s) and catches obvious failures fast — if lint fails, the 3-minute test run was wasted work. This is the fast feedback ordering principle: put cheap eliminations at the front. Lint and test run in parallel because they're independent — both consume source code and neither depends on the other's output. Parallelism halves the critical path for passing runs without adding any dependency complexity.

3Why are flaky tests dangerous beyond the inconvenience of occasional failures?›

Because they train humans to treat red pipelines as noisy signals rather than definitive ones. Once a team learns that "sometimes you just re-run it," the pipeline's behavioral authority is gone. The next real failure — actual broken code — gets re-run first and investigated second. Flaky tests are a trust problem, not a test problem: they poison the immune system by teaching the immune response to ignore its own signals.

4What does tagging the Docker image with github.sha give you, and why does it matter for the CD pipeline?›

It makes every image immutably traceable to its source commit — you can always look at a running container and know exactly which commit produced it. For CD, this means: what QA tested is what goes to production (same bytes, same tag), rollback is pointing at a previous immutable tag, and any production incident can be correlated precisely to a commit. Mutable tags like latest destroy this property: two deploys of latest might run different code.

Next: CD — releasing TraceLab without fear, and what that actually requires →