ProjekinAja. Software Engineering Stage 5 · Shipping & Running Lesson 5.4
● Stage 5 · Lesson 5.4

CD: Releasing Without Fear

Blue-green, rolling, canary, and feature flags — deploy frequency is a safety mechanism, and big-bang releases are the dangerous ones.

⏱ ~22 min read 🚢 Continuous Delivery 🔬 TraceLab v4

🎯 By the end you'll understand

Why blast radius, detection time, and rollback cost are the right axes for comparing deployment strategies — not speed or complexity. You will understand blue-green, rolling, and canary deployments mechanically (not just as marketing terms), how feature flags decouple deploy from release and create kill-switch capability, why flag debt is as real as technical debt, and how rollback should be a designed, rehearsed capability rather than a panic maneuver. The widget lets you ship the same latent bug under each strategy and measure — not believe — the difference.

KEY INSIGHTHigh deploy frequency is the safety property

The intuition is exactly backwards: the less often you deploy, the more dangerous each deploy becomes — because the blast radius of a mistake grows with the batch size.

Most engineers, when they start, believe the safest release strategy is the infrequent one. Review everything, test for weeks, ship on a Friday with a change freeze in place. This intuition is wrong in a precise, measurable way. The quantity of change in a release is its primary risk factor. A deploy that changes two lines of YAML and one dependency version is almost certainly safe. A deploy that bundles four months of changes from twelve engineers — the kind of "big bang" that emerges naturally from low deploy frequency — is one in which any bug could be anywhere, rollback means discarding weeks of work, and detection is slow because the signal-to-noise ratio in your error charts is terrible.

🔑 Hold onto this

The way to make releases safe is to make them small, and to make them small, you need to make them frequent. This is not recklessness; it is the Accelerate / DORA finding from studying thousands of engineering teams. High-performing teams deploy multiple times per day and have lower change failure rates than low-performing teams deploying quarterly. Frequency is the safety strategy. The progressive delivery techniques in this lesson are the mechanisms that make high frequency tenable in production.

CORE CONCEPTDeploy ≠ release: the crucial decoupling

Before comparing strategies, pin down a distinction that most teams conflate until something goes badly wrong. Deploy means copying artifacts to production infrastructure — the process that CI/CD automates. Release means making a feature available to users. Historically these were the same moment. Progressive delivery techniques, and feature flags in particular, let you decouple them entirely.

EventWhat actually happensWho decides timing
DeployNew container image starts running on infrastructure; routing may still send 0% of traffic to itCI/CD pipeline, fully automated
ReleaseReal users start exercising the new code path; blast radius becomes non-zeroProduct decision, can happen independently
RollbackTraffic or flag state reverted to the previous known-good configurationOn-call engineer or automated circuit breaker

When you can deploy at any time without releasing, your pipeline can run continuously in the background, landing changes dark while teams gather confidence. Releasing becomes a business decision, not a coordination event between infrastructure, QA, and the board. Rollback becomes reversing a config change, not reverting and re-deploying code. This decoupling is the architecture underpinning the "deploy many times a day" organizations you've heard about. None of their bravery is actual bravery — it is designed safety margin.

💡 TraceLab context

When TraceLab v4 ships its new canary-aware ingestion endpoint, the endpoint is deployed (container running, route registered) but traffic is initially 0% — the release dial starts at zero. Your robot fleet can be enrolled in the canary while your laptop stays on the stable version. That topology is what the widget below is simulating.

HOW IT WORKSFour strategies, three axes

Compare strategies on the three axes that determine your production risk posture, not on speed or implementation complexity:

1

Blue-green deployment

You maintain two identical production environments: blue (live) and green (idle). Deploy the new version to green, run smoke tests, then flip the load balancer to send 100% of traffic to green. Blue stays up as hot standby. On failure, the flip is reversed — rollback takes seconds, costs zero traffic. The catch: you pay for two full environments simultaneously, and the moment you flip, all users hit the new version. Blast radius is 0% until flip, then 100%. Good for predictable, rehearsable cutovers; bad for discovering subtle bugs that only surface under realistic load patterns.

2

Rolling deployment

Replace instances of the old version one-by-one (or in small batches). Kubernetes's default strategy. At any moment, old and new instances run simultaneously — a mixed fleet. Blast radius grows gradually as the rollout proceeds. Detection time depends on whether you have alerting and how many instances carry the bug before it fires. Rollback requires replacing new instances back to old, which takes time proportional to how far the rollout progressed. State compatibility matters: your new and old versions must speak the same API and database schema during overlap. Rolling is simple and requires no extra infrastructure, but "I'm 60% rolled out and errors are spiking" is an awkward position — your blast radius is already real.

3

Canary deployment

Send a small, controlled slice of traffic — the canary — to the new version. 1%, 5%, 10% — your choice, based on required statistical power. Monitor error rate, latency, and custom metrics for a defined window. Promote if healthy; halt and roll back instantly if not. This is named for the canary-in-a-coal-mine practice: the canary gets hurt so the miners don't. Blast radius is bounded and explicit: 10% canary means at most 10% of users see the bug. Detection time can be very short if your observability is good. Rollback cost is low — shift traffic back. Automated canary analysis (Spinnaker, Argo Rollouts, Flagger) can promote or roll back without human intervention based on thresholds you configure. This is the strategy the widget below makes you measure.

4

Feature flags (kill switches)

Feature flags are boolean (or multi-variant) configuration evaluated at runtime. The code path for a feature ships in every binary, but the flag gates whether it executes. This means you can deploy code dark, enable it for 1% of users, roll back instantaneously by flipping a bit in a config store, do A/B tests with the same artifact, and give operations a kill switch that doesn't require a new deploy. Flags can be per-user, per-tenant, geographic, or by any dimension. The blast radius is exactly the targeting rule. Rollback cost is near-zero: flip the flag, propagated in seconds. Tools: LaunchDarkly, Unleash (open source), Flagsmith, or even a simple Redis key your app polls.

Detection time (how fast the signal fires) → Blast radius → Big-bang 100% Blue- green Rolling grows Canary bounded Flag targeted slow → ← fast 100% ~0%
Strategies plotted on blast radius vs detection time. Feature flags and well-configured canaries land bottom-left. Big-bang releases live in the worst corner. Blue-green has low blast radius but poor detection — you only know about the bug when 100% of traffic sees it.

SEE ITDeployment strategy simulator

TraceLab v4 ships a new ingestion endpoint. Unknown to us, it contains a 10% defect rate — one in ten requests silently returns a corrupt row instead of rejecting the input. The same latent bug ships under all three strategies. You're going to measure blast radius, detection time, and rollback cost — not believe someone's marketing copy about why canaries are better.

⚙ Interactive · Deployment strategy simulator

Choose a strategy, step through the rollout, watch the defect propagate (or not), then ship the same bug behind a feature flag and hit the kill switch.

Blast radius
—
users exposed to bug
Detection time
—
minutes to alert firing
Rollback cost
—
minutes to full recovery
Rollout timeline
Press "Run rollout" to begin.
Now: ship the same bug behind a feature flag

The buggy build is already deployed. Use flags to control who sees it. Hit "Kill switch" to simulate an on-call engineer toggling the flag off mid-incident.

ingestion_v4_new_parser ON · 100% of traffic
ingestion_v4_canary_1pct OFF · 0% of traffic
Flag state: ingestion_v4_new_parser = ON (100%) | ingestion_v4_canary_1pct = OFF

Three things worth sitting with. Blue-green's rollback is fastest because the old environment never stopped running — but its detection time is worst because you only learn about the bug once 100% of traffic hit the new version. Canary's detection is fast (only 5–10% of traffic probes the bug before the alert fires) and rollback just reweights traffic. Rolling is the most dangerous middle ground: blast radius grows continuously during the rollout, and rollback is not instant. And the feature flag simulation shows what zero blast radius feels like: the bug was already deployed but isolated. The kill switch fires in seconds, and no new deploy is needed.

IN PRACTICEFeature flags: power and the debt they accumulate

Feature flags are one of the highest-leverage tools in a production engineering toolkit, and also one of the most reliably mismanaged. Used well, they give you:

⚠ Flag debt is the new dead code

A flag that was supposed to be temporary for a canary rollout in January, is still in the codebase in October, branching production code, adding cognitive load, and suppressing test coverage for the "disabled" path — is dead code that charges rent. Teams that don't aggressively expire flags end up with dozens of interacting boolean branches that no one fully understands. When a senior engineer at a large company (Knight Capital is the famous, awful example — though their tooling was homegrown flag-adjacent deployment automation, not a modern feature flag system) triggers the wrong flag state, the cost can be catastrophic. Treat every flag as a liability with an expiry date. Remove the old code path when the flag is fully rolled out. Review flags in every sprint retro.

Operationally, the right feature flag setup for TraceLab looks like this: a flag service (or at minimum, a YAML file in a versioned repo polled on startup), a structured naming convention (feature.ingestion_v4_new_parser), an expiry annotation in the code (# TODO(2026-07-01): remove when v4 fully rolled out), and a lint rule that fails CI if a flag has been in the codebase more than 90 days without the annotation being updated. That's the entire policy for a team of six. You can build in a day what LaunchDarkly charges for — until you need the SDKs, the audit trail, and the targeting rules, at which point you pay them.

flags.pyTraceLab flag client — minimal
# flags.py — reads YAML flag state; poll interval set externally.
# TODO(2026-08-01): remove ingestion_v4_new_parser once fully rolled out
import yaml, os, functools

def _load():
    path = os.environ.get("FLAG_FILE", "/etc/tracelab/flags.yaml")
    with open(path) as f:
        return yaml.safe_load(f) or {}

@functools.cache   # cached per process; call flags.cache_clear() on SIGHUP — add signal handler separately
def flags():
    return _load()

def is_enabled(name, default=False):
    return flags().get(name, default)

# Usage in ingestion handler:
# if is_enabled("ingestion_v4_new_parser"):
#     return new_parse(row)    # ← new, potentially buggy
# return legacy_parse(row)     # ← stable fallback

CORE CONCEPTRollback as a designed capability

Teams that treat rollback as a panic move — something that happens when everything is on fire, improvised by whoever happens to be on-call — pay for that neglect in exactly those moments. Rollback needs to be a rehearsed, documented, sub-minute procedure before any of the progressive delivery strategies in this lesson deliver their promised safety properties.

The hard part is database migrations. Code rollback is trivially easy (ship the previous container image). Database schema changes are not: if the new code added a column and wrote into it, the old code doesn't know the column exists and may fail on rows that have it, or corrupt them. The patterns that make rollback safe are:

1

Expand / contract (strangler fig for schema)

Never make a migration and a code change in the same deploy. First, expand the schema in a backwards-compatible way: add the new column as nullable, deploy the old code (which ignores it). Then deploy the new code that writes to the new column. Then, in a later deploy, remove the old column. At every step, the previous version of the code can run safely against the schema. Rollback is always safe.

2

Feature flags gate the data write

Even with expand/contract, the new code path can be flagged off until the canary completes. The new column remains dormant — it's in the schema but nothing writes to it — until the flag is enabled. Kill the flag, and the data migration simply hasn't happened yet. No rollback needed.

3

The rollback runbook lives in the repo, not in someone's head

docs/runbooks/rollback.md: current stable image SHA, one-liner to roll Kubernetes back (kubectl rollout undo deployment/tracelab-ingest), flag kill-switch command, expected recovery time, who to page next. That file should be boring because it's been used. Fire drill quarterly; it takes ten minutes and the value is enormous.

📖 Mean Time to Recovery (MTTR)

One of the four DORA key metrics. The time from an incident starting to the system being restored to a healthy state. Low MTTR requires fast detection (observability), fast action (the strategies in this lesson), and low rollback cost (designed rollback capability). DORA research found MTTR is a stronger differentiator between elite and medium-performing teams than even deployment frequency.

TRACELABTraceLab v4 ships and survives

TraceLab v4 is the Stage 5 milestone: containerized (Lesson 5.2), built by a CI pipeline that gates on the Stage-2 test suite (Lesson 5.3), and — this lesson — canary-deployed via declared infrastructure. Here's the production topology and the specific steps that make the v4 ship safe:

deploy/rollout.yamlArgo Rollouts canary config (abbreviated)
# TraceLab v4 ingestion service — Argo Rollouts canary strategy
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: tracelab-ingest
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5        # 5% canary first
        - pause: {duration: 10m}  # watch error rate for 10 minutes
        - setWeight: 20
        - pause: {duration: 10m}
        - setWeight: 100       # promote if no rollback triggered
      analysis:
        templates:
          - templateName: tracelab-error-rate
        args:
          - name: error-rate-max
            value: "0.02"     # abort canary if error rate > 2%

The analysis template queries the Prometheus metric exported by TraceLab's instrumentation (coming in Lesson 5.7 — the instrumentation is added before the pipeline but evaluated here). If error rate on the canary replica set exceeds 2%, Argo Rollouts automatically aborts and restores 100% traffic to the stable version. The on-call engineer is paged but the blast radius was capped at 5% for the first ten minutes.

deploy/flags.yamlTraceLab flag state — gitops managed
# flags.yaml — managed in git, applied to all environments via CI
# Remove flag + old code path after 2026-08-01 (v4 fully stable)
ingestion_v4_new_parser: true
ingestion_v4_bulk_endpoint: false    # not yet ready; ships dark
dashboard_realtime_charts: false     # performance work still in flight

The flag file lives in the same repo as the code and ships through the same CI pipeline. It is not an emergency-override mechanism — it is version-controlled configuration. Emergency overrides are kubectl patches applied by the operator and immediately reverted in git once stable; the runbook for that procedure is in docs/runbooks/flag-emergency.md.

🔑 TraceLab v4 debt ledger — what this lesson settles

The stage-3 concurrency debt ("no locking — concurrent writers interleave") is partly mitigated by the v4 ingestion service running as a single-writer replica set with the canary receiving 5% of traffic. The real lock-free design arrives in Stage 3's database lessons and has already been deployed; the canary is validating it under production concurrency for the first time. The stage-4 "schema lives in nobody's head" debt is addressed by the expand/contract migration pattern wired into the CI pipeline's pre-deploy check step.

WEAKNESSWhere this bites

✗ Treating canary as theatre without real analysis

Deploying 1% canary for 5 minutes and calling it safe is not a canary — it's a deployment with a placebo. Statistical significance matters: a 10% defect rate affecting 1% of traffic for 5 minutes at low volume may generate zero errors by chance. Canary analysis is a statistics problem: you need enough traffic and enough time to detect your target defect rate at acceptable confidence. Netflix's automated canary analysis (the original inspiration for tools like Spinnaker and Kayenta) uses Mann-Whitney U tests over request rate, error rate, and latency distributions. If your traffic is too low to give the canary statistical power, use a different signal (synthetic traffic, integration tests run against canary, or manual inspection) — but be honest that you're working around the math, not doing the math.

✗ Feature flag sprawl and the Knight Capital lesson

In August 2012, Knight Capital lost $440 million in 45 minutes because a deployment activated a dormant code path (via a repurposed flag bit from years earlier) that placed unintended market orders. The root cause involved a flag-like mechanism being used inconsistently across 8 servers — one server out of eight had old code that reacted to a flag value the new code used for something else. The lesson is not "feature flags are dangerous." The lesson is that flags require discipline: named consistently, documented, expired aggressively, applied uniformly across all replicas, and never repurposed for different meanings over time. "Ship it to 7 of 8 servers and forget the 8th" is the deployment failure; the flag was the mechanism through which the failure expressed itself.

✗ Rollback that isn't actually possible

The most dangerous words in production are "we'll just roll back if something goes wrong." For stateful systems — databases that have been written to, message queues that have been consumed, S3 objects that have been deleted — "just roll back" may be physically impossible. A rollback that requires manually deleting new-format rows, re-ingesting queued messages, or reconstructing deleted state is not a rollback plan; it is a crisis-response improvisation. Build rollback safety into every migration from the start (expand/contract), test the rollback path in staging, and be honest in your runbook if a particular deploy is one-way — because at least then the team knows they need to solve the problem forward, not backward.

KEY INSIGHTHow this connects

  • CI: Trust Through Automation (5.3) — the pipeline that builds and tests TraceLab v4 before the canary can begin; a broken build never reaches the rollout controller.
  • Cloud Primitives and the Bill (5.5) — the infrastructure the canary deploys onto; blue-green and canary both have real cost implications (two active environments vs weighted routing) that the next lesson makes calculable.
  • Testing (2.1) — the test suite that the CI gate runs before a canary is allowed to start; a canary that passes broken tests is not a safety mechanism.
  • Concurrency (3.1) — the mixed-fleet period of rolling deployments is a real concurrency scenario: old and new code run simultaneously against the same data store.

This lesson is the last piece of the deploy half of Stage 5. Everything else in the stage — observability, SLOs, incidents — assumes you have progressive delivery in place, because those tools are only valuable if you can act on what they tell you quickly. Observability without the ability to roll back safely is a smoke alarm in a room you can't exit.

GLOSSARYKey terms introduced

Blue-green deployment
A strategy maintaining two identical production environments; traffic is switched atomically from one to the other, with the old environment held as instant rollback.
Rolling deployment
Replacing instances of the old version incrementally; old and new run simultaneously during the rollout, making schema and API compatibility mandatory.
Canary deployment
Sending a controlled percentage of traffic to the new version before full promotion; named for the canary-in-a-coal-mine practice of using a sensitive indicator to detect danger before it spreads.
Feature flag
A boolean (or multi-variant) runtime configuration value that gates a code path without requiring a new deploy; enables decoupling deploy from release and provides kill-switch capability.
Blast radius
The fraction of users or systems exposed to a defect or outage before it is detected and contained; the primary risk dimension for comparing deployment strategies.
Expand / contract migration
A schema migration pattern where changes are made in two separate, backwards-compatible steps — add the new column first, deploy the new code second, remove the old column last — so rollback is always safe.
MTTR (Mean Time to Recovery)
One of the four DORA metrics: average time from an incident starting to service restoration. DORA research identifies MTTR as a key differentiator between elite and low-performing engineering teams.
Flag debt
The accumulated cognitive overhead, branching complexity, and dead-code risk from feature flags that have outlived their intended lifetime and not been removed.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Why does deploying more frequently make releases safer, not riskier?›

Deployment risk is primarily a function of batch size — the volume of change in a single deploy. Frequent deploys are small deploys: a bug is easier to find, blast radius is smaller, and rollback reverts little work. Infrequent deploys bundle months of changes: bugs can be anywhere, rollback loses weeks, and detection takes longer because signals are noisier. DORA research confirms that high-performing teams deploying many times per day have lower change failure rates than low-frequency teams. Frequency is the safety strategy.

2You ship a new TraceLab ingestion endpoint as a 5% canary. What determines whether the canary actually protects you?›

Two things: observability (does your error-rate metric actually fire on the defect you're worried about?) and statistical power (is 5% of your traffic volume enough, over the analysis window, to detect your target defect rate at acceptable confidence?). A canary with no automated analysis, or too little traffic to achieve statistical significance, is not a safety mechanism — it is a deployment with extra steps. The analysis must be defined before the canary starts, not improvised when errors appear.

3What is the expand / contract pattern, and why does it exist?›

A two-phase database migration strategy that keeps every intermediate state backwards-compatible with the previous code version. Phase 1 (expand): add the new column as nullable, deploy — the old code runs fine, it ignores the column. Phase 2: deploy the new code that writes to the new column. Phase 3 (contract): remove the old column in a later deploy. At every point, rolling back to the previous image is safe because the schema that image expects still works. The alternative — migrating schema and deploying new code together — makes rollback impossible if the new code writes data the old code can't handle.

4What is "flag debt" and what's a concrete policy to prevent it?›

Flag debt is the accumulation of feature flags that outlive their intended purpose — they keep branching production code, suppressing test coverage for the "off" path, and adding cognitive load for engineers who didn't write them. The Knight Capital incident (2012) illustrates the extreme case: a repurposed flag bit activated dormant code on a partial server set with catastrophic results. Concrete policy: every flag must have an expiry annotation in the code, a lint rule fails CI if the annotation is stale, and flags are reviewed each sprint retro. Remove the old code path when the flag reaches 100% rollout — don't leave the branch in the codebase.

Next: cloud primitives and the bill — what you're actually paying for →