ProjekinAja. Software Engineering Stage 5 · Shipping & Running Lesson 5.1
● Stage 5 · Lesson 5.1

Environments, Configuration, and Secrets

Dev, staging, prod, and the eternal mystery of 'works on my machine' — dissected into its actual causes, plus the four ways secrets leak.

⏱ ~18 min read 🔧 Configuration 🔐 Secrets hygiene

🎯 By the end you'll understand

Why "it works on my machine" is almost always a config-layer divergence, not a code bug. How 12-factor config separates credentials from code. The four-layer precedence chain (defaults < file < env < flags) and why it exists. The four channels through which secrets actually escape — git history, log output, build args, and AI chat transcripts — and how TraceLab v4's config layer will handle all of this correctly.

KEY INSIGHTCode on a laptop is a rumor

Software exists when it runs for someone else. Everything before that is a theory about what it will do in an environment you haven't fully described.

You've shipped TraceLab through four stages of the course. You've given it memory, testing, a database, a queue, and an architecture review. But right now it runs on your machine, against a local Postgres, with an API key in a file you've carefully not committed. The moment it needs to run somewhere else — a CI server, a staging box, a production host — you will discover that "the code" and "the configuration" are two different things you've been conflating, and that the gap between them is where most production bugs are born.

🔑 The fundamental split

Adam Wiggins's 12-factor app methodology (2011, Heroku) draws the line precisely: anything that varies between deployments is configuration, not code. Database URLs, API keys, port numbers, feature flags — all config. If you can open-source your codebase without exposing credentials, you've drawn the line in the right place. If you can't, you haven't. The discipline of config management is the discipline of holding that line under pressure.

CORE CONCEPTThe environment ladder

Most software systems graduate through a sequence of environments, each with a different purpose and a different audience for its failures. Knowing what each environment is for prevents a pervasive category of mistakes: running production secrets in dev, promoting untested code because staging was skipped, or treating CI as a staging environment and breaking it constantly.

dev fast feedback loop you pay when it breaks CI gate: tests must pass team pays when it breaks staging prod mirror: catches config drift team pays when it breaks prod users + revenue live here customers pay when it breaks staging is the most underinvested part of most systems — it is also the most important ← config divergence caught here, not in prod
Staging is not a formality. It exists to catch the class of bug that only appears when your config matches production — and that class is large.

The pattern that breaks teams: skipping staging because it's expensive to maintain, then debugging in production because that's more expensive. The 2017 GitLab database incident — 18 hours of downtime after a manual restore of the wrong target — is a canonical example of a staging gap. The restore procedure was never tested against a prod-mirror environment. The config was different. The result was different.

💡 What "prod mirror" actually means

Staging mirrors production in configuration, not necessarily in scale. Same secret management paths, same database engine (not SQLite when prod is Postgres), same OS, same TLS termination. A staging environment on a $5/month VM that uses the same config pipeline as prod will catch divergences. A $500/month staging cluster that uses a different secrets backend will not.

CORE CONCEPT12-factor config: the one rule that scales

The 12-factor methodology's config rule (Factor III) is deliberately aggressive: store config in environment variables. Not config files checked in with the code. Not constants in the source. Environment variables, set by the platform or an operator, read at runtime by the application.

The intuition is sound. An environment variable is:

config.py12-factor style
import os

# read from environment; fail loudly if missing in prod
DATABASE_URL  = os.environ["DATABASE_URL"]
SECRET_KEY    = os.environ["SECRET_KEY"]

# optional with a safe default
LOG_LEVEL     = os.environ.get("LOG_LEVEL", "INFO")
PORT          = int(os.environ.get("PORT", "8080"))

The critique of pure 12-factor: it works well for secrets and deployment-specific values, but falls apart for rich hierarchical configuration (a robot's 40-parameter sensor config doesn't belong in env vars). In practice, systems use a hybrid: secrets and environment handles via env vars; rich non-secret config via files (YAML, TOML); command-line flags for one-off overrides. The precedence question is what matters next.

⚠ The .env file trap

Tools like python-dotenv load a .env file into the environment for local development — which is genuinely useful. The trap: teams start checking .env into the repo ("it's just for dev!"), or copying it to servers ("it's easier than setting real env vars!"). The .env file is a local-dev convenience, not a config system. Add it to .gitignore day zero and never remove it.

HOW IT WORKSConfig precedence: defaults < file < env < flags

Real applications don't get their config from a single source. They layer it: a hardcoded default in code, overrideable by a config file, overrideable by an environment variable, overrideable by a command-line flag. This four-layer precedence chain is nearly universal across serious tooling (kubectl, git, PostgreSQL, Docker) because it covers all operator needs without requiring a single config format.

LayerSourceSet byOverride priority
1 · Defaulthardcoded in codedeveloper at write timelowest — always overrideable
2 · Fileconfig.yaml, .env, pyproject.tomldeveloper or operatorbeats default
3 · Env varexport PORT=9000 / platform secretsplatform or operatorbeats file — this is where prod values live
4 · Flag--port 9001 at invocationoperator at runtimehighest — surgical override for debugging

The chain matters because it makes environments composable. A staging deploy can share the same config file as production (non-secret defaults), override only the database URL via an env var, and let an operator pass --log-level debug for a one-off investigation — without touching the file or the application code.

tracelab/config.pyprecedence chain in 20 lines
import os, argparse, tomllib, pathlib

def load_config(argv=None):
    # Layer 1: defaults
    cfg = {"port": 8080, "log_level": "INFO", "db_pool": 5}

    # Layer 2: file (TOML, optional)
    p = pathlib.Path("tracelab.toml")
    if p.exists():
        cfg.update(tomllib.loads(p.read_text()))

    # Layer 3: env vars (secrets live here)
    if (v := os.environ.get("TRACELAB_PORT")):
        cfg["port"] = int(v)
    cfg["db_url"] = os.environ["DATABASE_URL"]     # required
    cfg["secret_key"] = os.environ["SECRET_KEY"]   # required

    # Layer 4: CLI flags (highest priority)
    p2 = argparse.ArgumentParser(add_help=False)
    p2.add_argument("--port", type=int)
    p2.add_argument("--log-level")
    args, _ = p2.parse_known_args(argv)
    if args.port:       cfg["port"]      = args.port
    if args.log_level:  cfg["log_level"] = args.log_level
    return cfg

SEE ITConfig matrix debugger

A bug appears only in production. The symptom: the ingestion worker crashes on startup with a database connection error — but the same code runs fine in dev and staging. Somewhere in the config chain, a value diverges. Your job: find which layer and which environment contains the actual divergence — then hunt the four places a secret leaked in the mock pipeline.

⚙ Interactive · Config matrix debugger

Two panels: (1) Inspect the env-var matrix across all three environments and identify the layer where the bug lives. (2) Hunt the four secret-leak vectors in the mock repo and pipeline — click each location to reveal how it leaked.

The ingestion worker reads its config via the four-layer chain. Inspect each environment column — the active (winning) value for each key is determined by the precedence resolver on the right. One key has a production-only value that causes the crash.

dev
default: port=8080
file: db_pool=5
env: DATABASE_URL=postgres://localhost:5432/tracelab
env: WORKER_THREADS=4
env: INGEST_QUEUE=ingest_dev
flag: —
staging
default: port=8080
file: db_pool=5
env: DATABASE_URL=postgres://staging-db:5432/tracelab
env: WORKER_THREADS=4
env: INGEST_QUEUE=ingest_staging
flag: —
prod
default: port=8080
file: db_pool=5
env: DATABASE_URL=postgres://prod-db:5432/tracelab
env: WORKER_THREADS=4
env: INGEST_QUEUE=ingest_prod
flag: —
Select a config key above to trace its precedence resolution across environments.

Two things to internalize from the exercise. The crash on DB_SSL_MODE is canonical: prod added a config requirement that dev and staging never knew about because they weren't using the same SSL-enforced Postgres endpoint. The divergence is invisible until prod breaks. The fix is not "add SSL to the code" — it's "make staging run the same config pipeline as prod." The secret-leak exercise illustrates why defense in depth is required: each vector requires a different countermeasure, and teams typically block one while leaving the other three open.

CORE CONCEPTThe honest taxonomy of how secrets escape

Secrets management has a culture problem: the advice is mostly "don't put secrets in code" (accurate) and "use a secrets manager" (also accurate) without addressing the four specific mechanisms by which secrets actually leave controlled environments. Blocking one while leaving the others open is the standard failure mode.

1

Git history

A secret committed to a repo is permanent — git push --force doesn't help because the object is already in GitHub's backend and in every clone. The 2022 Samsung breach exposed internal source code including AWS credentials that had been committed years prior. Countermeasures: pre-commit hooks (git-secrets, detect-secrets), secret scanning (GitHub Advanced Security, GitGuardian), and when breach is confirmed — rotate immediately, treat the old secret as fully public. Removal from history is cosmetic.

2

Log output

Application code that logs request headers, exception payloads, or config dumps will eventually log a secret. HTTP frameworks that log all headers log Authorization: Bearer sk-.... Config objects printed for debugging contain every key. Python's traceback.print_exc() will include the local variables of every frame if your logger is too verbose — including the frame where you just read the secret from env. Countermeasures: structured logging with explicit field allowlists, scrubbing middleware, never logging raw config objects.

3

Build arguments

Docker's ARG instructions for secret injection are preserved in the image manifest and visible in docker history, even if the ARG is used in a single layer and the layer appears empty. Any registry that receives the image now has the secret. The correct pattern: use Docker BuildKit's --secret mount (bind-mounted at build time, not persisted to any layer) or inject secrets at container runtime via env vars or a mounted volume — never as build args.

4

AI chat transcripts

This vector is new and underacknowledged. Engineers debugging a config issue paste environment variable dumps, Kubernetes manifests, or CI pipeline logs into AI chat windows — including production secrets in the payload. The context is sent to the AI provider's servers. Some providers retain logs; all logging infrastructure that processes those logs now holds the secret. Countermeasure: redact secrets from any paste before sending. A simple rule: if it contains anything that would fail a detect-secrets scan, redact it before pasting.

💡 Secrets managers: the correct long-term answer

HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, and similar tools exist to remove secrets from config files and environment variables entirely. The application authenticates to the secrets manager (via IAM role, Kubernetes service account, or Vault agent) and retrieves the secret at startup. The secret is never in any file, build artifact, or env var dump — only in the process's memory. For production TraceLab, this is the right architecture. For this lesson, the principle matters more than the specific tool.

Leak vectorDetectionCountermeasureAfter breach
git historydetect-secrets, GitHub scanningpre-commit hooks, never commit secretsrotate immediately; history removal is cosmetic
log outputlog review, SIEM patternsallowlist-based structured loggingrotate; audit log consumers
build argsdocker history <image>BuildKit --secret mount; runtime injectionrotate; re-build & re-push all affected images
AI chatDLP tooling, policy reviewredact before pasting; use local models for sensitive workrotate; notify affected provider if applicable

TRACELABTraceLab v4 — config and secrets layer

TraceLab is heading toward its v4 milestone: containerized, CI-built, canary-deployed, and instrumented. Before any of that machinery works, the config layer has to be correct — because every component in the milestone (the container, the CI pipeline, the infra-as-code declarations) will need to read secrets and environment-specific values cleanly. This is what we commit to building.

tracelab.tomlnon-secret config, safe to commit
# tracelab.toml — committed to the repo. No secrets here.
[server]
port       = 8080
log_level  = "INFO"

[ingestion]
queue_name = "ingest"
batch_size = 256
timeout_s  = 30

[database]
pool_size  = 5
pool_max   = 20
ssl_mode   = "prefer"   # overridden in prod via env: TRACELAB_DB_SSL_MODE=require
.env.examplecommitted; .env is gitignored
# Copy to .env for local dev. Never commit .env.
DATABASE_URL=postgres://localhost:5432/tracelab
SECRET_KEY=dev-secret-change-in-prod
TRACELAB_DB_SSL_MODE=disable
.gitignoreday-zero additions
.env
*.pem
*.key
secrets/
# detect-secrets baseline (next lesson: CI gate)
.secrets.baseline

The debt this repays: TraceLab v0 had no notion of config at all — the database path was hardcoded, there was no concept of environments, and every credential used in later versions would have leaked into the repository the moment someone reached for a quick fix. The four-layer chain and the gitignored .env eliminate the most common vectors. The next two lessons (containers and CI) will add build-time and runtime secret injection. By the end of the milestone the config layer will be solid enough to survive a production security audit.

📖 The debt ledger update

TraceLab v0's "the CSV schema lives in nobody's head" debt (noted in Lesson 1.1) was repaid by the Stage-4 architecture work. The implicit "config lives nowhere in particular" debt — visible since v0's hardcoded file paths — is repaid here. The remaining v4 debts (container layer discipline, CI pipeline gates, infra-as-code) are Lessons 5.2 through 5.5.

WEAKNESSWhere this bites

✗ Config parity theater

Teams declare that staging "mirrors production" while making exceptions: "we use SQLite in staging because it's faster to set up," "we skip SSL because the cert is annoying to renew," "we use a different message queue because the prod one costs money." Each exception is a divergence vector. If staging doesn't use the same database engine, SSL mode, and secrets injection path as production, it will miss exactly the class of bug — config-layer divergence — it exists to catch. Parity in the important dimensions is cheap; parity theater is expensive.

✗ The "it's just a dev key" fallacy

Engineers commit a secret with the mental footnote "this is a dev-only key with limited permissions." Two problems: git history is permanent and keys get rotated less often than that footnote implies; and "limited permissions" is a claim that requires audit, not faith. The GitHub token with only repo:read scope that was committed "for CI" in 2019 is still in the history of that repo, and its scope may have been upgraded since. The rule is binary: secrets do not go in the repo, regardless of their claimed scope.

✗ Treating config validation as optional

Many prod outages that look like "the app crashed on startup" are actually "a required environment variable was absent or malformed, and the app didn't notice until it tried to connect to the database forty seconds into boot." Validate your entire config schema at startup — fail immediately with a clear error message listing every missing or invalid value. Pydantic's BaseSettings, dynaconf, or even a 10-line custom validator will save hours of production debugging per year.

KEY INSIGHTHow this connects

  • Dependencies and Packaging (2.9) — virtualenvs and lockfiles are how you keep the runtime reproducible; this lesson adds the config layer that virtualenvs deliberately don't touch.
  • The Shell Is a Programming Language (1.4) — environment variables are a shell concept; the export, printenv, and source .env patterns used throughout this lesson originate there.
  • Containers (5.2) — containers consume config and secrets at runtime; the patterns from this lesson (env var injection, BuildKit secrets) map directly onto Dockerfile and docker-compose design.
  • Infrastructure as Code (5.6) — IaC tools declare what environment variables the infra provides; understanding the config layer makes IaC declarations legible instead of magical.

Config management is the first act of every deployment, and it is the hidden cause of most "it works on my machine" bugs. Everything in the rest of Stage 5 — the container, the pipeline, the infra declarations — is built on the assumption that config and secrets are handled by a principled system. This lesson is that foundation.

GLOSSARYKey terms introduced

12-factor config
Factor III of the twelve-factor app methodology: store config in environment variables so that credentials and environment-specific values are strictly separated from code.
Config precedence chain
The layered resolution order defaults < config file < environment variable < CLI flag, where each layer can override the one before it.
Environment parity
The property that staging and production use the same config pipeline, secrets injection path, and database engine — so staging actually catches config-layer divergences.
Secret leak vector
A channel through which a credential can escape controlled environments: git history, log output, Docker build arguments, and AI chat transcripts are the four primary vectors.
Build arg (Docker)
A value passed to docker build via --build-arg, persisted in the image manifest and visible in docker history; an unsafe channel for secrets.
Secrets manager
A dedicated service (Vault, AWS Secrets Manager, GCP Secret Manager) that stores credentials and serves them to authenticated callers at runtime — removing secrets from files, env var dumps, and build artifacts.
.env file
A file that sets environment variables for local development; should be gitignored and never deployed to servers or CI systems.
Config validation
Checking at startup that all required config values are present and correctly typed; failing immediately with a clear error if they are not.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1According to 12-factor, what is the defining test for whether something is config or code?›

The test is: does it vary between deployments? If you can open-source the codebase without exposing credentials, you've correctly separated config from code. Database URLs, API keys, feature flags, and port numbers vary between dev/staging/prod — they are config. Business logic, data transformations, and algorithms don't vary — they are code. The practical corollary: if it lives in an environment variable and is never committed to the repo, it's almost certainly config. If it would be the same on every developer's machine and on every server, it's probably code.

2A bug appears only in production. What is the first hypothesis to test, and what tooling would you use?›

The first hypothesis is a config-layer divergence: a value that is set differently in prod than in dev/staging. Compare the active config across environments — start with env vars, then config files, then defaults. If staging doesn't mirror prod's config pipeline (same secrets backend, same database engine, same SSL mode), staging won't catch it. Tooling: printenv | sort on each host, or a structured config-diff tool. If the application validates its config at startup and logs the resolved values (without logging secret contents), this comparison is fast.

3A colleague says "I committed a dev key with read-only permissions, so it's fine — I can leave it in the git history." What's the correct response and why?›

The response: rotate it now regardless of claimed scope, and treat the history as public. Two reasons. First, git history is permanent — force-pushing doesn't remove the object from GitHub's backend or any clone. Second, "read-only" is a claim about the key's scope at the time of this statement; keys get scopes added, permissions drift, and the original statement can't be retroactively verified years later. The rule is binary: secrets don't go in the repo. A one-time rotation is cheap; the alternative is permanent uncertainty about what that key can do.

4Why is docker build --build-arg API_KEY=secret unsafe even if the resulting image layer looks empty?›

Build arguments are preserved in the image manifest and are visible via docker history <image>, even when the layer that used the ARG appears empty or is squashed. Any registry that receives the image — Docker Hub, ECR, a private registry — now holds the secret in its metadata. The correct patterns are Docker BuildKit's --secret mount (which binds the secret into the build context without persisting it to any layer) or runtime injection via environment variables or a mounted Kubernetes secret volume. Build args are for build-time configuration, not credentials.

Next: what Docker actually is, and why layer discipline matters →