ProjekinAja. Software Engineering Stage 5 · Shipping & Running Lesson 5.6
● Stage 5 · Lesson 5.6

Infrastructure as Code

Clicking in a cloud console is unversioned mutation of production — desired state, plan/apply, and the drift that bites back.

⏱ ~20 min read ☁ Infrastructure 🔁 Declarative systems

🎯 By the end you'll understand

The declarative mental model: desired state, reconciliation, and why the plan-before-apply discipline exists. What a state file is, why it's the new single point of failure, and what happens when it diverges from reality. The signature failure mode of IaC — manual drift — where someone "quickly fixes" something in the console and the next plan threatens to destroy their hand-made resource. And how TraceLab v4's infrastructure will be declared, versioned, and safely reconciled.

KEY INSIGHTThe console is a write-only lie detector

Every change you make by clicking in a cloud console is a mutation with no author, no history, no review, and no rollback. It happened, but it is not written anywhere you control.

You've spent the last lesson understanding what you're renting from AWS or GCP. Now consider what happens the first time something goes wrong at 2 a.m. and someone — maybe you — clicks into the console and changes a security group rule "just to see if that's the issue." It works. They go to bed. Two weeks later, a colleague runs a Terraform apply to add a new subnet and the plan says it will destroy that security group rule and recreate it from the declared config. Panic. Did the manual change matter? Was it a fix or a workaround? Nobody remembers. The console doesn't know. This is the story IaC exists to prevent.

🔑 Desired state, not imperative commands

Infrastructure as Code tools — Terraform, Pulumi, AWS CDK, OpenTofu — work declaratively: you describe the desired state of your infrastructure ("I want a 2-CPU VM, this security group, this subnet") and the tool calculates the diff between that and current state, then applies only the necessary changes. This is fundamentally different from writing bash scripts that call AWS CLI commands imperatively. The bash script is a procedure; the HCL or YAML is a specification. One tells the system what to do. The other tells it what to be.

CORE CONCEPTDeclaring desired state

The key move in IaC is to stop thinking about actions ("create a VM") and start thinking about invariants ("a VM with these properties must exist"). Here's a minimal Terraform-style declaration for TraceLab's ingestion worker — simplified but structurally accurate:

infra/tracelab.tfdesired state
# Every block names a resource. The tool owns making reality match this.

resource "cloud_vm" "ingestion_worker" {
  name      = "tracelab-ingestion"
  machine   = "e2-standard-2"       # 2 vCPU, 8 GB
  image     = "ubuntu-22-04-lts"
  zone      = "us-central1-a"
  disk_gb   = 40
}

resource "cloud_firewall" "allow_ingest" {
  name    = "tracelab-allow-ingest"
  port    = 8080
  source  = "10.0.0.0/8"             # internal network only
  vm_id   = cloud_vm.ingestion_worker.id
}

resource "managed_db" "runs_db" {
  name    = "tracelab-runs"
  tier    = "db-g1-small"
  version = "POSTGRES_15"
}

Notice what this file does not say: no "create VM" command, no "if the firewall exists, skip it," no sequence of operations. It says what should be true. The tool's job is reconciliation — compute the gap between this spec and current reality, then close it with the minimum set of API calls. That gap computation is what the plan step produces, and it's the first thing you learn to read carefully.

📖 Reconciliation loop

The pattern — describe desired state, compare to observed state, apply delta — appears everywhere in systems software: Kubernetes controllers, configuration management (Ansible, Chef), DNS replication, git merges. Once you internalize it, you'll recognize it in places that don't advertise it. The correctness question always reduces to: "is the observed state a valid instantiation of the desired spec, and if not, what's the minimal change?"

The practical benefit extends far beyond disaster recovery. When desired state lives in a git repository, you get everything git gives you for free: authorship, diffs, code review, branches, tags, history. A Tuesday morning conversation of "when did we change the firewall rule?" has a five-second answer. A "revert to last month's infrastructure" is a git revert followed by an apply — not a tense conversation about what someone might have done in the console.

Git repo infra/*.tf read terraform plan desired vs actual state file (tfstate) diff terraform apply closes the delta API calls Cloud APIs real resources console click = bypasses all of this
The IaC workflow. Code in git → plan computes diff against state file and live cloud → apply executes minimal API calls. Console clicks short-circuit the loop, creating untracked mutations.

HOW IT WORKSPlan before apply: the discipline that saves you

The plan step is IaC's most important safety property. It separates the question "what would happen?" from "make it happen." Before any apply, you read a plan that shows — in color-coded lines — every resource that will be created, changed, or destroyed. The plan is the diff. You are responsible for reading it before approving it.

Plans have three line types:

SymbolMeaningSafe to auto-approve?
+ adda new resource will be created; no existing resource is touchedusually yes
~ changean existing resource will be modified in-place; the resource keeps runningdepends on the attribute
-/+ destroy & recreatethe old resource will be destroyed and a new one created; there is downtimeread carefully
- destroythe resource will be deleted entirelyalmost never

The destroy-and-recreate case is the one that surprises people most. Many cloud attributes are immutable — you cannot change them in-place. A VM's machine type in some providers, a database's engine version, a managed queue's partitioning scheme. If you update one of these in your desired-state config, the plan shows a destroy-then-create sequence. If that resource holds data or is in your production critical path, the plan just showed you an outage. This is information — the right response is to design around it, not to approve blindly.

💡 Plan output is a specification, not a suggestion

Experienced infrastructure engineers read the plan line by line before every apply, even for "small" changes. The plan told you. If you approved a destroy you didn't understand, that's on you — the tool was transparent. The discipline is simple: never type yes faster than you can read.

CORE CONCEPTThe state file: your new single point of failure

How does the IaC tool know what "current state" is? It keeps a state file — typically terraform.tfstate — a JSON record mapping every declared resource to its actual cloud ID, attributes, and dependencies. When you run a plan, the tool reads this file, compares it to the desired config, and queries the cloud API to check whether the state file's record is still accurate. The state file is the tool's model of reality.

This creates a new class of operational risk: the state file is now critical infrastructure. If it is lost, the tool has no record of which cloud resources it manages — future plans are blind. If it is corrupted, applies may attempt to recreate resources that already exist, or leave orphans. If it is not shared among teammates (stored locally on one person's laptop), the person who last ran apply is the de facto infrastructure owner. None of this is hypothetical.

⚠ State must be remote, locked, and backed up

The standard practice is remote state: store terraform.tfstate in a cloud bucket (S3, GCS) with versioning enabled and a DynamoDB/GCS lock table to prevent two engineers applying simultaneously. Without the lock, two concurrent applies read the same state, both compute "no conflict," both make API calls, and the state file ends up with exactly one apply's changes recorded and the other silently lost. This is called a state corruption race, and recovering from it ranges from tedious to catastrophic.

One subtlety worth internalizing: the state file sometimes drifts from reality even without console clicks. A cloud provider may migrate a resource to a new host and change its internal ID. A billing quota might prevent a resource from being created fully. A bug in the provider plugin might record the wrong attribute. Experienced teams run terraform apply -refresh-only periodically to reconcile the state file against live cloud APIs, separate from any apply (terraform refresh is deprecated since Terraform 0.15.3).

SEE ITIaC plan/apply simulator

TraceLab's infrastructure has three resources: a compute VM, a firewall rule, and a managed database. You'll make changes to the desired-state config, review the generated plan diff, apply it — and then a simulated teammate will click-ops the console to "fix" something. Your job is to spot the drift in the next plan and decide how to reconcile it without destroying production.

⚙ Interactive · TraceLab IaC plan/apply simulator

Edit desired state, review the plan, apply — then handle manual drift from the console.

Desired state (your .tf config)
Plan output
Run plan to see the diff →
Apply log
(nothing applied yet)

Phase 1: TraceLab has no cloud infrastructure yet. Review the plan before applying.

Three things to internalize from the simulation. First, the plan is your contract with the infrastructure — read every line, especially the destroy markers. Second, drift is invisible until the next plan reveals it — the console gave your teammate no warning that their change was untracked. Third, reconciling drift is a real decision with tradeoffs: reverting restores declared state but may undo an important workaround; importing accepts the change but requires updating the config and committing it; ignoring pushes the problem forward. None is automatically right.

EXAMPLEManual drift: the signature failure mode

Let's be precise about why manual drift is dangerous rather than just philosophically untidy. Here is a documented class of IaC incident:

  1. Team has a production Postgres database declared in Terraform. It's set to deletion_protection = true — the tool cannot delete the database even if told to.
  2. A developer needs to test something. They log into the console and disable deletion protection manually. They test. They forget to re-enable it.
  3. Two weeks later, someone refactors the Terraform config and mistakenly removes the database block. They run a plan. The plan says: - managed_db.runs_db will be destroyed.
  4. They approve, because the plan looked short and they were in a hurry. The database is deleted. The deletion protection flag that was supposed to save them was disabled in the console, untracked.

The root cause is not the final approval. It's the invisible console mutation in step 2. Every link in the safety chain was designed around the assumption that the state file and the declared config are the only sources of truth — and one console click silently violated that assumption.

⚠ "Read-only console" is a real policy

High-maturity infrastructure teams enforce IAM policies that give most engineers read-only access to the cloud console and require all writes to flow through the IaC pipeline with a PR and CI check. This isn't bureaucracy — it's the only reliable way to prevent drift in environments where multiple engineers share production access. TraceLab is a small platform and may not need this yet, but knowing what "production-grade IaC discipline" looks like helps you know when you're cutting corners.

There's a subtler version of drift that doesn't involve human error at all. Cloud providers regularly run automated maintenance: live migrations, storage rebalancing, security patches that modify resource metadata. Most of these are invisible to the state file. The resource still exists, behaves correctly, but its internal attributes (network interface IDs, storage volume IDs, host affinity metadata) may have changed. A plan against a stale state file will see this as drift and attempt to "fix" it — potentially triggering a destroy-and-recreate for an attribute you never intended to manage. The fix is to understand which attributes to mark as ignore_changes in your lifecycle config, and why.

TRACELABTraceLab v4 declared infrastructure

TraceLab v4 is the milestone this stage builds toward: containerized, CI-built, canary-deployed, and observed. The infrastructure it needs is now declared — not clicked. Here is the full resource set this lesson establishes for the remaining stage-5 lessons to build on:

infra/tracelab-v4.tfdeclarative — full resource set
# TraceLab v4 infrastructure — desired state, all resources declared.
# Run: terraform plan -out=tfplan && terraform show tfplan  (review)
#      terraform apply tfplan                                (commit)

terraform {
  backend "gcs" {
    bucket = "tracelab-tfstate"
    prefix = "v4"
  }
}

# ── compute ─────────────────────────────────────────────────────
resource "cloud_vm" "api" {
  name      = "tracelab-api"
  machine   = "e2-standard-2"     # 2 vCPU / 8 GB
  image     = "ubuntu-22-04-lts"
  zone      = "us-central1-a"
  disk_gb   = 40

  # Startup script — pulls the container image built by CI (Lesson 5.3)
  startup_script = file("scripts/startup.sh")

  lifecycle {
    ignore_changes = [startup_script]  # image tag managed by CI, not here
  }
}

resource "cloud_vm" "worker" {
  name    = "tracelab-worker"
  machine = "e2-medium"             # 1 vCPU / 4 GB — ingestion is async
  image   = "ubuntu-22-04-lts"
  zone    = "us-central1-a"
  disk_gb = 20
}

# ── database ────────────────────────────────────────────────────
resource "managed_db" "runs" {
  name               = "tracelab-runs-v4"
  tier               = "db-g1-small"
  version            = "POSTGRES_15"
  deletion_protection = true         # safeguard against accidental destroy
}

# ── networking ──────────────────────────────────────────────────
resource "cloud_firewall" "allow_api" {
  name    = "tracelab-allow-api"
  port    = 443
  source  = "0.0.0.0/0"             # public HTTPS — this is intentional
  vm_id   = cloud_vm.api.id
}

resource "cloud_firewall" "allow_worker_internal" {
  name    = "tracelab-allow-worker"
  port    = 8080
  source  = "10.0.0.0/8"            # internal only — ingestion not public
  vm_id   = cloud_vm.worker.id
}

Notice three design decisions that IaC makes explicit and enforced. First, deletion_protection = true on the database: this is a safeguard that must be explicitly removed before Terraform can destroy the database. It's the safety mechanism the incident above bypassed. Second, lifecycle { ignore_changes = [startup_script] } on the API VM: the container image tag changes on every CI build (Lesson 5.3), and letting Terraform manage that attribute would cause a noise-filled plan on every deploy. We declare "Terraform doesn't own this attribute." Third, the distinction between 0.0.0.0/0 on the API firewall and 10.0.0.0/8 on the worker: the intent is explicit in the code, reviewable, and auditable. In a console, these are just radio buttons.

💡 The remote backend block matters

The backend "gcs" block at the top stores the state file in a cloud bucket instead of terraform.tfstate on your laptop. Combined with bucket versioning and a lock table, this means two teammates can run plans safely, state is never lost with a laptop, and you can audit every state change. Set this up before you run your first real apply — retrofitting it later requires a state migration.

WEAKNESSWhere IaC bites

✗ The state file is not the source of truth for everything

IaC tools track the resources they manage. They have no record of the EBS snapshot your cloud provider took automatically, the support ticket that added a billing exception to your account, or the VPN gateway someone from the networking team created two years ago before you joined. Treating the state file as a complete map of your infrastructure is a slow-motion disaster — you will eventually run a plan that destroys a resource the tool never knew it was managing. Imports and explicit "data sources" exist to bring external reality into the declared model; use them aggressively when adopting IaC on an existing account.

✗ Slow feedback loops invite workarounds

A Terraform plan against a real cloud account can take 30–90 seconds. An apply that creates a managed database can take 10 minutes. Engineers under deadline pressure will start skipping the plan, merging config changes without CI checks, and — inevitably — making one-off console changes because "I don't have time to write the Terraform right now." IaC discipline erodes fastest when the tool is slow or painful. Investing in fast terraform plan (parallelism flags, targeted plans, mock providers in CI) is not gold-plating — it's the difference between "we use IaC" and "we used IaC for a while."

✗ IaC does not make you immune to cloud complexity

Every abstraction IaC provides has a floor below which you must understand the underlying cloud primitive. When Terraform says -/+ cloud_vm.api must be destroyed and recreated because you changed the machine type, you need to know whether that VM holds state, whether the IP address will change, whether there's a load balancer target group that needs updating. The declarative layer handles what; it cannot substitute for understanding what that means. The cloud economics lesson (5.5) and this one are designed to be read together.

KEY INSIGHTHow this connects

  • Cloud Primitives and the Bill (5.5) — you now know what you're declaring: compute, storage, networking, managed services, and what each costs per hour. IaC makes those primitives versioned and reviewable.
  • CI/CD Pipelines (5.3) — the pipeline that builds TraceLab's container image is what updates the VM startup script; the ignore_changes lifecycle block above is the seam between "infrastructure owners the IaC tool" and "CI owns the image tag."
  • Observability: Logs, Metrics, Traces (5.7) — the next lesson wires up TraceLab's observability stack. The Prometheus endpoint, the log shipper sidecar, and the tracing agent are all resources that will be declared in the same .tf files — infrastructure as code all the way down.

The stage-5 milestone — TraceLab v4 shipped and survived — requires every layer to be declared: the container, the pipeline, the infrastructure, and the observability. IaC is the layer that turns "it works on my machine" into "it works in production, and I have the git history to prove why."

GLOSSARYKey terms introduced

Desired state
A declarative description of what resources should exist and with what attributes. The IaC tool's job is to make reality match it.
Reconciliation
The process of comparing desired state to observed state and applying the minimal set of changes to close the gap.
Plan
A preview of the changes an IaC apply would make: shows adds (+), changes (~), and destroys (-/+) without executing them. Reading the plan is the primary safety gate.
State file
The IaC tool's record mapping declared resource names to their actual cloud IDs and attributes. Loss or corruption is serious; remote state with locking is standard practice.
Drift
Divergence between the state file (or declared config) and actual cloud reality, typically caused by manual console changes or provider-side maintenance.
Immutable attribute
A resource attribute the cloud provider cannot change in-place; updating it forces a destroy-and-recreate, which may cause downtime.
Remote state
State file stored in a shared cloud bucket (S3, GCS) rather than locally, enabling team collaboration and preventing concurrent-apply races.
lifecycle ignore_changes
An IaC directive telling the tool to ignore changes to specific attributes — used when another system (e.g., CI) manages those values, to prevent noisy plans.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What is the difference between an imperative infrastructure script and a declarative IaC config?›

An imperative script tells the system what to do — "create a VM," "add a firewall rule." It must handle "if it already exists, skip it" logic manually and can be run-once or idempotent depending on how carefully it's written. A declarative config tells the system what to be — "a VM with these properties must exist." The tool handles idempotency by design: it computes the diff between desired and current state and applies only the delta. Declarative is the right tool when you want version-controlled, reviewable infrastructure.

2Why does manual drift — a console change after an IaC apply — become a problem on the next plan?›

The state file records what the IaC tool last applied. A console change mutates the actual cloud resource but does not update the state file or the declared config. On the next plan, the tool compares the declared config (which still has the old value) against the state file (also old) and possibly against the live resource (which has the new value). It sees the console change as unsanctioned drift and generates a plan that will revert it — potentially destroying a production fix someone made at 2 a.m. The console gave no warning; the plan reveals it, but only if someone reads it carefully.

3What is a state file, why must it be stored remotely, and what happens if two engineers apply simultaneously without a lock?›

The state file is the IaC tool's map of declared resource names to real cloud IDs and attributes. It must be remote so all teammates share the same ground truth rather than each having their own (divergent) local copy. Without a lock, two concurrent applies both read the same state, both compute diffs, both make API calls, but only one of them writes back to the state file — the other's changes are silently absent. The state now incorrectly records the infrastructure, and future plans will attempt to create resources that already exist or miss ones that need changes.

4What does a destroy-and-recreate line in a plan mean, and when should you not approve it automatically?›

A destroy-and-recreate line (-/+) means an attribute you changed is immutable — it cannot be updated in-place. The tool will delete the existing resource and create a new one. You should not approve automatically when: the resource holds stateful data (a database — recreating it loses data); when the resource's IP or endpoint is referenced elsewhere and changing it would break dependent services; when there is no maintenance window and the destroy causes production downtime. The plan showed you this — reading it before typing yes is the entire safety guarantee.

Next: wiring TraceLab's logs, metrics, and traces →