Orchestration: Kubernetes and Why TraceLab Says No
What the scheduler/reconciler model genuinely solves, what it costs, and the honest decision tree that concludes TraceLab shouldn't use it.
🎯 By the end you'll understand
How Kubernetes' scheduler/reconciler model works as a mechanism — declarative desired state, bin-packing, self-healing, autoscaling — and what it genuinely costs: operational complexity, YAML archaeology, and the platform team you probably don't have. You'll run a scheduler simulator, compare operational overhead between Kubernetes and plain VMs with real numbers, and understand why TraceLab stays on plain VMs — not because the tool is bad, but because judgment is mostly the discipline of not using things.
KEY INSIGHTKubernetes solves a real problem you might not have
Teaching a technology and then declining to use it is the rarest move in a course — because most courses are written by people who want you to use the thing they're teaching.
Kubernetes is genuinely impressive engineering. The scheduler/reconciler model is one of the cleaner ideas in distributed systems. The correctness guarantees — bin-packing, self-healing, declarative desired state — are real. And if you're running thousands of heterogeneous workloads across hundreds of nodes for a company with a platform team, Kubernetes is the right answer.
TraceLab is not that. It is one API server, one ingestion worker, one Postgres instance, and one background job — three replicas of the hot path, tops. The honest engineering question is not "does Kubernetes have features?" but "does the value of those features exceed the cost of the platform for this workload?" This lesson answers that question numerically. The answer is no. Teaching you to reach this conclusion deliberately — and defend it — is the whole point.
The discipline of saying no to capable technology is harder than saying yes. Yes is free. No requires you to understand the technology well enough to price its costs, map those costs against your actual constraints, and stand behind the conclusion. You cannot decline what you haven't understood — which is why we teach Kubernetes fully before refusing it.
HOW IT WORKSThe scheduler/reconciler model
Kubernetes is a container orchestration platform: you tell it what you want (desired state), and it figures out where to run it and keeps it there. The architecture has three conceptual layers worth understanding precisely.
Declarative desired state
You write a manifest — a YAML or JSON document saying "I want 3 replicas of this container image, each needing 500m CPU and 256Mi memory, with this health check, reachable on port 8000." The cluster figures out how to make that true and keeps it true. You never say where; you say what. This is the generalization of the Infrastructure-as-Code idea from Lesson 5.6 — but now applied to live running workloads, not just provisioning.
The scheduler bins-packs pods onto nodes
The scheduler watches for unscheduled pods (the smallest deployable unit — one or more containers sharing a network namespace and storage). For each unscheduled pod it filters nodes by resource availability and constraints, then scores the survivors by bin-packing efficiency — it tries to pack workloads densely to leave whole nodes free, rather than spreading them thinly. The winning node gets the pod. This happens automatically, continuously, without human input.
Controllers reconcile desired vs. actual
The heart of Kubernetes is a fleet of controllers, each a tight loop: observe the world, compare actual state to desired state, act to reduce the diff, repeat. The ReplicaSet controller watches pod counts; if a node dies and takes two pods with it, the controller notices (actual: 1 pod, desired: 3 pods) and schedules two new ones. This is self-healing. It requires no operator action; it happens in seconds.
Autoscaling watches metrics and adjusts replicas
The Horizontal Pod Autoscaler (HPA) watches a metric — usually CPU utilization or a custom metric like queue depth — and adjusts the replica count on your Deployment automatically. If TraceLab's ingestion worker hits 80% CPU, HPA scales from 2 to 4 replicas without a human. When load drops, it scales back. Combine this with a Cluster Autoscaler and the node pool itself grows and shrinks — you pay for what you use, on demand.
The key insight in the diagram: the control plane components never talk to each other directly. Everything flows through the API server, with state persisted in etcd — a distributed key-value store with linearizable reads. This design makes the control plane remarkably composable (any controller just watches the API server) and remarkably debuggable (all state is in one place, versioned, query-able). It also means that when etcd misbehaves, you have a very bad day.
CORE CONCEPTSelf-healing is a reconciler, not magic
The word "self-healing" sounds like magic until you see the loop. Every Kubernetes controller follows the same pattern — the watch-compare-act loop:
def replica_set_controller(): while True: desired = get_desired_replicas(rs_spec) # from the spec you wrote actual = count_running_pods(selector) # from the API server diff = desired - actual if diff > 0: for _ in range(diff): schedule_new_pod(template) # scheduler finds a node elif diff < 0: for pod in pick_excess_pods(-diff): delete_pod(pod) # scale down gracefully sleep_until_api_event() # event-driven, not a busy-loop
That's it. Self-healing is not magic; it's a controller noticing that actual ≠ desired and scheduling new pods. The power is that this loop runs continuously — a node dies at 3am, and within seconds (typically 30-60 depending on health-check configuration) the controller has scheduled replacements on surviving nodes. No pager alert needed for this specific failure mode.
The same pattern drives rolling deployments. You push a new image version; the Deployment controller replaces pods one at a time (or N at a time — configurable), always keeping enough healthy pods to serve traffic, always checking readiness probes before advancing. If the new version fails its readiness check, the rollout stops. Human intervention only if the failure itself is ambiguous.
In Lesson 5.6, Terraform's declarative infrastructure worked the same way: desired state in a file, reconcile against actual cloud state. Kubernetes applies the same idea one layer down — to live running processes. Once you understand one reconciler loop, you understand the pattern; the details are just the domain (infrastructure vs. pods).
SEE ITScheduler simulator + VM comparison
Declare resource requests for TraceLab's three services, watch the scheduler bin-pack them onto nodes, then kill a node and observe rescheduling. The second tab compares the same workload on plain VMs to put the operational cost difference in concrete numbers.
Notice what happens when Node B dies: the scheduler sees the diff (desired 4, actual 2) and reschedules immediately. This is genuinely powerful — especially at 2am. But notice also what the VM comparison shows: the orchestration layer itself runs on resources, manages its own components, and is the thing that can break. The cost of self-healing is a more complex system that has more parts to fail.
WEAKNESSWhat Kubernetes actually costs
The feature list is real. The costs are less often taught honestly. Here are the ones that actually affect small teams.
Running a production Kubernetes cluster means maintaining etcd (a distributed consensus system), the control plane API server, the scheduler, the controller manager, kube-proxy on every node, a CNI plugin (networking), a CSI plugin (storage), cert-manager, an ingress controller, a metrics server, and — at minimum — basic RBAC policy. Every one of these has upgrade cycles, failure modes, and CVE streams. A managed offering (GKE, EKS, AKS) offloads some of this — and charges a flat cluster fee ($0.10/hour on EKS, roughly $73/month) before you run a single container. On a $72/month VM workload, that's already a 100% overhead.
A minimal production Kubernetes deployment for TraceLab's three services requires: Deployments (one each), Services, an Ingress, a HorizontalPodAutoscaler, PodDisruptionBudgets, ResourceQuotas, NetworkPolicies, ConfigMaps, Secrets, ServiceAccounts, and RBAC Role/RoleBindings. Many of these reference each other with string names that nothing validates until runtime. The "did I break something" loop is: kubectl apply, wait, watch pods, read events. When it breaks silently — a selector label mismatch, an HPA pointing at a non-existent metric — the failure mode is "pods deploy fine but autoscaling never fires," which you may not notice for weeks.
When something breaks on a VM you know what questions to ask: what process is running, what did journalctl -u tracelab say, does the port respond. When something breaks on Kubernetes you need to know: which namespace, which pod, which container in the pod, is it a scheduling issue or a runtime issue, is it a probes-failing issue or an OOM issue, is the liveness probe killing healthy pods, are the resource limits too tight. kubectl describe pod is helpful but the skill of reading it fluently takes months. The system that eliminates human intervention for node failures adds human intervention overhead for everything else.
TRACELABTraceLab v4 stays on plain VMs — with a defense
TraceLab's v4 milestone (the Stage 5 goal) is containerized, CI-built, canary-deployed, and instrumented. It runs on two application VMs and one database VM, managed by Ansible, deployed by the CI pipeline from Lesson 5.5. Here is the architecture that the Kubernetes decision tree declined:
# TraceLab v4 production topology (plain VMs, no orchestrator) # # app-1 (t3.small, $16.50/mo) app-2 (t3.small, $16.50/mo) # └─ tracelab-api (systemd) └─ tracelab-api (systemd) # └─ tracelab-worker (systemd) └─ tracelab-worker (systemd) # # db-1 (t3.medium, $38.50/mo) # └─ postgres 16 (systemd) # └─ wal-g → S3 backups (cron) # # nginx load balancer (t3.micro, $8.50/mo) # # Total: $80/mo Ansible playbook: 3 roles, ~120 lines # Deployments: CI pipeline → ansible-pull on each node # Self-healing: systemd Restart=always recovers process crashes # Scaling: add a VM, update inventory, re-run playbook (15 min) # What we're explicitly NOT doing and why: # # - Kubernetes: orchestration overhead exceeds value for 4 services # - ECS: vendor lock-in, Fargate cost premium, still needs IAM/task-def YAML # - Docker Swarm: simpler than K8s but still a distributed system; not worth # the complexity for a workload that needs to scale at most 2× per year # # Revisit trigger: fleet >50 robots AND >10 distinct service types AND # team has someone willing to own the platform layer
The comment at the bottom is the most important engineering artifact in this file: the revisit trigger. "We're not using Kubernetes" is not a permanent verdict — it's a verdict given today's constraints. Writing down the exact conditions under which you'd revisit the decision means future-you (and your team) can update it without relitigating the entire argument from scratch. This is what engineering judgment looks like when written down: a decision, a defense, and a trigger.
Restart=always in a systemd unit file gives you process-level self-healing with zero orchestration overhead. The tracelab-api service restarts within 2 seconds of a crash. Combined with a load balancer health check that pulls the VM from rotation during downtime, you get meaningful resilience — not as comprehensive as Kubernetes' node-level healing, but sufficient for a workload where a rolling VM failure is a once-per-year event.
IN PRACTICEThe honest decision tree
The question "should I use Kubernetes?" deserves a framework, not an opinion. Here is the one that produces defensible answers:
| Question | If yes → | If no → |
|---|---|---|
| Do you have >20 distinct service types that need independent scaling? | Kubernetes is a serious candidate. | A scheduler is solving a bin-packing problem you don't have. Continue. |
| Does your scale vary by >10× within a day, and is the cost of over-provisioning significant? | HPA + cluster autoscaler's value is real. | Autoscaling's value is limited. Continue. |
| Do you have at least one engineer willing to own the platform (upgrades, CVEs, etcd, networking)? | Proceed to next question. | Stop. Managed offerings (GKE Autopilot) reduce but don't eliminate this requirement. |
| Is the monthly cluster overhead (managed: ~$73-100/mo, self-managed: 2 dedicated nodes + labor) <20% of your compute bill? | The cost is proportionate. | The orchestration layer costs more than the work it's doing. Use simpler tooling. |
| Does your team already know Kubernetes, or is there a business case to learn it? | Proceed — skill investment is amortized. | The learning curve is real (months to production competence). Budget for it explicitly. |
TraceLab answers "no" to the first two questions and "no" to the cost test (cluster overhead would be 100% of the compute bill). The verdict is plain VMs. If TraceLab's fleet reaches 200 robots with heterogeneous sensor configurations requiring different compute profiles, the answer to question 1 changes, and the decision reverses.
Kubernetes earns its complexity for workloads where: (a) you have many heterogeneous services with genuinely different resource profiles and scaling behavior; (b) your utilization is uneven enough that bin-packing meaningfully reduces your node count; (c) you deploy frequently enough that rolling deployments and instant rollbacks pay for themselves; (d) you have or are building a platform team. The canonical examples — large CI/CD platforms, ML training infrastructure, microservice-heavy SaaS products — are real. TraceLab is not one of them.
There is a second failure mode worth naming explicitly: learning Kubernetes for TraceLab in order to put it on a resume. This is a completely legitimate goal — and it has nothing to do with the engineering decision. If you want to learn Kubernetes, run a local cluster (kind, k3s, minikube), deploy a toy workload, and build the skill deliberately. Don't use TraceLab's production budget to buy the education. Separate the learning goal from the deployment decision.
WEAKNESSWhere this reasoning bites
The argument "start on VMs and migrate to Kubernetes when we need it" sounds pragmatic. The trap: by the time you need Kubernetes, you have stateful services with hand-rolled deployment scripts, networking assumptions baked into application code, and an operations team who's never touched kubectl. The migration cost at that point is enormous — far more than if you'd started on Kubernetes or chosen a managed abstraction (ECS, Cloud Run, Fly.io) that would have migrated more cleanly. If your decision is "VMs for now," make sure your architecture doesn't accumulate migration-hostile assumptions. The TraceLab playbook approach is deliberately clean: all config is external, services are containerized, network topology is documented.
A common failure mode: small team adopts Kubernetes because it's the "modern" choice, onboards two engineers who learn the basics, then one leaves. The cluster is now operated by people who can run kubectl apply but can't debug a networking issue, upgrade the control plane safely, or investigate an etcd performance problem. Kubernetes operational complexity has a step function: the skills required to run it safely at production are significantly higher than the skills required to deploy to it. Teams that underestimate this gap end up with brittle infrastructure they're afraid to touch.
GKE, EKS, and AKS handle the control plane upgrade lifecycle — but you still own node upgrades, RBAC policy, network policy, admission controllers, and the full configuration surface. The 2019 AWS post-mortem on the Capital One breach — which involved misconfigured IAM roles on EC2 instances hosting containers — illustrates that managed services don't protect you from misconfiguration at the workload layer. Managed Kubernetes reduces operational overhead; it does not reduce security surface area.
KEY INSIGHTHow this connects
- Containers (5.2) — Kubernetes orchestrates containers; the image format and layer model from 5.2 are what it schedules. Your pod spec references an image digest, not a VM snapshot.
- Infrastructure as Code (5.6) — Kubernetes extends IaC's declarative model from provisioning infrastructure to running workloads. The reconciler loop is the same pattern as Terraform's plan/apply, applied continuously and in-process rather than on-demand and out-of-band.
- Distribution fundamentals (4.2) — Kubernetes' etcd uses the Raft consensus algorithm. The failure modes you saw in 4.2 — leader election, split-brain, quorum loss — apply directly to the control plane. A 3-node etcd cluster requires 2 nodes to be available; losing 2 of 3 control-plane nodes makes the cluster read-only until you restore quorum.
- Next: The Outage (5.10) — TraceLab v4 ships on VMs. The Stage 5 final exam is a scripted ingestion outage you command on that platform. The decisions made here — what to run, on what, with what observability — are the ones you'll defend under fire.
The meta-lesson of this stage's arc: the artifact comes before the pipeline; the primitives before the code that declares them; judgment last. Judgment requires knowing what you're judging. You now know Kubernetes well enough to decline it with a clean conscience.
GLOSSARYKey terms
- Pod
- The smallest schedulable unit in Kubernetes — one or more containers sharing a network namespace and a set of mounted volumes. Usually one container per pod for most workloads.
- Desired state / Actual state
- The core abstraction: you declare what you want (desired), the cluster observes what exists (actual), and controllers close the gap. The diff drives all autonomous actions.
- Scheduler
- The control-plane component that assigns unscheduled pods to nodes, using resource requests and constraints to filter candidates and bin-packing heuristics to score them.
- Controller / Reconciler loop
- A continuous watch-compare-act loop that maintains one aspect of desired state. The ReplicaSet controller watches pod counts; the Deployment controller manages rolling updates; HPA watches metrics.
- etcd
- The distributed key-value store that persists all Kubernetes state. Uses the Raft consensus algorithm. Its health is the control plane's health — etcd quorum loss makes the cluster read-only.
- HPA (Horizontal Pod Autoscaler)
- A controller that adjusts replica count based on observed metrics (CPU, memory, or custom). Scales up when utilization exceeds a target; scales down (with a cooldown delay) when load drops.
- kubelet
- The agent that runs on every worker node, watches the API server for pods assigned to its node, starts/stops containers via the container runtime, and reports node and pod health back to the API server.
- Bin-packing
- The scheduling strategy of placing workloads densely on nodes — preferring to fill some nodes fully rather than spreading load thinly — to leave whole nodes free for scale-down or workload isolation.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What is the watch-compare-act loop, and which Kubernetes component uses it to implement self-healing?›
The watch-compare-act loop is the pattern where a controller continuously observes the current state of the world, compares it to desired state, and takes action to close the gap. The ReplicaSet controller (part of the controller manager) uses this to implement self-healing: when a node dies and pods disappear, the controller notices that actual replica count is less than desired, and schedules replacement pods on surviving nodes. The loop runs continuously — no human action required.
2What does the Kubernetes scheduler actually do, and what is bin-packing?›
The scheduler watches for unscheduled pods and assigns each one to a worker node. It first filters nodes based on resource availability and constraints (taints, affinity rules, resource requests), then scores the survivors — preferring to pack workloads densely onto fewer nodes rather than spreading them thinly. This density preference is bin-packing: by filling some nodes fully, it leaves other nodes empty and available for scale-down or new workloads, improving utilization while preserving headroom.
3Why does TraceLab v4 stay on plain VMs rather than Kubernetes, and what conditions would reverse that decision?›
TraceLab has only four service types (API, ingestion worker, Postgres, nginx) — well below the threshold where bin-packing meaningfully reduces node count. Load is stable enough that autoscaling saves negligible cost. The managed cluster overhead (~$73/month) would equal or exceed the compute bill. And there's no platform engineer to own upgrades and failures. The decision reverses if the fleet exceeds ~50 robots requiring heterogeneous compute profiles, the team grows to include a platform role, and daily load variance exceeds 10× (making autoscaling cost-effective).
4What is etcd, and why does its health matter so much to a Kubernetes cluster?›
etcd is a distributed key-value store that persists all Kubernetes state — every object definition, every status update, every config — using the Raft consensus algorithm. The API server is stateless; it reads and writes to etcd. If etcd loses quorum (more than half its members fail), the API server can still serve reads from its cache but can no longer accept writes — no new scheduling decisions, no new config changes, no scale events. This makes etcd the single most critical component in the control plane; losing it means the cluster freezes in its last known state.