The Operating System Contract
Your program is a tenant, not an owner — processes, threads, syscalls, signals, and exit codes are the lease terms.
🎯 By the end you'll understand
What a process is at the kernel level — its address space, open file descriptors, scheduling state, and PID. How processes come to life via fork and exec, how they die via exit, and what happens to their children when they don't. What signals actually are (and why kill -9 is not a debugging technique). Why environment variables are a one-way communication channel. And why every mysterious "it works on my machine" is ultimately a statement about OS state.
KEY INSIGHTYour program is a tenant, not an owner
Every process believes it owns the machine. The OS is the lie that makes that belief safe for everyone.
When you ran python tracelab.py log exp042 loss=0.31 in Lesson 1.1, we noted that the OS hands a fresh process a private virtual address space and schedules it onto a CPU. That sentence contains a semester of operating systems. Let's unpack it — not as theory, but as the contract your code signs every time it runs, with consequences you've already hit without knowing the cause.
A process is the OS's unit of a running program: a private virtual address space, a set of open file descriptors, a parent PID, a working directory, a table of environment variables, and some scheduled time on a CPU. Your code cannot touch memory, files, the network, or the clock without asking the kernel via a syscall. The kernel enforces the contract — it can revoke CPU time, kill the process, or refuse a syscall entirely. You are always the tenant. Understanding the lease terms is the difference between code that runs predictably everywhere and code that "works on my machine."
CORE CONCEPTWhat a process actually is
The kernel keeps a per-process table entry — think of it as a struct — with fields that represent everything it tracks about your running program. The important ones:
Process ID
A small positive integer the kernel assigns at birth. os.getpid() in Python, $$ in bash. Used by everything that refers to this process from outside: signals, wait(), ps, log files, crash reports. It's recycled when a process dies.
Virtual address space
The illusion that each process owns the entire address range. The kernel's MMU maps virtual addresses to physical RAM pages on demand. Two processes can have the same virtual address and refer to entirely different memory — which is exactly what happens after fork(). Pages are copied only when written (copy-on-write), which is why forking a large Python process is cheap until it writes.
File descriptor table
A small array indexed by integers: 0 is stdin, 1 is stdout, 2 is stderr — always, by POSIX convention. Higher numbers point to open files, sockets, pipes, or /dev/null. A file descriptor is not a filename; it's a kernel-managed reference to an open file description that tracks the current read/write position. Forked child processes inherit a copy of this table.
Environment variables
A flat key=value string store passed from parent to child at fork/exec time, accessible via os.environ. One-directional: a child can modify its own environment freely — it never touches the parent's copy. This is why export VAR=foo in a subprocess doesn't affect your shell.
Scheduling state
The kernel's scheduler tracks whether a process is running (on a CPU right now), runnable (ready to run, waiting for a free CPU), or sleeping (blocked on I/O or a timer). A "sleeping" process uses zero CPU — the scheduler doesn't give it time until the event it's waiting for fires. This is the entire mechanism behind async I/O, and it's why you should prefer it over threads for I/O-bound workloads. More in Stage 3.
Run python -c "import os; print(os.getpid(), os.getppid(), dict(list(os.environ.items())[:3]))". You'll see your PID, your parent PID (likely your shell), and the first three inherited environment variables. Everything visible there is data the kernel handed to this process when it was born.
HOW IT WORKSFork, exec, and exit: three syscalls that explain everything
Processes don't spring into existence from nothing. On Unix/Linux/macOS, the entire process tree is born from one ancestor — init or systemd (PID 1) — using just three syscalls repeated throughout a system's lifetime.
fork() — clone the current process
The kernel creates an almost-exact copy of the calling process. Both the parent and the child continue executing from the same point in code, but fork() returns different values: the child's PID to the parent, and 0 to the child. The child inherits the parent's address space (copy-on-write), file descriptor table, environment, and working directory. It gets a new PID and a fresh memory copy if it writes anything. Every Python multiprocessing worker starts this way.
exec() — replace this process's program
The exec family of syscalls replaces the current process's address space with a new program loaded from disk, keeping the same PID and file descriptors. This is what happens after fork() when you want to run a different program: the child forks, then execs the new binary. Your shell does this for every command you type. The pattern is so universal it has a name: the fork–exec idiom.
exit(code) — terminate and report
The process calls exit() with an integer code. The kernel freezes the process, releases most of its resources, and leaves behind a minimal record — the zombie state — until the parent calls wait() to collect the exit code. 0 means success (by convention); anything else signals failure. Unix pipelines and shell scripts depend on this convention; it's why sys.exit(1) is the correct way to signal script failure in Python.
The fork–exec idiom also explains why environment variables propagate the way they do. When your shell runs CUDA_VISIBLE_DEVICES=2 python train.py, it sets the variable in the forked child's environment before exec'ing Python. Python's os.environ inherits it. If you then fork a subprocess from Python, it inherits it again — but Python's environment changes after fork are invisible to the parent shell. No magic, no mystery: it's copy-on-fork.
SEE ITProcess lifecycle simulator
Below is a toy operating system with three slots for processes. Use the controls to spawn, fork, I/O-block, and signal them. Watch scheduling states change, file descriptors inherit and diverge, and exit codes propagate to the parent. Try creating a zombie by letting a child exit before you collect it.
A few things worth noticing as you step through. In the CPU-bound scenario, the process never voluntarily yields — the scheduler preempts it by force after its time-slice expires, which is why it's called preemptive multitasking. In the I/O scenario, the process blocks itself by issuing a syscall (read() on a file) and is parked immediately — no CPU wasted while waiting for disk. The orphan scenario shows what happens when a child outlives its parent: PID 1 adopts it (a real Linux/macOS guarantee), preventing true orphan zombies from accumulating. Watch the zombie appear at the moment the child calls exit() and disappear the moment the parent (or PID 1) calls wait().
CORE CONCEPTFile descriptors: everything is a file
Unix's most powerful unification is that processes communicate with almost everything through the same interface: an integer file descriptor backed by a kernel object that implements read() and write(). A file on disk, a socket, a pipe, a terminal, a /dev/null — all are addressable by integers 0, 1, 2, 3…
| FD number | Conventional meaning | What connects to it by default |
|---|---|---|
0 | stdin | keyboard / pipe from previous command |
1 | stdout | terminal / redirect to file or next pipe stage |
2 | stderr | terminal (separate from stdout — deliberately) |
3+ | application-opened resources | any open file, socket, pipe, or device |
When you write python tracelab.py log ... > out.txt, your shell opens out.txt before exec'ing Python and replaces FD 1 with the new file descriptor — a technique called dup2. Python wakes up and thinks stdout is the terminal; it's already been redirected. No cooperation required from Python.
The three-argument version of open() in TraceLab v0 is missing an encoding argument, but the more subtle issue is that it opens a new file descriptor and forgets to use a context manager. At the OS level, open() is the openat() syscall, which allocates a new entry in the FD table. If your process crashes before calling close(), the descriptor leaks — the file remains open, buffers may not flush, and on long-running servers you eventually hit the per-process limit (typically 1024 by default on Linux, 256 on macOS). Python's with open(...) calls close() in a __exit__ handler that runs even on exception. This is not a style preference; it's the correct use of the OS contract.
FD 2 exists so error messages don't corrupt data output. A shell script that parses python tool.py | grep result will fail silently if Python writes a crash traceback to stdout. Unix tools write status, warnings, and errors to stderr; stdout is for machine-readable data. Python's print(..., file=sys.stderr) and the logging module (which defaults to stderr) exist for exactly this reason. TraceLab v0 violates this: its find_run function prints to stdout whether it's data or an error.
CORE CONCEPTSignals and exit codes: the process communication layer
Once a process is running, how does anything talk to it from outside? Not through shared memory, not through files — through signals: small integer notifications that the kernel delivers asynchronously to a process, interrupting whatever it's doing.
| Signal | Number | Default action | What it means |
|---|---|---|---|
SIGINT | 2 | terminate | Ctrl+C in a terminal. Python raises KeyboardInterrupt from this. Catchable. |
SIGTERM | 15 | terminate | Polite shutdown request — what kill <pid> sends by default. Catchable: use it to flush buffers and clean up. |
SIGKILL | 9 | terminate (forced) | Cannot be caught, blocked, or ignored. The kernel kills the process instantly, with no cleanup. This is the problem with kill -9. |
SIGCHLD | 17 | ignored | Sent to a parent when a child exits. The correct trigger to call wait() and reap the zombie. |
SIGHUP | 1 | terminate | Terminal closed. Daemons re-read config on SIGHUP by convention — the "graceful reload" of web servers. |
SIGSEGV | 11 | terminate + core dump | Memory access violation. CPython rarely generates these; a C extension crashing will. |
In Python you catch signals with signal.signal(signal.SIGTERM, handler). A well-behaved server process handles SIGTERM to flush its database connections and write its last metrics before dying. TraceLab v0 handles no signals, meaning kill <pid> while it's mid-write can corrupt a row in runs.csv.
kill -9 (SIGKILL) is the operating system equivalent of pulling a power plug. It is not "a stronger kill" — it is a complete bypass of the process's ability to clean up. Any files left open have unflushed buffers. Any database transactions in flight are aborted. Any child processes become orphans. If you reach for kill -9 as a first response to a stuck process, you're saying "I don't understand why this process is stuck, and I'd rather destroy it than find out." The right first move is kill <pid> (SIGTERM). If that doesn't work within a few seconds, find out why — strace, lsof, or a stack trace — before escalating.
Exit codes are how processes communicate terminal status to their parents. By POSIX convention: 0 is success, non-zero is some form of failure. Shells capture this in $?, Python's subprocess module exposes it as returncode, and CI systems use it to distinguish pass from fail. Exit code interpretation is conventional, not enforced — but violating the convention (exiting 0 after an error) will silently break every pipeline downstream.
Run this snippet: the parent forks a child that prints its PID and exits with code 42, and the parent collects and prints that code.
import os, sys pid = os.fork() # fork returns twice if pid == 0: # we are the child print(f"child PID {os.getpid()} is running") sys.exit(42) # exit code 42 else: # we are the parent — pid is the child's PID _, status = os.waitpid(pid, 0) # block until child exits code = os.waitstatus_to_exitcode(status) print(f"parent collected exit code: {code}")
IN PRACTICEEnvironment variables: the one-way channel
Environment variables are the main mechanism for injecting configuration into a process at launch without modifying source code. They're inherited from parent to child, they're process-local after that, and they arrive before your code runs a single line.
Common patterns in real systems:
- Python virtual environments work entirely through
VIRTUAL_ENVand a prependedPATH: when you runsource venv/bin/activate, the shell modifies its own environment before exec'ing any Python. - GPU assignment in multi-GPU workstations uses
CUDA_VISIBLE_DEVICES. The CUDA runtime reads it at startup; changing it after import has no effect. This is the literal meaning of "it works on my machine, GPU 0 always has the right model weights" — the environment was different. - Secrets and credentials should live in environment variables or a secrets manager, not in source code or plain config files.
os.environ['API_KEY']is not great security, but it's vastly better than a hardcoded string in a repo.
Setting an environment variable in Python (os.environ['FOO'] = 'bar') affects only the current process and any children it spawns afterward. It cannot affect the parent shell — that copy was made at fork time and is gone. This is why subprocess.run(['export', 'FOO=bar']) has no effect on your script's environment: the subprocess's environment change is gone when the subprocess exits. The only way to propagate changes upward is to print them and have the parent shell eval the output — the pattern used by conda activate and direnv.
There is a deeper point here about "it works on my machine." Almost every case is one of three OS-state differences: the working directory (what paths are relative to), the environment variables (what configuration your code sees), or the installed software on the machine (what's on PATH, which shared libraries are present). Containers, which Stage 5 covers in depth, exist precisely to make these three things deterministic across machines. A Dockerfile is, in a real sense, a notarized copy of the OS contract your program requires.
WEAKNESSWhere this mental model bites
The single most common engineering failure mode in collaborative work. Your process saw a different PATH, a different working directory, a different version of a shared library, or an environment variable your colleague never set. None of these are visible in the code diff. Before debugging "why is this failing in CI?", run env | sort in both environments and compare. Before reporting a bug, ask: "what is my working directory, what is on my PATH, and what environment variables does this code read?" Most "mysterious" failures are answered in ten seconds this way.
A zombie is a process that has called exit() but whose parent has not yet called wait(). It consumes almost no resources — just a kernel table entry — but in a server that spawns many children without reaping them, you can exhaust the PID namespace, preventing any new processes from starting. Python's subprocess.Popen creates zombies if you never call .wait() or use it as a context manager. Always use subprocess.run() when you don't need the process handle, or with subprocess.Popen(...) as p: when you do. The multiprocessing module's daemon processes also become zombies if the parent exits without joining them.
A thread is a scheduling unit within a process: it shares the same virtual address space, file descriptor table, and heap as its siblings. This is why Python's GIL matters for CPU-bound threading (two threads cannot run Python bytecode in parallel), why a crash in one thread can corrupt memory accessed by another, and why multiprocessing (separate address spaces) is the Python answer for CPU parallelism. "Just add threads" is not a safe default — especially for a logger like TraceLab where concurrent writes to a single file will shear rows without a lock. Stage 3 will make all of this precise, but the foundation is here: the process boundary is also the safety boundary.
TRACELABTraceLab v0 and the OS
TraceLab v0 was introduced in Lesson 1.1 as a deliberately scruffy script. Now, with the OS contract in hand, we can be much more precise about which debts are OS-level and why they bite.
Apply this lesson's vocabulary to f.write(row + "\n") — the line we traced all the way to a syscall in Lesson 1.1. Here is what actually happens at the OS level:
open("runs.csv", "a") — two issues
This is the openat() syscall. It allocates a new FD in the process's FD table, positioned at end-of-file (append mode). Issue one: no encoding specified, so Python defaults to the locale encoding — which differs between macOS (UTF-8), many Linux containers (ASCII or UTF-8), and Windows (cp1252). Issue two: no context manager, so if an exception occurs before f.close(), the FD leaks and the write buffer may not flush.
f.write(row + "\n") — the locking gap
Python issues a write() syscall with the row bytes. O_APPEND on a regular file makes the kernel serialize the seek-to-end and write pair atomically within a single syscall — but Python buffers writes in userspace first. If two processes have the file open and both have buffered partial rows, the kernel-level atomicity doesn't help: the interleaving happens before the syscall. (Note: PIPE_BUF is a separate POSIX constant that applies only to pipes, not regular files.) The correct fix is fcntl.flock() or an advisory lock — or, better, a proper logging architecture that serializes writes through a single process.
No SIGTERM handler — silent corruption on shutdown
If you Ctrl+C or send SIGTERM while a write is in progress, the OS kills the process mid-write. The CSV row is left incomplete, often without the newline — a silent corruption that shows up only when find_run tries to parse the last line months later. A SIGTERM handler that flushes and closes the file before exiting costs five lines of code and makes the tool production-worthy.
None of these debts were visible in Lesson 1.1 because they require OS-level vocabulary to name. Now you can name them. The formal ledger in Lesson 1.9 will include all three; the concurrency debt is repaid in Stage 3.
import signal, sys _file = open("runs.csv", "a", encoding="utf-8") # encoding specified def _shutdown(sig, frame): print("signal received — flushing", file=sys.stderr) _file.flush() _file.close() sys.exit(0) # clean exit code — downstream scripts stay happy signal.signal(signal.SIGTERM, _shutdown) signal.signal(signal.SIGINT, _shutdown) # Ctrl+C also covered # the rest of tracelab.py follows…
This is still not production-grade — it still lacks locking — but it is a process that understands its own lease terms. The file descriptor is named, the encoding is specified, and the exit code will be 0 on clean shutdown and non-zero if the process crashes abnormally. Every CI pipeline and container orchestrator that wraps this script will thank you.
KEY INSIGHTHow this connects
- From Source to Execution (1.1) — we introduced the process as "the OS's unit of a running program" and traced
f.write()to a syscall; this lesson is the full explanation of both those claims. - The Shell Is a Programming Language (1.4) — the shell is the process that runs your commands; it uses fork, exec, pipes (file-descriptor plumbing), and environment variables as its primitives. Everything in the next lesson is a direct application of what you just learned.
The load-bearing nature of this lesson becomes clear in later stages: containers (Stage 5) are primarily OS namespace isolation — giving each container its own PID namespace, FD table, and environment, all concepts from today. Concurrency (Stage 3) is about processes and threads contending for shared OS resources, with locks as the protocol for orderly access. And zombie accumulation — PID-table exhaustion in a daemon that never calls wait() — has caused cascading production failures on real systems: a server that cannot fork new processes is effectively down, with no obvious error message pointing to the cause. Mechanism is not optional background knowledge. It is the only knowledge that holds up when something goes wrong.
GLOSSARYKey terms introduced
- Process
- The OS's unit of a running program: a virtual address space, FD table, environment, PID, parent PID, and a scheduled state. Isolated from other processes by the kernel's MMU.
- Fork
- The
fork()syscall that creates a near-exact copy of the calling process. Child and parent continue from the same point; the child gets PID 0 as the return value and a new PID in the kernel table. - Exec
- The
exec()family of syscalls that replaces the current process's address space with a new program, keeping the same PID and FD table. Paired with fork in the fork–exec idiom. - File descriptor (FD)
- A small integer indexing the process's open-resource table. 0/1/2 are stdin/stdout/stderr by convention. The uniform interface behind files, sockets, pipes, and devices.
- Signal
- An asynchronous integer notification delivered by the kernel to a process. Catchable signals (SIGTERM, SIGINT) allow graceful shutdown; SIGKILL cannot be caught or ignored.
- Exit code
- The integer a process passes to
exit(). By POSIX convention, 0 is success, non-zero is failure. Collected by the parent viawait(); drives shell conditionals and CI pass/fail. - Zombie process
- A process that has exited but whose exit code has not been collected by its parent via
wait(). Holds a kernel table entry; accumulates if parents don't reap children. - Syscall
- A request from a process to the kernel to perform a privileged operation (I/O, memory mapping, process creation). The only door between user-space code and hardware.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What is the difference between a process and a program?›
A program is an inert file on disk — bytecode, machine code, or a script. A process is a running instance: the OS has loaded it into a private virtual address space, given it a PID and a set of open file descriptors, and the scheduler is giving it CPU time. Multiple processes can run the same program simultaneously (e.g., two terminal windows both running python) — they share the program but each has a completely independent process state.
2Why does kill -9 sometimes make things worse rather than better?›
SIGKILL cannot be caught, blocked, or handled. The kernel kills the process instantly, with no opportunity for the process to flush file buffers, close database transactions, write a final log entry, or notify its children. This can leave files corrupt, databases in an inconsistent state, and child processes orphaned. SIGTERM (the default kill) is the correct first move: it asks the process to clean up and exit. If the process doesn't respond, that's a bug worth diagnosing — not bypassing.
3You set os.environ['MODEL_PATH'] = '/new/path' in Python, then launch a subprocess. What does the subprocess see? What does your parent shell see?›
The subprocess inherits its environment from the parent process at fork time. Since you modified os.environ in Python before spawning the subprocess, the subprocess sees the new value /new/path. The parent shell sees nothing — its environment copy was made at the moment it forked Python, and Python's subsequent modifications to its own environment never propagated back. Environment variables flow one way: parent to child at fork/exec. They cannot flow backward.
4What is a zombie process and why does it matter for a server that spawns many short-lived children?›
A zombie is a process that has called exit() but whose parent hasn't called wait() to collect the exit code. It's nearly free in resources — just a PID and a kernel table entry — but PIDs are finite (on Linux, the default maximum is 32,768). A server that spawns thousands of short-lived workers without calling wait() (or using a signal handler for SIGCHLD) will fill the PID table, making it impossible to create new processes. The fix: always call .wait() on Popen objects, use subprocess.run(), or handle SIGCHLD to reap children asynchronously.