ProjekinAja. Software Engineering Stage 3 · Systems Lesson 3.3
● Stage 3 · Lesson 3.3

Inside the Server: The Request Lifecycle

What FastAPI actually does between the socket and your function — sockets, routing, middleware, workers, and where the time hides.

⏱ ~18 min read 🧠 Mechanism 🔥 Failure modes

🎯 By the end you'll understand

How a TCP connection becomes a parsed request and eventually a serialized response — the complete socket → router → middleware → validation → handler → serializer → response stack. What WSGI and ASGI actually mean and why the distinction matters for concurrency. How worker models scale (and bottleneck). Where global state hides in middleware. Why one slow middleware taxes every request. What determines whether an unhandled exception leaks a stack trace to the client or returns a clean 500.

KEY INSIGHTYour handler is the last thing called, not the first

You wrote a function. Six layers of framework machinery run before it touches your code — and six more run after. Most of what makes a server fast, safe, or broken lives in the layers you didn't write.

When you add a @app.post("/runs") route to TraceLab's FastAPI server, it feels like you wrote the server. You didn't. You wrote a handler. The framework stitches it into a pipeline that manages sockets, parses HTTP, runs authentication middleware, validates the request body against your Pydantic schema, catches unhandled exceptions, serializes your return value to JSON, and writes the response to the socket — and all of that happens whether or not you thought about it. Understanding that pipeline is the difference between debugging in production and debugging in development.

🔑 Hold onto this

A web framework is a pipeline constructor, not a router. Your route handler sits at the center of that pipeline; middleware, validation, and serialization wrap it on both sides. Every millisecond of latency and every unhandled exception propagates through the entire stack. When something breaks, the answer to "where?" is almost always "somewhere in the pipeline you trusted but never traced."

CORE CONCEPTWSGI and ASGI: the contract between server and framework

Python web stacks are split into two independent pieces: a server that speaks TCP and HTTP (Gunicorn, Uvicorn, uWSGI), and a framework that builds responses (Flask, Django, FastAPI, Starlette). The interface between them is a spec.

WSGI (Web Server Gateway Interface, PEP 3333) is the original Python web interface. A WSGI app is a plain callable: it receives an environment dict and a start_response callback and returns an iterable of bytes. The server calls it synchronously — one thread per request. For the server to handle a new request while the previous one waits for the database, it must have another thread ready. That's the classic thread-per-request model.

wsgi_minimum.pythe whole contract
# Any WSGI-compliant application must be this shape.
# Flask, Django (in WSGI mode) are both just this, with extras.
def application(environ, start_response):
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [b"hello"]

ASGI (Asynchronous Server Gateway Interface) generalises WSGI for async Python. An ASGI app is a three-layer callable: it receives scope, receive, and send coroutines. The server calls it with await. This single change is what allows a single OS thread to interleave thousands of in-flight requests by suspending at every await — no thread pool required.

asgi_minimum.pythree callables
# scope: what kind of connection this is (http, websocket, lifespan)
# receive: coroutine to read the next event from the client
# send: coroutine to send an event to the client
async def application(scope, receive, send):
    await send({"type": "http.response.start",
                "status": 200,
                "headers": [(b"content-type", b"text/plain")]})
    await send({"type": "http.response.body",
                "body": b"hello"})

FastAPI is built on Starlette, which is an ASGI framework. The server that runs it is Uvicorn — an ASGI server based on asyncio and uvloop. Every async def route you write hands control back to the event loop while it waits for I/O, allowing other requests to proceed. Every def (sync) route FastAPI runs in a thread pool so it doesn't block the loop.

📖 Why this distinction matters for TraceLab

TraceLab's API will have an endpoint that persists a telemetry log to Postgres. If that's a sync function, it blocks the entire event loop for its duration — one concurrent write grinds every other request to a halt. Writing it as async def with await db.execute() lets hundreds of writes interleave at the I/O suspension points. Stage 3's concurrency lesson (3-7) will push this to its limits.

HOW IT WORKSThe eight-layer pipeline: socket to response

Here is what actually happens between a TCP packet arriving and your handler's return value leaving the machine. Each layer has a real cost and a real failure mode.

1

Socket accept — the OS layer

The server calls accept() on a listening socket. The OS hands over a connected socket file descriptor. This is a syscall; if the server's event loop or accept queue is saturated, the client waits here. Uvicorn's event loop calls accept() non-blocking; Gunicorn's worker processes block in their own accept().

2

HTTP parse — bytes → structured request

The server reads bytes and parses them into an HTTP message: method, path, headers, body. Uvicorn uses httptools (a Cython wrapper around Node.js's battle-tested http_parser). A malformed request is rejected here — the framework never sees it. Content-Length or chunked encoding determines when the body is fully received.

3

Router — path + method → handler candidate

The ASGI app (Starlette/FastAPI) receives the scope dict and matches the path against its route table. Starlette builds a compiled regex from your path templates. A miss returns a 404. A match with wrong HTTP method returns a 405. The route match also extracts path parameters (/runs/{run_id} → run_id="exp042").

4

Middleware stack — inbound pass

Middleware wraps the handler in an onion. Every registered middleware runs on every request, inbound and outbound. Typical middleware: CORS headers, request ID injection, authentication token validation, rate limiting, request logging. Middleware can short-circuit (return a response immediately, e.g., 401) or pass through. Order matters: add an outer middleware after an inner one and it runs before it.

5

Dependency injection — FastAPI's DI system

FastAPI resolves the handler's Depends() declarations: DB sessions, auth principals, config objects. Dependencies run in declaration order; failed dependencies short-circuit with their own exceptions. The resolved values are injected as function arguments. This is pure FastAPI — Starlette has no equivalent.

6

Request validation — Pydantic parses the body

FastAPI calls Pydantic to coerce and validate the request body against your declared model. Type errors, missing required fields, constraint violations all become 422 Unprocessable Entity responses automatically — FastAPI generates the error detail JSON for you. The handler never sees a malformed input; validation is the gate.

7

Handler — your code, finally

Your function runs. It should be the thinnest thing in the pipeline: receive a validated model, do business logic, return a value. If it raises an unhandled exception, it propagates up through the middleware stack to the server's exception handler — which is where the security-critical decision of "what does the client see?" is made.

8

Serializer → response → socket

FastAPI runs the return value through jsonable_encoder (converts dataclasses, datetime, UUID, etc.) then through json.dumps. The bytes are wrapped in an HTTP response and written back to the socket via send(). Middleware runs again on the outbound pass — adding response headers, logging the final status code, etc.

socket OS accept() HTTP parse bytes router path match middleware (onion) validate middleware (cont.) handler your code serialize JSON bytes response HTTP framing socket send() your code lives here everything else runs whether you thought about it or not middleware onion — runs inbound AND outbound
The full request pipeline. The middleware layer is an onion: each middleware wraps everything to its right. On the inbound pass it runs top-to-bottom; on the outbound pass it unwinds in reverse order. Your handler is surrounded on both sides.

SEE ITTrace a request through the TraceLab backend

Every request hides a time budget. Pick a scenario, inject a fault or a slow middleware, then step through all eight layers to see exactly where the milliseconds go and where exceptions surface.

⚙ Interactive · Request lifecycle step-runner

Step a request through the TraceLab backend layer by layer. Inject a slow middleware or an unhandled exception to see exactly what changes.

Layer
—
Elapsed (simulated)
0 ms
Response
—
Client sees
—

TCP connection accepted. Bytes are sitting in the socket buffer — nothing has been parsed yet.

Two things to notice. In the "slow auth middleware" scenario, every other request hitting the server pays that cost — not just the authenticated ones. The middleware runs unconditionally. In the "unhandled exception" scenario, the exception propagates up through the middleware stack; only the outermost exception handler (the server's 500 handler) decides whether the client sees a stack trace. If that handler is the default one, your secrets might be in the client's browser.

CORE CONCEPTWorker models: where concurrency happens (and hides)

Your framework pipeline explains one request. The worker model explains how a server handles many. There are three mainstream approaches, each with a precise tradeoff.

ModelToolConcurrency unitGood forBottleneck
Multi-process Gunicorn (prefork) OS processes, typically 2×CPU+1 CPU-bound work; isolation (one crash stays in one process) Memory (each process is a full Python runtime); process startup; shared nothing means shared-cache is hard
Multi-thread Gunicorn (sync or gthread) OS threads inside a process I/O-bound WSGI apps; keeps shared in-process state CPython GIL limits true CPU parallelism; shared mutable state; a deadlock kills the entire worker
Async event loop Uvicorn + asyncio Coroutines on one thread High-concurrency I/O-bound work (many simultaneous DB/HTTP calls) One blocking call (a slow sync library, time.sleep()) freezes all concurrent requests; requires the whole call stack to be async-aware

In practice, TraceLab's recommended deployment is Uvicorn workers under Gunicorn: multiple Gunicorn processes each running a Uvicorn event loop. This combines process isolation (one loop crash doesn't kill the server) with async concurrency (each loop handles thousands of simultaneous requests). gunicorn app:app -w 4 -k uvicorn.workers.UvicornWorker is five words of config that compress a decade of lessons about scaling Python web services.

💡 Worker count rule of thumb

The classic rule is 2 × CPU cores + 1 for synchronous workers (enough to keep CPUs busy while some workers wait for I/O). For async workers, fewer is fine — 1–2 per CPU — because each event loop handles many concurrent requests. The limit for async is the event loop's ability to get CPU time, not the number of simultaneous I/O waits. Starting with 2 Uvicorn workers per CPU core is reasonable; let metrics tell you from there.

4 Gunicorn processes × 1 Uvicorn event loop each worker 1 event loop req A await await req A runs while B and C wait for DB 1 OS thread, many concurrent requests worker 2 event loop req D req E await worker 3 event loop await await req F worker 4 event loop req G await await
Four Uvicorn workers (OS processes) each running an asyncio event loop. Green = currently executing; grey = suspended at an await point (I/O in flight). One OS thread handles many concurrent requests by suspending rather than blocking. A single blocking call in any coroutine stalls the entire loop — its requests and all the ones waiting behind them.

IN PRACTICEMiddleware: the power and the trap

Middleware is where cross-cutting concerns live: authentication, logging, CORS, rate limiting, request tracing. FastAPI inherits Starlette's middleware system; you add it with app.add_middleware() or as an ASGI middleware class. The stack unwinds like a call stack — the first middleware you add is the outermost layer, running first on the inbound pass and last on the outbound pass.

tracelab/middleware.pywhat a middleware actually is
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
import time, uuid

class RequestIDMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        request_id = str(uuid.uuid4())
        # inbound: attach an ID before the handler sees the request
        request.state.request_id = request_id
        t0 = time.perf_counter()

        response = await call_next(request)   # ← everything inside runs here

        # outbound: add header AFTER the handler has run
        elapsed = (time.perf_counter() - t0) * 1000
        response.headers["X-Request-ID"] = request_id
        response.headers["X-Duration-Ms"] = f"{elapsed:.1f}"
        return response

This pattern is clean, but notice: call_next is an awaitable that runs the entire rest of the pipeline. Anything you do before that call runs on the inbound pass; anything after runs on the outbound pass. The middleware sees both the request and the response — which is exactly why a bug in middleware can corrupt both.

⚠ Global state in middleware

Because all requests share the same middleware instance, any mutable attribute on self is shared across concurrent requests. Store per-request data in request.state, not on the middleware class. The mistake looks like this:

bug: shared statedangerous
class BrokenMiddleware(BaseHTTPMiddleware):
    def __init__(self, app):
        super().__init__(app)
        self.current_user = None        # ← shared across ALL concurrent requests

    async def dispatch(self, request, call_next):
        self.current_user = await auth(request)   # request B overwrites A's user mid-flight
        response = await call_next(request)
        return response

The exception lifecycle deserves its own attention. When your handler raises an unhandled exception, it propagates up through call_next in each middleware's dispatch. Any middleware that does not catch it passes it up. Eventually it reaches Starlette's ServerErrorMiddleware — the outermost layer, added automatically. By default, Starlette's error middleware returns a plain 500 response with no body in production (when debug=False), or a full HTML traceback in debug=True mode. That debug mode, left on in a deployed server, is how tracebacks with database credentials reach the client.

⚠ Custom exception handlers versus middleware exception handlers

FastAPI offers @app.exception_handler(SomeException). This runs inside the middleware stack — it only catches what the router and handler raise, not what middleware raises. If a middleware itself throws, the exception handler doesn't see it. Add a try/except in dispatch if middleware can raise, or use a dedicated outermost ASGI middleware that wraps everything.

TRACELABTraceLab v2 gets a server

Stage 3 is where TraceLab leaves the single file and becomes a real client–server system. The architecture we're building toward has a FastAPI backend, a Postgres database, token authentication, and the original CLI reborn as an HTTP client. This lesson establishes the server's skeleton — the exact pipeline you traced in the widget above.

tracelab/app.pyv2 server skeleton
from fastapi import FastAPI, Depends, HTTPException, status
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import Optional
import time

app = FastAPI(title="TraceLab", version="2.0.0", debug=False)  # debug=False in prod!

# --- middleware stack (outermost added last) ---------------------------
app.add_middleware(CORSMiddleware, allow_origins=["*"])  # outermost

# --- schemas -----------------------------------------------------------
class RunCreate(BaseModel):
    name: str
    metrics: dict[str, float]
    tags: list[str] = []

class RunOut(BaseModel):
    id: int
    name: str
    metrics: dict[str, float]

# --- routes ------------------------------------------------------------
async def get_current_user(token: str):
    # Lesson 3-5 will wire this to the real token table
    if not token:
        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED)
    return token

@app.post("/runs", response_model=RunOut, status_code=201)
async def create_run(body: RunCreate,
                     user=Depends(get_current_user)):
    # Lesson 3-4 will replace this stub with real Postgres inserts
    return RunOut(id=1, name=body.name, metrics=body.metrics)

A few things worth noticing in this skeleton. debug=False is explicit: that one flag is the difference between "clean 500 response" and "stack trace with DB connection string in the client's browser." The middleware is registered with a comment about ordering — this is the kind of comment that saves an hour at 2 a.m. when someone wonders why auth runs after CORS. The schemas are separate from the route functions, because in a real API they're what the rest of the codebase and the OpenAPI spec both depend on.

The v2 milestone for this stage: the CLI is reborn as an HTTP client that calls POST /runs instead of writing directly to a CSV. The server persists runs to Postgres (Lesson 3-4). Token auth gates every write (Lesson 3-5). Multiple concurrent writers will expose a real lost-update race that Lesson 3-6 diagnoses and Lesson 3-7 fixes with transaction isolation. The v2 "God module" — a single app.py with schemas, routes, database access, and auth all in one file — is left exactly as is, on purpose. It is Stage 4's raw material.

💡 The pipeline this lesson teaches is the God module's skeleton

When you later refactor the God module into separate routers/, models/, and deps/ packages, you'll be decomposing it along exactly the pipeline layers this lesson named: routing lives in one place, dependency injection in another, validation schemas in another. Good project structure follows the request lifecycle.

WEAKNESSWhere the pipeline bites

✗ One slow middleware, every request pays

Authentication middleware that calls a remote token validation service adds its latency to every request, including health checks, metrics endpoints, and WebSocket upgrades. Latency in middleware is latency that cannot be avoided — it has no routing context. The fix: use FastAPI's Depends() for route-specific concerns (auth, rate limiting per endpoint) and reserve true middleware for genuinely universal concerns (CORS, request IDs, response timing). Profile middleware separately with a middleware that measures time of call_next against total dispatch time.

✗ Global state in the wrong place

A counter, cache, or connection pool stored as a module-level global is shared across all requests in a process — which is sometimes what you want (connection pool: yes; per-request counter: no). The subtler version: a mutable default argument on a Pydantic model, a class variable on an ASGI middleware, or an in-memory list appended to by route handlers. Concurrency makes these bugs probabilistic; the bug manifests in load tests, not unit tests. Use request.state for per-request data, dependency injection for per-request resources, and an actual database for anything that needs to persist.

✗ debug=True in production (or forgetting to set it False)

FastAPI/Starlette's debug mode sends a full Python traceback as the HTTP response body when an exception is unhandled. That traceback includes file paths, local variable values, and — if you are unlucky — the value of a database URL, API key, or secret token that happened to be in scope. The check: grep -r "debug=True" tracelab/ before every deploy, and make debug a config value loaded from an environment variable, not a hardcoded literal.

KEY INSIGHTHow this connects

  • HTTP: The Lingua Franca (3.1) — the bytes that the socket layer hands up are the HTTP message this lesson built. Method, path, headers, body — the lifecycle's first two layers are just the parsing of what 3.1 described.
  • The OS Contract (1.3) — accept(), file descriptors, and syscalls from Lesson 1.3 are the literal mechanism of Layer 1. The event loop's non-blocking I/O is the same epoll/kqueue discussed there.
  • Databases I: Relational Thinking (3.4) → — next, the handler makes its first real I/O call: a SQL query. The await point inside the handler is where Layer 7 suspends and the event loop runs other coroutines.

The request lifecycle is the thread that ties together what you learned about OS processes (Stage 1), HTTP (Lesson 3.1), and what you're about to learn about databases. When something goes wrong at 2 a.m., it will be somewhere in this pipeline — and now you can name every layer.

GLOSSARYKey terms introduced

WSGI (Web Server Gateway Interface)
PEP 3333 — the Python standard for synchronous web apps. A WSGI app is a callable (environ, start_response) → iterable[bytes]. One thread per concurrent request.
ASGI (Asynchronous Server Gateway Interface)
The async generalisation of WSGI. An ASGI app is async (scope, receive, send). One event loop thread can handle thousands of concurrent requests by suspending at await points.
Middleware
A layer in the request/response pipeline that wraps the handler. Runs on every request in both directions. Correct for cross-cutting concerns (CORS, request IDs); dangerous for per-request state.
Event loop
The asyncio scheduler. It maintains a queue of ready coroutines, runs each until it hits an await, suspends it, and runs the next. One thread, many concurrent I/O waits.
Dependency injection (FastAPI)
FastAPI's Depends() system. Declares that a route needs a value (a DB session, an auth principal) and FastAPI resolves and injects it before the handler runs. Failed dependencies short-circuit the request.
Worker
A single OS process (Gunicorn prefork) or thread running one instance of the application. Worker count controls parallelism at the process level; the event loop inside each worker controls concurrency within that process.
422 Unprocessable Entity
HTTP status returned by FastAPI when Pydantic validation fails on the request body or query parameters. The client receives a machine-readable error detail listing every field that failed and why.
ServerErrorMiddleware
Starlette's outermost automatic middleware. Catches any unhandled exception that escapes the rest of the stack and converts it to a 500 response — either a plain body (debug=False) or a traceback (debug=True).

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What is the difference between WSGI and ASGI, and why does it matter for TraceLab?›

WSGI is synchronous — the server calls your app and blocks until it returns, requiring a thread per concurrent request. ASGI is async — the server awaits your app, which can suspend at I/O points and let other requests run on the same thread. For TraceLab, which makes many database calls per request, ASGI (Uvicorn + FastAPI) lets a single event loop handle many concurrent telemetry writers without the per-thread overhead of WSGI.

2A colleague adds a middleware that calls a remote config service to check a feature flag. Every request now has 80 ms of extra latency. Why?›

Middleware runs on every request, regardless of route. The config service call adds 80 ms to the total request time for every endpoint — health checks, metrics, WebSocket upgrades, everything. Middleware is the wrong place for per-route or conditional work. A FastAPI Depends() only runs for the routes that declare it; that's where route-specific I/O should live.

3An unhandled ValueError is raised in a route handler. Trace its journey from the handler to the client.›

The ValueError propagates up through call_next in each middleware's dispatch. Any middleware without a try/except passes it up. FastAPI's registered exception_handler catches it if one is registered for ValueError; otherwise it reaches Starlette's ServerErrorMiddleware, which returns a 500. If debug=False, the body is empty or generic; if debug=True, the full Python traceback is sent to the client — potentially including secrets in local variables.

4Why does running four Uvicorn workers handle more load than running one — even though each worker is single-threaded?›

A single event loop is still bound to one CPU core. Four workers are four OS processes, each with their own event loop, scheduled across four cores by the OS. Within each worker, async concurrency handles many I/O-bound requests simultaneously. Between workers, process-level parallelism uses all available CPU. The two mechanisms are additive: async concurrency handles I/O saturation; multiple workers handle CPU saturation.

Next: TraceLab meets Postgres — relational thinking from first principles →