Inside the Server: The Request Lifecycle
What FastAPI actually does between the socket and your function — sockets, routing, middleware, workers, and where the time hides.
🎯 By the end you'll understand
How a TCP connection becomes a parsed request and eventually a serialized response — the complete socket → router → middleware → validation → handler → serializer → response stack. What WSGI and ASGI actually mean and why the distinction matters for concurrency. How worker models scale (and bottleneck). Where global state hides in middleware. Why one slow middleware taxes every request. What determines whether an unhandled exception leaks a stack trace to the client or returns a clean 500.
KEY INSIGHTYour handler is the last thing called, not the first
You wrote a function. Six layers of framework machinery run before it touches your code — and six more run after. Most of what makes a server fast, safe, or broken lives in the layers you didn't write.
When you add a @app.post("/runs") route to TraceLab's FastAPI server, it feels like you wrote the server. You didn't. You wrote a handler. The framework stitches it into a pipeline that manages sockets, parses HTTP, runs authentication middleware, validates the request body against your Pydantic schema, catches unhandled exceptions, serializes your return value to JSON, and writes the response to the socket — and all of that happens whether or not you thought about it. Understanding that pipeline is the difference between debugging in production and debugging in development.
A web framework is a pipeline constructor, not a router. Your route handler sits at the center of that pipeline; middleware, validation, and serialization wrap it on both sides. Every millisecond of latency and every unhandled exception propagates through the entire stack. When something breaks, the answer to "where?" is almost always "somewhere in the pipeline you trusted but never traced."
CORE CONCEPTWSGI and ASGI: the contract between server and framework
Python web stacks are split into two independent pieces: a server that speaks TCP and HTTP (Gunicorn, Uvicorn, uWSGI), and a framework that builds responses (Flask, Django, FastAPI, Starlette). The interface between them is a spec.
WSGI (Web Server Gateway Interface, PEP 3333) is the original Python web interface. A WSGI app is a plain callable: it receives an environment dict and a start_response callback and returns an iterable of bytes. The server calls it synchronously — one thread per request. For the server to handle a new request while the previous one waits for the database, it must have another thread ready. That's the classic thread-per-request model.
# Any WSGI-compliant application must be this shape. # Flask, Django (in WSGI mode) are both just this, with extras. def application(environ, start_response): start_response("200 OK", [("Content-Type", "text/plain")]) return [b"hello"]
ASGI (Asynchronous Server Gateway Interface) generalises WSGI for async Python. An ASGI app is a three-layer callable: it receives scope, receive, and send coroutines. The server calls it with await. This single change is what allows a single OS thread to interleave thousands of in-flight requests by suspending at every await — no thread pool required.
# scope: what kind of connection this is (http, websocket, lifespan) # receive: coroutine to read the next event from the client # send: coroutine to send an event to the client async def application(scope, receive, send): await send({"type": "http.response.start", "status": 200, "headers": [(b"content-type", b"text/plain")]}) await send({"type": "http.response.body", "body": b"hello"})
FastAPI is built on Starlette, which is an ASGI framework. The server that runs it is Uvicorn — an ASGI server based on asyncio and uvloop. Every async def route you write hands control back to the event loop while it waits for I/O, allowing other requests to proceed. Every def (sync) route FastAPI runs in a thread pool so it doesn't block the loop.
TraceLab's API will have an endpoint that persists a telemetry log to Postgres. If that's a sync function, it blocks the entire event loop for its duration — one concurrent write grinds every other request to a halt. Writing it as async def with await db.execute() lets hundreds of writes interleave at the I/O suspension points. Stage 3's concurrency lesson (3-7) will push this to its limits.
HOW IT WORKSThe eight-layer pipeline: socket to response
Here is what actually happens between a TCP packet arriving and your handler's return value leaving the machine. Each layer has a real cost and a real failure mode.
Socket accept — the OS layer
The server calls accept() on a listening socket. The OS hands over a connected socket file descriptor. This is a syscall; if the server's event loop or accept queue is saturated, the client waits here. Uvicorn's event loop calls accept() non-blocking; Gunicorn's worker processes block in their own accept().
HTTP parse — bytes → structured request
The server reads bytes and parses them into an HTTP message: method, path, headers, body. Uvicorn uses httptools (a Cython wrapper around Node.js's battle-tested http_parser). A malformed request is rejected here — the framework never sees it. Content-Length or chunked encoding determines when the body is fully received.
Router — path + method → handler candidate
The ASGI app (Starlette/FastAPI) receives the scope dict and matches the path against its route table. Starlette builds a compiled regex from your path templates. A miss returns a 404. A match with wrong HTTP method returns a 405. The route match also extracts path parameters (/runs/{run_id} → run_id="exp042").
Middleware stack — inbound pass
Middleware wraps the handler in an onion. Every registered middleware runs on every request, inbound and outbound. Typical middleware: CORS headers, request ID injection, authentication token validation, rate limiting, request logging. Middleware can short-circuit (return a response immediately, e.g., 401) or pass through. Order matters: add an outer middleware after an inner one and it runs before it.
Dependency injection — FastAPI's DI system
FastAPI resolves the handler's Depends() declarations: DB sessions, auth principals, config objects. Dependencies run in declaration order; failed dependencies short-circuit with their own exceptions. The resolved values are injected as function arguments. This is pure FastAPI — Starlette has no equivalent.
Request validation — Pydantic parses the body
FastAPI calls Pydantic to coerce and validate the request body against your declared model. Type errors, missing required fields, constraint violations all become 422 Unprocessable Entity responses automatically — FastAPI generates the error detail JSON for you. The handler never sees a malformed input; validation is the gate.
Handler — your code, finally
Your function runs. It should be the thinnest thing in the pipeline: receive a validated model, do business logic, return a value. If it raises an unhandled exception, it propagates up through the middleware stack to the server's exception handler — which is where the security-critical decision of "what does the client see?" is made.
Serializer → response → socket
FastAPI runs the return value through jsonable_encoder (converts dataclasses, datetime, UUID, etc.) then through json.dumps. The bytes are wrapped in an HTTP response and written back to the socket via send(). Middleware runs again on the outbound pass — adding response headers, logging the final status code, etc.
SEE ITTrace a request through the TraceLab backend
Every request hides a time budget. Pick a scenario, inject a fault or a slow middleware, then step through all eight layers to see exactly where the milliseconds go and where exceptions surface.
Two things to notice. In the "slow auth middleware" scenario, every other request hitting the server pays that cost — not just the authenticated ones. The middleware runs unconditionally. In the "unhandled exception" scenario, the exception propagates up through the middleware stack; only the outermost exception handler (the server's 500 handler) decides whether the client sees a stack trace. If that handler is the default one, your secrets might be in the client's browser.
CORE CONCEPTWorker models: where concurrency happens (and hides)
Your framework pipeline explains one request. The worker model explains how a server handles many. There are three mainstream approaches, each with a precise tradeoff.
| Model | Tool | Concurrency unit | Good for | Bottleneck |
|---|---|---|---|---|
| Multi-process | Gunicorn (prefork) | OS processes, typically 2×CPU+1 | CPU-bound work; isolation (one crash stays in one process) | Memory (each process is a full Python runtime); process startup; shared nothing means shared-cache is hard |
| Multi-thread | Gunicorn (sync or gthread) | OS threads inside a process | I/O-bound WSGI apps; keeps shared in-process state | CPython GIL limits true CPU parallelism; shared mutable state; a deadlock kills the entire worker |
| Async event loop | Uvicorn + asyncio | Coroutines on one thread | High-concurrency I/O-bound work (many simultaneous DB/HTTP calls) | One blocking call (a slow sync library, time.sleep()) freezes all concurrent requests; requires the whole call stack to be async-aware |
In practice, TraceLab's recommended deployment is Uvicorn workers under Gunicorn: multiple Gunicorn processes each running a Uvicorn event loop. This combines process isolation (one loop crash doesn't kill the server) with async concurrency (each loop handles thousands of simultaneous requests). gunicorn app:app -w 4 -k uvicorn.workers.UvicornWorker is five words of config that compress a decade of lessons about scaling Python web services.
The classic rule is 2 × CPU cores + 1 for synchronous workers (enough to keep CPUs busy while some workers wait for I/O). For async workers, fewer is fine — 1–2 per CPU — because each event loop handles many concurrent requests. The limit for async is the event loop's ability to get CPU time, not the number of simultaneous I/O waits. Starting with 2 Uvicorn workers per CPU core is reasonable; let metrics tell you from there.
IN PRACTICEMiddleware: the power and the trap
Middleware is where cross-cutting concerns live: authentication, logging, CORS, rate limiting, request tracing. FastAPI inherits Starlette's middleware system; you add it with app.add_middleware() or as an ASGI middleware class. The stack unwinds like a call stack — the first middleware you add is the outermost layer, running first on the inbound pass and last on the outbound pass.
from starlette.middleware.base import BaseHTTPMiddleware from starlette.requests import Request import time, uuid class RequestIDMiddleware(BaseHTTPMiddleware): async def dispatch(self, request: Request, call_next): request_id = str(uuid.uuid4()) # inbound: attach an ID before the handler sees the request request.state.request_id = request_id t0 = time.perf_counter() response = await call_next(request) # ← everything inside runs here # outbound: add header AFTER the handler has run elapsed = (time.perf_counter() - t0) * 1000 response.headers["X-Request-ID"] = request_id response.headers["X-Duration-Ms"] = f"{elapsed:.1f}" return response
This pattern is clean, but notice: call_next is an awaitable that runs the entire rest of the pipeline. Anything you do before that call runs on the inbound pass; anything after runs on the outbound pass. The middleware sees both the request and the response — which is exactly why a bug in middleware can corrupt both.
Because all requests share the same middleware instance, any mutable attribute on self is shared across concurrent requests. Store per-request data in request.state, not on the middleware class. The mistake looks like this:
class BrokenMiddleware(BaseHTTPMiddleware): def __init__(self, app): super().__init__(app) self.current_user = None # ← shared across ALL concurrent requests async def dispatch(self, request, call_next): self.current_user = await auth(request) # request B overwrites A's user mid-flight response = await call_next(request) return response
The exception lifecycle deserves its own attention. When your handler raises an unhandled exception, it propagates up through call_next in each middleware's dispatch. Any middleware that does not catch it passes it up. Eventually it reaches Starlette's ServerErrorMiddleware — the outermost layer, added automatically. By default, Starlette's error middleware returns a plain 500 response with no body in production (when debug=False), or a full HTML traceback in debug=True mode. That debug mode, left on in a deployed server, is how tracebacks with database credentials reach the client.
FastAPI offers @app.exception_handler(SomeException). This runs inside the middleware stack — it only catches what the router and handler raise, not what middleware raises. If a middleware itself throws, the exception handler doesn't see it. Add a try/except in dispatch if middleware can raise, or use a dedicated outermost ASGI middleware that wraps everything.
TRACELABTraceLab v2 gets a server
Stage 3 is where TraceLab leaves the single file and becomes a real client–server system. The architecture we're building toward has a FastAPI backend, a Postgres database, token authentication, and the original CLI reborn as an HTTP client. This lesson establishes the server's skeleton — the exact pipeline you traced in the widget above.
from fastapi import FastAPI, Depends, HTTPException, status from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel from typing import Optional import time app = FastAPI(title="TraceLab", version="2.0.0", debug=False) # debug=False in prod! # --- middleware stack (outermost added last) --------------------------- app.add_middleware(CORSMiddleware, allow_origins=["*"]) # outermost # --- schemas ----------------------------------------------------------- class RunCreate(BaseModel): name: str metrics: dict[str, float] tags: list[str] = [] class RunOut(BaseModel): id: int name: str metrics: dict[str, float] # --- routes ------------------------------------------------------------ async def get_current_user(token: str): # Lesson 3-5 will wire this to the real token table if not token: raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED) return token @app.post("/runs", response_model=RunOut, status_code=201) async def create_run(body: RunCreate, user=Depends(get_current_user)): # Lesson 3-4 will replace this stub with real Postgres inserts return RunOut(id=1, name=body.name, metrics=body.metrics)
A few things worth noticing in this skeleton. debug=False is explicit: that one flag is the difference between "clean 500 response" and "stack trace with DB connection string in the client's browser." The middleware is registered with a comment about ordering — this is the kind of comment that saves an hour at 2 a.m. when someone wonders why auth runs after CORS. The schemas are separate from the route functions, because in a real API they're what the rest of the codebase and the OpenAPI spec both depend on.
The v2 milestone for this stage: the CLI is reborn as an HTTP client that calls POST /runs instead of writing directly to a CSV. The server persists runs to Postgres (Lesson 3-4). Token auth gates every write (Lesson 3-5). Multiple concurrent writers will expose a real lost-update race that Lesson 3-6 diagnoses and Lesson 3-7 fixes with transaction isolation. The v2 "God module" — a single app.py with schemas, routes, database access, and auth all in one file — is left exactly as is, on purpose. It is Stage 4's raw material.
When you later refactor the God module into separate routers/, models/, and deps/ packages, you'll be decomposing it along exactly the pipeline layers this lesson named: routing lives in one place, dependency injection in another, validation schemas in another. Good project structure follows the request lifecycle.
WEAKNESSWhere the pipeline bites
Authentication middleware that calls a remote token validation service adds its latency to every request, including health checks, metrics endpoints, and WebSocket upgrades. Latency in middleware is latency that cannot be avoided — it has no routing context. The fix: use FastAPI's Depends() for route-specific concerns (auth, rate limiting per endpoint) and reserve true middleware for genuinely universal concerns (CORS, request IDs, response timing). Profile middleware separately with a middleware that measures time of call_next against total dispatch time.
A counter, cache, or connection pool stored as a module-level global is shared across all requests in a process — which is sometimes what you want (connection pool: yes; per-request counter: no). The subtler version: a mutable default argument on a Pydantic model, a class variable on an ASGI middleware, or an in-memory list appended to by route handlers. Concurrency makes these bugs probabilistic; the bug manifests in load tests, not unit tests. Use request.state for per-request data, dependency injection for per-request resources, and an actual database for anything that needs to persist.
debug=True in production (or forgetting to set it False)FastAPI/Starlette's debug mode sends a full Python traceback as the HTTP response body when an exception is unhandled. That traceback includes file paths, local variable values, and — if you are unlucky — the value of a database URL, API key, or secret token that happened to be in scope. The check: grep -r "debug=True" tracelab/ before every deploy, and make debug a config value loaded from an environment variable, not a hardcoded literal.
KEY INSIGHTHow this connects
- HTTP: The Lingua Franca (3.1) — the bytes that the socket layer hands up are the HTTP message this lesson built. Method, path, headers, body — the lifecycle's first two layers are just the parsing of what 3.1 described.
- The OS Contract (1.3) —
accept(), file descriptors, and syscalls from Lesson 1.3 are the literal mechanism of Layer 1. The event loop's non-blocking I/O is the sameepoll/kqueuediscussed there. - Databases I: Relational Thinking (3.4) → — next, the handler makes its first real I/O call: a SQL query. The await point inside the handler is where Layer 7 suspends and the event loop runs other coroutines.
The request lifecycle is the thread that ties together what you learned about OS processes (Stage 1), HTTP (Lesson 3.1), and what you're about to learn about databases. When something goes wrong at 2 a.m., it will be somewhere in this pipeline — and now you can name every layer.
GLOSSARYKey terms introduced
- WSGI (Web Server Gateway Interface)
- PEP 3333 — the Python standard for synchronous web apps. A WSGI app is a callable
(environ, start_response) → iterable[bytes]. One thread per concurrent request. - ASGI (Asynchronous Server Gateway Interface)
- The async generalisation of WSGI. An ASGI app is
async (scope, receive, send). One event loop thread can handle thousands of concurrent requests by suspending atawaitpoints. - Middleware
- A layer in the request/response pipeline that wraps the handler. Runs on every request in both directions. Correct for cross-cutting concerns (CORS, request IDs); dangerous for per-request state.
- Event loop
- The asyncio scheduler. It maintains a queue of ready coroutines, runs each until it hits an
await, suspends it, and runs the next. One thread, many concurrent I/O waits. - Dependency injection (FastAPI)
- FastAPI's
Depends()system. Declares that a route needs a value (a DB session, an auth principal) and FastAPI resolves and injects it before the handler runs. Failed dependencies short-circuit the request. - Worker
- A single OS process (Gunicorn prefork) or thread running one instance of the application. Worker count controls parallelism at the process level; the event loop inside each worker controls concurrency within that process.
- 422 Unprocessable Entity
- HTTP status returned by FastAPI when Pydantic validation fails on the request body or query parameters. The client receives a machine-readable error detail listing every field that failed and why.
- ServerErrorMiddleware
- Starlette's outermost automatic middleware. Catches any unhandled exception that escapes the rest of the stack and converts it to a 500 response — either a plain body (debug=False) or a traceback (debug=True).
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What is the difference between WSGI and ASGI, and why does it matter for TraceLab?›
WSGI is synchronous — the server calls your app and blocks until it returns, requiring a thread per concurrent request. ASGI is async — the server awaits your app, which can suspend at I/O points and let other requests run on the same thread. For TraceLab, which makes many database calls per request, ASGI (Uvicorn + FastAPI) lets a single event loop handle many concurrent telemetry writers without the per-thread overhead of WSGI.
2A colleague adds a middleware that calls a remote config service to check a feature flag. Every request now has 80 ms of extra latency. Why?›
Middleware runs on every request, regardless of route. The config service call adds 80 ms to the total request time for every endpoint — health checks, metrics, WebSocket upgrades, everything. Middleware is the wrong place for per-route or conditional work. A FastAPI Depends() only runs for the routes that declare it; that's where route-specific I/O should live.
3An unhandled ValueError is raised in a route handler. Trace its journey from the handler to the client.›
The ValueError propagates up through call_next in each middleware's dispatch. Any middleware without a try/except passes it up. FastAPI's registered exception_handler catches it if one is registered for ValueError; otherwise it reaches Starlette's ServerErrorMiddleware, which returns a 500. If debug=False, the body is empty or generic; if debug=True, the full Python traceback is sent to the client — potentially including secrets in local variables.
4Why does running four Uvicorn workers handle more load than running one — even though each worker is single-threaded?›
A single event loop is still bound to one CPU core. Four workers are four OS processes, each with their own event loop, scheduled across four cores by the OS. Within each worker, async concurrency handles many I/O-bound requests simultaneously. Between workers, process-level parallelism uses all available CPU. The two mechanisms are additive: async concurrency handles I/O saturation; multiple workers handle CPU saturation.