Concurrency II: Async and the Event Loop
What await actually yields to — cooperative scheduling, I/O-bound concurrency without threads, and the cardinal sin of blocking the loop.
🎯 By the end you'll understand
Why async exists as a distinct answer to I/O-bound concurrency, how the event loop works as a concrete piece of runnable machinery (not a black box), what await actually yields to and when control returns, why one blocking call starves every other task in the loop, and the hard boundary: async is for I/O-bound work, never CPU-bound. You will also upgrade TraceLab's ingest endpoint to async — with eyes open about exactly what that buys and what it doesn't.
KEY INSIGHTawait yields to the scheduler, not to oblivion
When your coroutine hits await, it doesn't block — it hands a note to the event loop saying "wake me when this I/O is ready," then steps aside so every other coroutine gets a turn.
You've seen threads: the OS interrupts them preemptively, context-switches them at will, and charges you for synchronization primitives whenever they share state. Async is a different deal. It's cooperative — each task runs until it explicitly awaits something that isn't ready yet. The event loop, not the OS scheduler, decides what runs next. That's simultaneously async's superpower and its trap: nothing preempts you, so if you forget to yield, you starve everyone else.
The event loop is a single-threaded loop. It pulls the next "ready" callback off a queue and runs it. When a coroutine hits await asyncio.sleep(1), it registers a timer callback and suspends itself — other coroutines run in that second. When the timer fires, the coroutine is put back on the ready queue. If instead you call time.sleep(1), you call into the OS, the entire thread parks, and the event loop stops turning for a full second. Nothing runs. That is the cardinal sin of async programming.
CORE CONCEPTWhy async exists: the C10K problem and its economics
The year was 1999. Dan Kegel published "The C10K Problem" — how do you handle 10,000 simultaneous network connections on a single server? The threading approach has a hard ceiling: Linux default stack size is 8 MB, so 10,000 threads costs ~80 GB of RAM just for stacks. Context switches between 10,000 threads consume measurable CPU. The OS scheduler pays attention to every thread even if 9,900 are blocked waiting for network packets that haven't arrived yet.
The observation that unlocked async is deceptively simple: a blocked thread isn't doing anything useful. If you can unblock the thread the instant its I/O is ready — instead of parking it in the OS — you need far fewer threads. The OS already provides the mechanism: epoll on Linux, kqueue on macOS, IOCP on Windows. These are file-descriptor multiplexers — you hand them a thousand sockets and say "tell me when any one becomes readable." They return you a list of ready ones. No sleeping threads needed.
An event loop is the application-level loop around one of those syscalls:
while tasks_exist(): ready_fds = epoll_wait(watched_fds, timeout=next_timer) for fd in ready_fds: callbacks[fd]() # resume the coroutine waiting on this fd for timer in expired_timers(): timer.callback() # resume coroutines whose sleep() elapsed
Python's asyncio, Node.js's libuv, and Go's runtime all implement this pattern — the differences are in how they expose it. CPython's implementation lives in asyncio/base_events.py and is readable. The core loop is literally a while True around a self._selector.select(timeout) call plus a callback dispatch step. It is not magic.
HOW IT WORKSThe event loop as a visible, dissectable mechanism
Python's asyncio exposes the loop explicitly. Here is what the loop actually manages for each "tick":
Run ready callbacks
Anything scheduled with loop.call_soon() runs immediately on this tick — no I/O wait involved. This is how asyncio.sleep(0) works: it schedules a "wake me at start of next tick" callback and yields, giving every other ready task a chance to run before you continue.
Poll I/O (epoll/select)
Call selector.select(timeout). The timeout is min(time_until_next_timer, 0) — it's zero if there are ready callbacks left, or however long until the earliest scheduled timer fires. The OS returns a list of (fileobj, event) pairs whose file descriptors became readable/writable.
Fire I/O callbacks
For each ready file descriptor, call the registered callback — which typically resumes the coroutine that was awaiting it. The coroutine runs until its next await, at which point control returns to the loop.
Fire expired timers
Check the timer heap (a min-heap ordered by scheduled time). Fire any callback whose deadline has passed — asyncio.sleep(), asyncio.wait_for() timeouts, and similar.
Repeat until no tasks remain
The loop exits when there are no more tasks, callbacks, or scheduled timers. In a server that's never — loop.run_forever() blocks until you call loop.stop().
asyncio.get_event_loop().time() returns the loop's monotonic clock. asyncio.all_tasks() returns every live Task object. task.get_stack() gives you a traceback for each one. When something is mysteriously slow, these three calls tell you exactly what the loop is juggling and where each task is stuck — no profiler required for a first diagnosis.
CORE CONCEPTCoroutines, Tasks, and the suspension protocol
A function decorated with async def is a coroutine function. Calling it returns a coroutine object — a heap-allocated frame that can be suspended and resumed. This is the same heap-frame trick you saw in Lesson 1.1: Python frames live on the heap so they can outlive a C-stack call, which is exactly how generators (and therefore coroutines) work. async def builds on PEP 342 generators; await is syntactic sugar for yield from.
import asyncio async def fetch_metric(run_id: str) -> dict: # 1. coroutine suspends itself, registers the TCP socket with the loop response = await some_http_client.get(f"/runs/{run_id}") # 2. loop runs other coroutines while we wait for bytes on the socket # 3. when socket becomes readable, loop resumes THIS coroutine here return response.json() async def main(): # asyncio.gather wraps each coroutine in a Task — they run concurrently results = await asyncio.gather( fetch_metric("run-001"), fetch_metric("run-002"), fetch_metric("run-003"), ) # 3 HTTP requests — total wall time ≈ max(latencies), not sum(latencies) return results
The key economics: three sequential HTTP calls at 100 ms each cost 300 ms. Three concurrent async calls cost ~100 ms — the loop issues all three, suspends all three, and resumes each one as its bytes arrive. The CPU does nothing interesting during the wait in either case; async just stops wasting thread slots on nothing.
A Task is an asyncio-managed wrapper around a coroutine. When you do asyncio.create_task(coro), the loop adds the Task to its runqueue. Tasks can be cancelled, inspected, and awaited. asyncio.gather() creates Tasks internally and waits for all of them — if one raises, the rest are cancelled (by default).
A coroutine object does nothing until something drives it. await coro drives it directly (no concurrency — you wait for it to finish). asyncio.create_task(coro) wraps it in a Task and schedules it; then it runs concurrently with other tasks in the same loop. Forgetting create_task and just awaiting sequentially is one of the most common async mistakes in real codebases.
SEE ITEvent loop visualizer — watch a blocking call starve the loop
The only way to truly believe the starvation problem is to watch it happen on a timeline. This visualizer runs a deterministic simulation of an event loop with three coroutines. Select a scenario, press Run, and watch what each coroutine does per loop tick on the Gantt timeline below.
Notice the difference between the second and third scenarios in raw numbers: the sync-sleep scenario shows Ingest Handler blocked for 8 consecutive ticks while everything else — health checks, telemetry ingestion — queues up and waits. Users experience this as ~80 ms of added latency for every request that arrives while the sleep runs. The third scenario replaces time.sleep() with await asyncio.sleep() and the timeline snaps back to health. Same logic, radically different behavior.
CORE CONCEPTStructured concurrency: nurseries and task lifetimes
Classic async code with bare asyncio.create_task() has a task-lifetime problem: you spawn a background task, the function returns, and now you have a task floating in the event loop with no parent keeping track of it. It might succeed silently, fail silently, leak connections, or run long after the work that spawned it is logically complete. This is the async equivalent of fire-and-forget thread spawning.
Structured concurrency — popularized by Nathaniel J. Smith's 2018 essay "Notes on structured concurrency, or: Go statement considered harmful" — applies the principle of structured programming to async: tasks are scoped to a block, and the block does not exit until all tasks within it have either finished or been cancelled.
import asyncio # ✗ unstructured — ingest_task can outlive handle_request if we're not careful async def handle_request_bad(run_id): ingest_task = asyncio.create_task(ingest_to_db(run_id)) return {"status": "queued"} # task still running — nobody watching it # ✓ structured — TaskGroup (Python 3.11+) keeps all tasks scoped to the block async def handle_request_good(run_id): async with asyncio.TaskGroup() as tg: ingest_task = tg.create_task(ingest_to_db(run_id)) health_task = tg.create_task(emit_health_metric()) # both tasks done or both cancelled here — block cannot exit otherwise return {"status": "ok"}
Python 3.11 added asyncio.TaskGroup as the standard-library implementation of this pattern. The trio library made nurseries famous first and has a stronger implementation; the anyio library provides a portable abstraction over both. For TraceLab, TaskGroups are the right tool for handling a request that fans out to multiple I/O operations — write to the database, emit a telemetry event, update a cache — and must not leave orphaned tasks behind.
When using TaskGroup, if multiple tasks fail, Python collects all their exceptions into a single ExceptionGroup and raises it when the block exits. You handle these with the new except* syntax: except* ValueError as eg:. This was impossible in earlier error-propagation models — a good reason to prefer structured concurrency over bare gather() in new code.
IN PRACTICEThe hard boundary: async is for I/O, never CPU
Here is where async mythology does real damage. People see "concurrency" and reach for it everywhere. Async gives you concurrency for free only when tasks spend most of their time waiting for I/O. For CPU-bound work, async buys you nothing — worse than nothing, because you're adding coroutine overhead to a single thread that's fully occupied.
| Work type | Thread-blocked while waiting? | Right tool | Wrong tool |
|---|---|---|---|
| HTTP requests to external APIs | Yes — waiting on network | async + httpx/aiohttp | time.sleep() loops, blocking requests |
| Database queries (PostgreSQL) | Yes — waiting on DB server | async + asyncpg / SQLAlchemy async | sync psycopg2 in async handlers |
| File I/O (large files) | Yes — waiting on disk | asyncio.to_thread() or aiofiles | open() inside async handlers |
| Matrix multiplication, ML training | No — CPU is at 100% | multiprocessing / ProcessPoolExecutor | asyncio (GIL-bound, no parallelism) |
| Image resizing, video encoding | No — CPU is at 100% | ProcessPoolExecutor or C extension | asyncio.create_task() (still one thread!) |
| Mixed: HTTP ingest + metric computation | Split | async for I/O, offload CPU to thread pool | doing CPU work directly in async handler |
The escape hatch for CPU-bound work inside an async application is asyncio.to_thread() (Python 3.9+) or loop.run_in_executor(). These hand your blocking or CPU-bound function to a thread pool, and await the result — the loop can service other coroutines while the thread grinds through the computation. This is the standard pattern for, say, calling a scikit-learn model inside a FastAPI endpoint.
import asyncio from concurrent.futures import ProcessPoolExecutor _pool = ProcessPoolExecutor(max_workers=4) async def compute_reward_stats(trajectory: list) -> dict: # heavy numpy work — must NOT run on the event loop thread loop = asyncio.get_running_loop() result = await loop.run_in_executor(_pool, _numpy_stats, trajectory) return result def _numpy_stats(trajectory): # runs in a subprocess — GIL released, real parallelism import numpy as np arr = np.array(trajectory) return {"mean": float(np.mean(arr)), "std": float(np.std(arr))}
A common misconception: "the GIL prevents threads from running in parallel, so I switched to async." The GIL is about CPU-parallel Python bytecode execution. It has no bearing on I/O concurrency — asyncio never tries to run Python concurrently in multiple threads; it runs it in one. The GIL is irrelevant to the question of whether your async handler blocks the loop. That question is answered entirely by whether you call blocking functions.
TRACELABGiving the ingest endpoint async eyes
TraceLab v2 now has an HTTP API, a Postgres backend, and concurrent telemetry writers. The ingest endpoint — POST /runs/{run_id}/metrics — is the hot path: it receives a JSON payload, writes a row to Postgres, and optionally fires a webhook to notify downstream consumers. Under load this endpoint will receive dozens of simultaneous requests, each spending most of its time waiting for the database round-trip (~2–15 ms). This is the ideal async target.
from fastapi import APIRouter, Depends, HTTPException import asyncio, asyncpg router = APIRouter() # asyncpg connection pool — each coroutine borrows a connection from the pool, # awaits the query, and returns it. No threads parked on idle connections. async def get_pool() -> asyncpg.Pool: return app.state.db_pool # set at startup in lifespan() async def maybe_fire_webhook(run_id: str, payload: dict) -> None: """Best-effort webhook — we do NOT await this in the main path.""" # httpx async client; this I/O is on the event loop too try: await http_client.post(webhook_url, json={"run_id": run_id, **payload}) except Exception: pass # best-effort; log and move on @router.post("/runs/{run_id}/metrics") async def ingest_metrics(run_id: str, body: MetricPayload, pool=Depends(get_pool)): async with pool.acquire() as conn: # await suspends this coroutine while Postgres processes the write # the event loop serves OTHER requests during those ~5 ms await conn.execute( "INSERT INTO metrics(run_id,step,key,value) VALUES($1,$2,$3,$4)", run_id, body.step, body.key, body.value ) # create_task: fire webhook concurrently, don't block the response asyncio.create_task(maybe_fire_webhook(run_id, body.dict())) return {"status": "ok"} # What this does NOT fix: if body.value requires heavy computation # (say, reward normalization across 50k historical steps), that must # go to run_in_executor — do NOT do it inline here.
The crucial decisions made here — and why they were made:
- asyncpg over psycopg2: asyncpg is a native async PostgreSQL driver; psycopg2's
execute()is a blocking call and would park the thread even inside anasync def. The naive mistake is wrapping a sync DB call inasync defand wondering why concurrency doesn't improve. - Connection pool, not per-request connections: Each
await conn.execute()borrows from a pool of TCP connections to Postgres. Without a pool, you'd pay TCP handshake latency on every ingest call. - Best-effort webhook with
create_task: The response to the caller doesn't wait for the webhook. The task floats — which is acceptable here because it's explicitly fire-and-forget and we log failures. In a more critical path, aTaskGroupwould be the right tool. - What this doesn't fix: The v2 God module is still here. The ingest endpoint is now async-correct, but the concurrency architecture still has the lost-update race from concurrent telemetry writers (lesson 3-6: transactions). Async concurrency and transactional correctness are orthogonal concerns.
FastAPI runs async def endpoints directly on the event loop. If you mark an endpoint async def but call blocking code inside it — requests.get(), time.sleep(), psycopg2.execute() — you have blocked the event loop. FastAPI will run plain def (sync) endpoints in a thread pool, which is safer for truly blocking code, but you lose async concurrency for them. The correct path is async throughout with async-native libraries.
WEAKNESSWhere async bites engineers — the real failure modes
The cardinal sin, and it's entirely silent. time.sleep(0.1) inside an async handler doesn't crash, doesn't warn, doesn't emit a log line — it just makes every other user in the system wait 100 ms they have no idea they're waiting. Tools that catch this: asyncio.set_debug(True) enables slow-callback detection (logs a warning when any callback takes more than 100 ms). The aiomonitor library attaches a live inspector to a running event loop. In production, histogram-bucket your request latencies by percentile: if p99 spikes while p50 is fine, a blocking call is the first hypothesis.
Two common mistakes: (1) calling asyncio.run() inside a function that's already inside a running event loop — this raises RuntimeError: This event loop is already running. Jupyter notebooks suffer from this because they run their own event loop. Fix: use await coro directly in Jupyter, or nest_asyncio as a workaround. (2) Passing a coroutine object to something expecting a value — result = my_async_func() — you get back a coroutine object, not the result. The coroutine runs nothing until driven. Python emits RuntimeWarning: coroutine was never awaited since Python 3.4+ when the coroutine object is garbage-collected unawaited — it is easy to miss in complex async code; asyncio.set_debug(True) surfaces additional diagnostics.
The most conceptually damaging mistake: reaching for asyncio.gather() to speed up CPU-heavy computations. If you have 8 coroutines each doing 100 ms of pure Python arithmetic, gather runs them serially on one thread — total time is still ~800 ms, plus overhead. Async does not parallelize CPU work. For real CPU parallelism in Python: multiprocessing, concurrent.futures.ProcessPoolExecutor, or step outside Python entirely (Cython, Rust extension, a dedicated worker process). An async wrapper around CPU work is the worst of both worlds: the complexity of async with none of the concurrency benefits.
KEY INSIGHTHow this connects
- Concurrency I: Threads (3.7) — the contrast is explicit: threads are preemptive and OS-scheduled; async is cooperative and event-loop scheduled. Both address concurrency but at different cost structures and for different problem shapes. If you haven't read 3-7, the OS contract for threads and the GIL give context that makes async's tradeoffs clearer.
- The OS Contract (1.3) — the epoll/kqueue syscalls that power the event loop are exactly the OS I/O multiplexing mechanisms introduced there.
select(),poll(),epoll(), andkqueue()are the kernel primitives every event loop wraps. Async is these syscalls, given a pleasant coroutine API. - AuthN and AuthZ (3.9) → — TraceLab now has an async ingest endpoint. Next up: who is allowed to ingest, and how the API verifies identity. Token authentication in an async context means
awaiting a database lookup on each request — the patterns here apply directly there.
The through-line for Stage 3: every layer we've built — HTTP, API design, server internals, relational DB, queries, transactions, threads, async — is a mechanism you can inspect and break. Stage 4 takes the TraceLab v2 God module that's been accumulating debt and architects it deliberately: the same machinery, organized so the 2 a.m. incident has a clear owner.
GLOSSARYKey terms introduced
- Event loop
- A single-threaded loop that polls I/O readiness (via epoll/kqueue/select), fires callbacks for ready file descriptors and expired timers, and runs scheduled callbacks — the scheduler for cooperative async concurrency.
- Coroutine
- A function defined with
async defwhose execution can be suspended atawaitpoints and resumed later. Implemented as a heap-allocated frame object — the same mechanism as Python generators. - Task (asyncio)
- An event-loop-managed wrapper around a coroutine. Created with
asyncio.create_task(); the loop drives it autonomously, concurrently with other Tasks. Distinguishes scheduled-concurrency from sequential coroutine chaining. - Cooperative scheduling
- A concurrency model where each task runs until it voluntarily yields (via
await). The scheduler never interrupts a running task — contrast with preemptive OS scheduling of threads. - Structured concurrency
- The discipline that task lifetimes are scoped to a block: all spawned tasks must complete or be cancelled before the block exits. Prevents orphaned tasks and makes cancellation and error propagation deterministic. Implemented by
asyncio.TaskGroupandtrionurseries. - Blocking call
- Any function call that makes the OS park the current thread until an operation completes —
time.sleep(), synchronous socket reads, blocking file I/O. Inside an async handler this stalls the event loop because the loop and your code share the same OS thread. - epoll / kqueue / IOCP
- OS-level I/O multiplexing interfaces. You register a set of file descriptors and ask "which are ready?" — the kernel returns only the ready ones, enabling one thread to monitor thousands of connections efficiently. The kernel-level mechanism beneath every async I/O framework.
- asyncio.to_thread()
- Python 3.9+ API that runs a blocking (or CPU-bound) function in a thread pool and returns an awaitable. Lets async code delegate non-async work without blocking the event loop.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What does await asyncio.sleep(1) actually do, step by step, inside the event loop?›
The coroutine registers a timer callback with the event loop scheduled 1 second from now, then suspends itself — returning control to the event loop. The loop continues running other ready coroutines and processing I/O. After 1 second the timer fires, the callback is placed in the ready queue, and on the next loop tick the original coroutine resumes execution at the line following the await. The OS thread never parks — only the coroutine pauses.
2You have a FastAPI endpoint that calls time.sleep(2). It's declared async def. What happens when 10 requests arrive simultaneously?›
time.sleep(2) parks the OS thread for 2 seconds. The event loop stops turning. All 9 other requests wait in the queue — no I/O is processed, no responses sent, no health checks served. Total time for all 10 requests: approximately 20 seconds (each waits for the previous to unblock). The fix: use await asyncio.sleep(2), or move the blocking operation to a thread pool with await asyncio.to_thread(blocking_fn, args).
3Why doesn't asyncio.gather(cpu_heavy_1(), cpu_heavy_2(), cpu_heavy_3()) speed up CPU-bound work?›
The event loop runs on a single OS thread. CPU-bound coroutines that never await anything will each run to completion sequentially — gather can only interleave tasks at await points. With no yields, the loop dispatches task 1 through to completion, then task 2, then task 3. Total time equals the sum of all three. And even if they did yield, the GIL would prevent multiple Python threads from running Python bytecode in parallel. For CPU parallelism: ProcessPoolExecutor with loop.run_in_executor().
4What is the difference between await some_coro() and asyncio.create_task(some_coro())?›
await some_coro() runs the coroutine sequentially in the current coroutine — current execution suspends and waits for the result before continuing. No other code runs concurrently during this time (except whatever some_coro itself awaits). asyncio.create_task(some_coro()) wraps the coroutine in a Task and schedules it on the event loop, then immediately returns the Task object. The task runs concurrently with the current coroutine — both make progress during each other's await points. create_task is the gateway to actual async concurrency; await alone is sequential.