ProjekinAja. Software Engineering Stage 3 · Systems Lesson 3.8
● Stage 3 · Lesson 3.8

Concurrency II: Async and the Event Loop

What await actually yields to — cooperative scheduling, I/O-bound concurrency without threads, and the cardinal sin of blocking the loop.

⏱ ~20 min read 🧠 Mechanism + failure mode ⚡ TraceLab async ingest

🎯 By the end you'll understand

Why async exists as a distinct answer to I/O-bound concurrency, how the event loop works as a concrete piece of runnable machinery (not a black box), what await actually yields to and when control returns, why one blocking call starves every other task in the loop, and the hard boundary: async is for I/O-bound work, never CPU-bound. You will also upgrade TraceLab's ingest endpoint to async — with eyes open about exactly what that buys and what it doesn't.

KEY INSIGHTawait yields to the scheduler, not to oblivion

When your coroutine hits await, it doesn't block — it hands a note to the event loop saying "wake me when this I/O is ready," then steps aside so every other coroutine gets a turn.

You've seen threads: the OS interrupts them preemptively, context-switches them at will, and charges you for synchronization primitives whenever they share state. Async is a different deal. It's cooperative — each task runs until it explicitly awaits something that isn't ready yet. The event loop, not the OS scheduler, decides what runs next. That's simultaneously async's superpower and its trap: nothing preempts you, so if you forget to yield, you starve everyone else.

🔑 Hold onto this

The event loop is a single-threaded loop. It pulls the next "ready" callback off a queue and runs it. When a coroutine hits await asyncio.sleep(1), it registers a timer callback and suspends itself — other coroutines run in that second. When the timer fires, the coroutine is put back on the ready queue. If instead you call time.sleep(1), you call into the OS, the entire thread parks, and the event loop stops turning for a full second. Nothing runs. That is the cardinal sin of async programming.

CORE CONCEPTWhy async exists: the C10K problem and its economics

The year was 1999. Dan Kegel published "The C10K Problem" — how do you handle 10,000 simultaneous network connections on a single server? The threading approach has a hard ceiling: Linux default stack size is 8 MB, so 10,000 threads costs ~80 GB of RAM just for stacks. Context switches between 10,000 threads consume measurable CPU. The OS scheduler pays attention to every thread even if 9,900 are blocked waiting for network packets that haven't arrived yet.

The observation that unlocked async is deceptively simple: a blocked thread isn't doing anything useful. If you can unblock the thread the instant its I/O is ready — instead of parking it in the OS — you need far fewer threads. The OS already provides the mechanism: epoll on Linux, kqueue on macOS, IOCP on Windows. These are file-descriptor multiplexers — you hand them a thousand sockets and say "tell me when any one becomes readable." They return you a list of ready ones. No sleeping threads needed.

An event loop is the application-level loop around one of those syscalls:

conceptual event loop (pseudocode)the mechanism
while tasks_exist():
    ready_fds = epoll_wait(watched_fds, timeout=next_timer)
    for fd in ready_fds:
        callbacks[fd]()          # resume the coroutine waiting on this fd
    for timer in expired_timers():
        timer.callback()         # resume coroutines whose sleep() elapsed

Python's asyncio, Node.js's libuv, and Go's runtime all implement this pattern — the differences are in how they expose it. CPython's implementation lives in asyncio/base_events.py and is readable. The core loop is literally a while True around a self._selector.select(timeout) call plus a callback dispatch step. It is not magic.

Event Loop single thread · epoll/select Ready Queue callbacks to run now coro A await network I/O coro B await DB query coro C await sleep(0.5) wake OS kernel — epoll/kqueue/IOCP — the real I/O watcher fd ready? time.sleep(1) parks the OS thread loop stops turning everything starves ⚠
The event loop suspends coroutines on I/O, asks the OS when their file descriptors are ready, then resumes them. A blocking call bypasses this entirely and parks the thread — starving every other coroutine for its duration.

HOW IT WORKSThe event loop as a visible, dissectable mechanism

Python's asyncio exposes the loop explicitly. Here is what the loop actually manages for each "tick":

1

Run ready callbacks

Anything scheduled with loop.call_soon() runs immediately on this tick — no I/O wait involved. This is how asyncio.sleep(0) works: it schedules a "wake me at start of next tick" callback and yields, giving every other ready task a chance to run before you continue.

2

Poll I/O (epoll/select)

Call selector.select(timeout). The timeout is min(time_until_next_timer, 0) — it's zero if there are ready callbacks left, or however long until the earliest scheduled timer fires. The OS returns a list of (fileobj, event) pairs whose file descriptors became readable/writable.

3

Fire I/O callbacks

For each ready file descriptor, call the registered callback — which typically resumes the coroutine that was awaiting it. The coroutine runs until its next await, at which point control returns to the loop.

4

Fire expired timers

Check the timer heap (a min-heap ordered by scheduled time). Fire any callback whose deadline has passed — asyncio.sleep(), asyncio.wait_for() timeouts, and similar.

5

Repeat until no tasks remain

The loop exits when there are no more tasks, callbacks, or scheduled timers. In a server that's never — loop.run_forever() blocks until you call loop.stop().

💡 You can inspect the loop live

asyncio.get_event_loop().time() returns the loop's monotonic clock. asyncio.all_tasks() returns every live Task object. task.get_stack() gives you a traceback for each one. When something is mysteriously slow, these three calls tell you exactly what the loop is juggling and where each task is stuck — no profiler required for a first diagnosis.

CORE CONCEPTCoroutines, Tasks, and the suspension protocol

A function decorated with async def is a coroutine function. Calling it returns a coroutine object — a heap-allocated frame that can be suspended and resumed. This is the same heap-frame trick you saw in Lesson 1.1: Python frames live on the heap so they can outlive a C-stack call, which is exactly how generators (and therefore coroutines) work. async def builds on PEP 342 generators; await is syntactic sugar for yield from.

coroutine_anatomy.pywhat these constructs actually do
import asyncio

async def fetch_metric(run_id: str) -> dict:
    # 1. coroutine suspends itself, registers the TCP socket with the loop
    response = await some_http_client.get(f"/runs/{run_id}")
    # 2. loop runs other coroutines while we wait for bytes on the socket
    # 3. when socket becomes readable, loop resumes THIS coroutine here
    return response.json()

async def main():
    # asyncio.gather wraps each coroutine in a Task — they run concurrently
    results = await asyncio.gather(
        fetch_metric("run-001"),
        fetch_metric("run-002"),
        fetch_metric("run-003"),
    )
    # 3 HTTP requests — total wall time ≈ max(latencies), not sum(latencies)
    return results

The key economics: three sequential HTTP calls at 100 ms each cost 300 ms. Three concurrent async calls cost ~100 ms — the loop issues all three, suspends all three, and resumes each one as its bytes arrive. The CPU does nothing interesting during the wait in either case; async just stops wasting thread slots on nothing.

A Task is an asyncio-managed wrapper around a coroutine. When you do asyncio.create_task(coro), the loop adds the Task to its runqueue. Tasks can be cancelled, inspected, and awaited. asyncio.gather() creates Tasks internally and waits for all of them — if one raises, the rest are cancelled (by default).

📖 The difference between a coroutine and a Task

A coroutine object does nothing until something drives it. await coro drives it directly (no concurrency — you wait for it to finish). asyncio.create_task(coro) wraps it in a Task and schedules it; then it runs concurrently with other tasks in the same loop. Forgetting create_task and just awaiting sequentially is one of the most common async mistakes in real codebases.

SEE ITEvent loop visualizer — watch a blocking call starve the loop

The only way to truly believe the starvation problem is to watch it happen on a timeline. This visualizer runs a deterministic simulation of an event loop with three coroutines. Select a scenario, press Run, and watch what each coroutine does per loop tick on the Gantt timeline below.

⚙ Interactive · Event loop timeline visualizer

Each row is a coroutine. Each column is one loop tick (~10 ms). Green = running, grey = waiting for I/O, red = blocked (starved by a synchronous call in another task). Select a scenario and press Run.

Tick
—
Running task
—
Avg response (ms)
—
Throughput (req/tick)
—
📋 Scenario

Select a scenario and press Run to see the event loop in action.

Notice the difference between the second and third scenarios in raw numbers: the sync-sleep scenario shows Ingest Handler blocked for 8 consecutive ticks while everything else — health checks, telemetry ingestion — queues up and waits. Users experience this as ~80 ms of added latency for every request that arrives while the sleep runs. The third scenario replaces time.sleep() with await asyncio.sleep() and the timeline snaps back to health. Same logic, radically different behavior.

CORE CONCEPTStructured concurrency: nurseries and task lifetimes

Classic async code with bare asyncio.create_task() has a task-lifetime problem: you spawn a background task, the function returns, and now you have a task floating in the event loop with no parent keeping track of it. It might succeed silently, fail silently, leak connections, or run long after the work that spawned it is logically complete. This is the async equivalent of fire-and-forget thread spawning.

Structured concurrency — popularized by Nathaniel J. Smith's 2018 essay "Notes on structured concurrency, or: Go statement considered harmful" — applies the principle of structured programming to async: tasks are scoped to a block, and the block does not exit until all tasks within it have either finished or been cancelled.

structured_concurrency.pyTaskGroup vs bare create_task
import asyncio

# ✗ unstructured — ingest_task can outlive handle_request if we're not careful
async def handle_request_bad(run_id):
    ingest_task = asyncio.create_task(ingest_to_db(run_id))
    return {"status": "queued"}  # task still running — nobody watching it

# ✓ structured — TaskGroup (Python 3.11+) keeps all tasks scoped to the block
async def handle_request_good(run_id):
    async with asyncio.TaskGroup() as tg:
        ingest_task = tg.create_task(ingest_to_db(run_id))
        health_task = tg.create_task(emit_health_metric())
    # both tasks done or both cancelled here — block cannot exit otherwise
    return {"status": "ok"}

Python 3.11 added asyncio.TaskGroup as the standard-library implementation of this pattern. The trio library made nurseries famous first and has a stronger implementation; the anyio library provides a portable abstraction over both. For TraceLab, TaskGroups are the right tool for handling a request that fans out to multiple I/O operations — write to the database, emit a telemetry event, update a cache — and must not leave orphaned tasks behind.

💡 Python 3.11 ExceptionGroup

When using TaskGroup, if multiple tasks fail, Python collects all their exceptions into a single ExceptionGroup and raises it when the block exits. You handle these with the new except* syntax: except* ValueError as eg:. This was impossible in earlier error-propagation models — a good reason to prefer structured concurrency over bare gather() in new code.

IN PRACTICEThe hard boundary: async is for I/O, never CPU

Here is where async mythology does real damage. People see "concurrency" and reach for it everywhere. Async gives you concurrency for free only when tasks spend most of their time waiting for I/O. For CPU-bound work, async buys you nothing — worse than nothing, because you're adding coroutine overhead to a single thread that's fully occupied.

Work typeThread-blocked while waiting?Right toolWrong tool
HTTP requests to external APIsYes — waiting on networkasync + httpx/aiohttptime.sleep() loops, blocking requests
Database queries (PostgreSQL)Yes — waiting on DB serverasync + asyncpg / SQLAlchemy asyncsync psycopg2 in async handlers
File I/O (large files)Yes — waiting on diskasyncio.to_thread() or aiofilesopen() inside async handlers
Matrix multiplication, ML trainingNo — CPU is at 100%multiprocessing / ProcessPoolExecutorasyncio (GIL-bound, no parallelism)
Image resizing, video encodingNo — CPU is at 100%ProcessPoolExecutor or C extensionasyncio.create_task() (still one thread!)
Mixed: HTTP ingest + metric computationSplitasync for I/O, offload CPU to thread pooldoing CPU work directly in async handler

The escape hatch for CPU-bound work inside an async application is asyncio.to_thread() (Python 3.9+) or loop.run_in_executor(). These hand your blocking or CPU-bound function to a thread pool, and await the result — the loop can service other coroutines while the thread grinds through the computation. This is the standard pattern for, say, calling a scikit-learn model inside a FastAPI endpoint.

cpu_in_async_app.pyoffloading CPU work correctly
import asyncio
from concurrent.futures import ProcessPoolExecutor

_pool = ProcessPoolExecutor(max_workers=4)

async def compute_reward_stats(trajectory: list) -> dict:
    # heavy numpy work — must NOT run on the event loop thread
    loop = asyncio.get_running_loop()
    result = await loop.run_in_executor(_pool, _numpy_stats, trajectory)
    return result

def _numpy_stats(trajectory):
    # runs in a subprocess — GIL released, real parallelism
    import numpy as np
    arr = np.array(trajectory)
    return {"mean": float(np.mean(arr)), "std": float(np.std(arr))}
⚠ The GIL does not protect you from blocking the loop

A common misconception: "the GIL prevents threads from running in parallel, so I switched to async." The GIL is about CPU-parallel Python bytecode execution. It has no bearing on I/O concurrency — asyncio never tries to run Python concurrently in multiple threads; it runs it in one. The GIL is irrelevant to the question of whether your async handler blocks the loop. That question is answered entirely by whether you call blocking functions.

TRACELABGiving the ingest endpoint async eyes

TraceLab v2 now has an HTTP API, a Postgres backend, and concurrent telemetry writers. The ingest endpoint — POST /runs/{run_id}/metrics — is the hot path: it receives a JSON payload, writes a row to Postgres, and optionally fires a webhook to notify downstream consumers. Under load this endpoint will receive dozens of simultaneous requests, each spending most of its time waiting for the database round-trip (~2–15 ms). This is the ideal async target.

tracelab/api/metrics.pyv2 async ingest — with eyes open
from fastapi import APIRouter, Depends, HTTPException
import asyncio, asyncpg

router = APIRouter()

# asyncpg connection pool — each coroutine borrows a connection from the pool,
# awaits the query, and returns it. No threads parked on idle connections.
async def get_pool() -> asyncpg.Pool:
    return app.state.db_pool   # set at startup in lifespan()

async def maybe_fire_webhook(run_id: str, payload: dict) -> None:
    """Best-effort webhook — we do NOT await this in the main path."""
    # httpx async client; this I/O is on the event loop too
    try:
        await http_client.post(webhook_url, json={"run_id": run_id, **payload})
    except Exception:
        pass   # best-effort; log and move on

@router.post("/runs/{run_id}/metrics")
async def ingest_metrics(run_id: str, body: MetricPayload, pool=Depends(get_pool)):
    async with pool.acquire() as conn:
        # await suspends this coroutine while Postgres processes the write
        # the event loop serves OTHER requests during those ~5 ms
        await conn.execute(
            "INSERT INTO metrics(run_id,step,key,value) VALUES($1,$2,$3,$4)",
            run_id, body.step, body.key, body.value
        )
    # create_task: fire webhook concurrently, don't block the response
    asyncio.create_task(maybe_fire_webhook(run_id, body.dict()))
    return {"status": "ok"}

    # What this does NOT fix: if body.value requires heavy computation
    # (say, reward normalization across 50k historical steps), that must
    # go to run_in_executor — do NOT do it inline here.

The crucial decisions made here — and why they were made:

⚠ FastAPI does not magically make sync functions async

FastAPI runs async def endpoints directly on the event loop. If you mark an endpoint async def but call blocking code inside it — requests.get(), time.sleep(), psycopg2.execute() — you have blocked the event loop. FastAPI will run plain def (sync) endpoints in a thread pool, which is safer for truly blocking code, but you lose async concurrency for them. The correct path is async throughout with async-native libraries.

WEAKNESSWhere async bites engineers — the real failure modes

✗ Blocking the event loop with sync calls

The cardinal sin, and it's entirely silent. time.sleep(0.1) inside an async handler doesn't crash, doesn't warn, doesn't emit a log line — it just makes every other user in the system wait 100 ms they have no idea they're waiting. Tools that catch this: asyncio.set_debug(True) enables slow-callback detection (logs a warning when any callback takes more than 100 ms). The aiomonitor library attaches a live inspector to a running event loop. In production, histogram-bucket your request latencies by percentile: if p99 spikes while p50 is fine, a blocking call is the first hypothesis.

✗ Mixing async and sync worlds incorrectly

Two common mistakes: (1) calling asyncio.run() inside a function that's already inside a running event loop — this raises RuntimeError: This event loop is already running. Jupyter notebooks suffer from this because they run their own event loop. Fix: use await coro directly in Jupyter, or nest_asyncio as a workaround. (2) Passing a coroutine object to something expecting a value — result = my_async_func() — you get back a coroutine object, not the result. The coroutine runs nothing until driven. Python emits RuntimeWarning: coroutine was never awaited since Python 3.4+ when the coroutine object is garbage-collected unawaited — it is easy to miss in complex async code; asyncio.set_debug(True) surfaces additional diagnostics.

✗ Applying async to CPU-bound work

The most conceptually damaging mistake: reaching for asyncio.gather() to speed up CPU-heavy computations. If you have 8 coroutines each doing 100 ms of pure Python arithmetic, gather runs them serially on one thread — total time is still ~800 ms, plus overhead. Async does not parallelize CPU work. For real CPU parallelism in Python: multiprocessing, concurrent.futures.ProcessPoolExecutor, or step outside Python entirely (Cython, Rust extension, a dedicated worker process). An async wrapper around CPU work is the worst of both worlds: the complexity of async with none of the concurrency benefits.

KEY INSIGHTHow this connects

  • Concurrency I: Threads (3.7) — the contrast is explicit: threads are preemptive and OS-scheduled; async is cooperative and event-loop scheduled. Both address concurrency but at different cost structures and for different problem shapes. If you haven't read 3-7, the OS contract for threads and the GIL give context that makes async's tradeoffs clearer.
  • The OS Contract (1.3) — the epoll/kqueue syscalls that power the event loop are exactly the OS I/O multiplexing mechanisms introduced there. select(), poll(), epoll(), and kqueue() are the kernel primitives every event loop wraps. Async is these syscalls, given a pleasant coroutine API.
  • AuthN and AuthZ (3.9) → — TraceLab now has an async ingest endpoint. Next up: who is allowed to ingest, and how the API verifies identity. Token authentication in an async context means awaiting a database lookup on each request — the patterns here apply directly there.

The through-line for Stage 3: every layer we've built — HTTP, API design, server internals, relational DB, queries, transactions, threads, async — is a mechanism you can inspect and break. Stage 4 takes the TraceLab v2 God module that's been accumulating debt and architects it deliberately: the same machinery, organized so the 2 a.m. incident has a clear owner.

GLOSSARYKey terms introduced

Event loop
A single-threaded loop that polls I/O readiness (via epoll/kqueue/select), fires callbacks for ready file descriptors and expired timers, and runs scheduled callbacks — the scheduler for cooperative async concurrency.
Coroutine
A function defined with async def whose execution can be suspended at await points and resumed later. Implemented as a heap-allocated frame object — the same mechanism as Python generators.
Task (asyncio)
An event-loop-managed wrapper around a coroutine. Created with asyncio.create_task(); the loop drives it autonomously, concurrently with other Tasks. Distinguishes scheduled-concurrency from sequential coroutine chaining.
Cooperative scheduling
A concurrency model where each task runs until it voluntarily yields (via await). The scheduler never interrupts a running task — contrast with preemptive OS scheduling of threads.
Structured concurrency
The discipline that task lifetimes are scoped to a block: all spawned tasks must complete or be cancelled before the block exits. Prevents orphaned tasks and makes cancellation and error propagation deterministic. Implemented by asyncio.TaskGroup and trio nurseries.
Blocking call
Any function call that makes the OS park the current thread until an operation completes — time.sleep(), synchronous socket reads, blocking file I/O. Inside an async handler this stalls the event loop because the loop and your code share the same OS thread.
epoll / kqueue / IOCP
OS-level I/O multiplexing interfaces. You register a set of file descriptors and ask "which are ready?" — the kernel returns only the ready ones, enabling one thread to monitor thousands of connections efficiently. The kernel-level mechanism beneath every async I/O framework.
asyncio.to_thread()
Python 3.9+ API that runs a blocking (or CPU-bound) function in a thread pool and returns an awaitable. Lets async code delegate non-async work without blocking the event loop.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1What does await asyncio.sleep(1) actually do, step by step, inside the event loop?›

The coroutine registers a timer callback with the event loop scheduled 1 second from now, then suspends itself — returning control to the event loop. The loop continues running other ready coroutines and processing I/O. After 1 second the timer fires, the callback is placed in the ready queue, and on the next loop tick the original coroutine resumes execution at the line following the await. The OS thread never parks — only the coroutine pauses.

2You have a FastAPI endpoint that calls time.sleep(2). It's declared async def. What happens when 10 requests arrive simultaneously?›

time.sleep(2) parks the OS thread for 2 seconds. The event loop stops turning. All 9 other requests wait in the queue — no I/O is processed, no responses sent, no health checks served. Total time for all 10 requests: approximately 20 seconds (each waits for the previous to unblock). The fix: use await asyncio.sleep(2), or move the blocking operation to a thread pool with await asyncio.to_thread(blocking_fn, args).

3Why doesn't asyncio.gather(cpu_heavy_1(), cpu_heavy_2(), cpu_heavy_3()) speed up CPU-bound work?›

The event loop runs on a single OS thread. CPU-bound coroutines that never await anything will each run to completion sequentially — gather can only interleave tasks at await points. With no yields, the loop dispatches task 1 through to completion, then task 2, then task 3. Total time equals the sum of all three. And even if they did yield, the GIL would prevent multiple Python threads from running Python bytecode in parallel. For CPU parallelism: ProcessPoolExecutor with loop.run_in_executor().

4What is the difference between await some_coro() and asyncio.create_task(some_coro())?›

await some_coro() runs the coroutine sequentially in the current coroutine — current execution suspends and waits for the result before continuing. No other code runs concurrently during this time (except whatever some_coro itself awaits). asyncio.create_task(some_coro()) wraps the coroutine in a Task and schedules it on the event loop, then immediately returns the Task object. The task runs concurrently with the current coroutine — both make progress during each other's await points. create_task is the gateway to actual async concurrency; await alone is sequential.

Next: who's allowed in — token authentication for the TraceLab API →