Queues and Async Work: Decoupling in Time
The ingest path must never block on the slow work — backpressure, at-least-once delivery, idempotent consumers, and dead-letter queues.
🎯 By the end you'll understand
Why a synchronous ingest path breaks under bursty load. What at-least-once, at-most-once, and exactly-once delivery actually mean and which is achievable without distributed transactions. How idempotency keys (introduced in Lesson 3-2 for HTTP) pay off here as consumer idempotency. What backpressure is, how queues can hide overload until they explode, and how a dead-letter queue makes the invisible visible. Where to draw the line between queuing and over-engineering — and what TraceLab's ingest module looks like once a queue is in front of it.
KEY INSIGHTA queue is a promise to do work later
The robot's telemetry burst at 10 kHz should never wait on your database write, your metric rollup, or your alert evaluation. A queue lets the sender say "noted" and move on — but it shifts all the hard problems downstream.
The synchronous model feels natural: a client sends a request, the server does the work, the server replies. It breaks the moment the work takes longer than the client can afford to wait, or arrives faster than the worker can process. Message queues are the standard answer. They decouple the ingest rate from the processing rate. They buffer bursts. They let workers scale independently. But they are not free: you trade synchronous simplicity for a new class of problems — delivery guarantees, duplicates, ordering, poison messages, and the seductive lie that an unbounded queue means you're keeping up.
A queue decouples producer from consumer in time: the producer enqueues and moves on; the consumer processes when ready. This is powerful, but it re-introduces every hard problem distributed systems normally hide: what if the consumer crashes mid-message? The broker must redeliver — which means the consumer will see the same message twice. The only safe response is to make processing idempotent: applying the same message twice produces the same state as applying it once. The idempotency key you applied to HTTP in Lesson 3-2 is the exact same idea, carried into the queue.
HOW IT WORKSWhy a synchronous ingest path breaks
TraceLab's telemetry ingest endpoint currently does this synchronously:
def post_metric(run_id, key, value): # 1. validate _validate(run_id, key, value) # ~1 ms # 2. persist to DB db.insert(run_id, key, value) # ~8 ms p50, 400 ms p99 # 3. update rollup cache.invalidate(run_id) # ~2 ms + cache miss cascade # 4. check alert rules evaluate_alerts(run_id, key, value) # ~15 ms, CPU-bound return "ok"
At 10 RPS this is fine. A robot bursts to 400 RPS for three seconds — 1,200 requests. The DB is the bottleneck at ~100 concurrent writes before latency degrades. The endpoint latency goes from 26 ms to several hundred milliseconds. Clients time out and retry, doubling the load. This is a thundering-herd collapse caused by coupling ingest throughput to storage throughput.
The queue fix: post_metric validates and enqueues. It returns 202 Accepted — acknowledged, not done — in under 2 ms. A separate pool of workers drains the queue at a controlled rate. The burst is absorbed by the queue depth; latency stays flat for the caller; the DB sees a steady, bounded write rate regardless of producer spikes.
Producer — fast path only
Validates the message and enqueues it. Returns immediately. No DB writes, no cache operations, no alert evaluation. The caller gets confirmation that the message is durably queued, not that it is processed.
Broker — durable buffer
Stores messages until a consumer acknowledges them. If the broker crashes before a consumer acks, the message is redelivered on broker restart. This is the source of at-least-once semantics: the broker guarantees you'll see it, but may deliver it more than once.
Consumer — slow path, controlled concurrency
Pulls messages, does the expensive work (DB write, rollup, alerts), then acks. If it crashes between processing and acking, the broker redeliveries. The consumer must handle this gracefully — which means idempotency.
CORE CONCEPTDelivery semantics: the three promises
When a broker says a message is delivered, it makes one of three promises. Only one of them is achievable without distributed transactions, and understanding why is essential before you can reason about correctness.
| Semantic | What it means | How it breaks | Achievable without 2PC? |
|---|---|---|---|
| At-most-once | The broker fires and forgets. No ack, no retry. Best throughput. | Worker crashes before processing? Message is gone forever. | Yes — the default when you don't ack. |
| At-least-once | The broker holds the message until the consumer acks. On timeout or crash, it redelivers. | Consumer processes but crashes before acking. Same message processed twice. This is the normal operating mode of every real queue. | Yes — ack after processing. Requires idempotent consumers. |
| Exactly-once | Every message processed exactly once, even across crashes. | Requires atomic: "mark message consumed" + "apply effect." That's a two-phase commit across two systems. | No — not in the general case. Kafka's exactly-once works only within Kafka itself. External effects cannot be made exactly-once without idempotent writes. |
Kafka's transactional producer/consumer can give exactly-once within the Kafka log. But the moment you write to a database, call an external API, or send an email, you're outside the Kafka transaction. A crash after the DB write but before the Kafka commit means the message is replayed — which hits the DB again. The only general solution is: use at-least-once delivery + idempotent consumers. This is not a limitation to route around; it is the correct architecture.
The practical upshot: design all consumers as if they will receive every message at least twice. For TraceLab's metric ingest, that means a metric write with the same (run_id, key, timestamp) triple is a no-op if it already exists — the idempotency key pattern from Lesson 3-2, applied to the database write.
def process_message(msg): # idempotency key = (run_id, metric_key, timestamp_ms) # INSERT OR IGNORE: if row already exists, silently skip db.execute( """INSERT OR IGNORE INTO metrics (run_id, key, value, ts) VALUES (?, ?, ?, ?)""", (msg['run_id'], msg['key'], msg['value'], msg['ts']) ) # ack ONLY after successful write broker.ack(msg['id']) # if we crash here: broker redelivers, INSERT OR IGNORE skips duplicate → correct
Acking before processing (fire-and-forget) gives at-most-once: crash after ack = lost. Acking after processing gives at-least-once: crash before ack = redelivery = possible duplicate. The idempotency guarantee is what turns "at-least-once delivery + idempotent consumer" into the practical equivalent of exactly-once outcomes, without distributed transactions.
SEE ITQueue / backpressure simulator
The widget below simulates TraceLab's ingest queue over 60 ticks of simulated time. Tune producer burst size, consumer count, and retry behavior; then crash a consumer mid-run and watch redelivery create duplicate metrics — until you toggle idempotency on.
Key observations to take away: with idempotency off and a crash enabled, the DB accumulates duplicates for every redelivered message — data corruption that is invisible until someone notices their loss metrics are double. With idempotency on, the second write to the same (run_id, key, ts) is silently skipped; the DB row count is correct. The DLQ fills when a poison message exceeds its retry limit — it does not drain the queue, which is the point: the rest of the stream keeps flowing.
IN PRACTICEIdempotent consumers: the correct mental model
An idempotent operation produces the same result no matter how many times you apply it. The classic examples from math: |x| (absolute value), x AND x = x. In distributed systems: a DB write guarded by a unique constraint, a cache SET (overwriting with the same value is harmless), a DELETE that returns success whether or not the row existed.
The non-examples that will hurt you: appending to a list, incrementing a counter, sending an email. If your consumer appends a row on every delivery and gets redelivered, you get a duplicate row. The fix is to structure the write so re-applying it has no additional effect:
| Effect | Non-idempotent form | Idempotent form |
|---|---|---|
| Write a metric | INSERT INTO metrics ... |
INSERT OR IGNORE INTO metrics ... with a unique constraint on (run_id, key, ts) |
| Update a count | UPDATE runs SET count = count + 1 |
Replace event-derived count with a snapshot derived from canonical events: SELECT COUNT(*) FROM events WHERE run_id=? |
| Send a notification | send_email(user, subject) |
Gate on a deduplication table: INSERT OR IGNORE INTO sent_notifications (msg_id) first; email only if insert succeeded. |
| Charge a customer | stripe.charge(amount) |
Pass an idempotency key to Stripe: stripe.charge(amount, idempotency_key=msg_id) — Stripe stores the result and returns the cached response on replay. |
An idempotency key is a client-supplied token that says "this is the same logical operation." The server stores the key and returns the cached result on replay. Introduced in Lesson 3-2 for safe HTTP retries. A deduplication key is the same concept on the broker side: if a producer sends the same message ID twice (e.g., after a network hiccup), the broker drops the second copy before it ever reaches a consumer. Both protect against duplication; they operate at different layers.
CORE CONCEPTBackpressure and dead-letter queues
A queue absorbs bursts. But what happens when the producer is systematically faster than the consumer? The queue grows. If the queue is unbounded, it grows until memory is exhausted and the process crashes — losing everything in the queue. If the queue is bounded, producers must eventually block or drop. Both are better outcomes than silent accumulation, but neither is invisible. Backpressure is the mechanism by which a full queue signals to its upstream that they need to slow down or wait.
Real systems choose one of three responses to a full queue:
- Block — the producer's enqueue call blocks until space is available. Provides natural flow control; risks deadlock if producers and consumers share resources.
- Drop — new messages are discarded. Appropriate for non-critical data (metrics sampling, health pings) where losing a few values is acceptable.
- Error — enqueue returns an error; the caller decides. The right call for ingest paths where every event must be counted: the caller can retry, buffer locally, or raise an alert.
In a well-instrumented system, the queue depth metric tells you about overload before it becomes an outage. A steadily growing queue at 02:00 on a Tuesday is not "fine" just because nothing has crashed yet — it is the overload in slow motion. Alert on queue depth, not just errors. A good rule: if the queue depth exceeds (consumer throughput × target latency budget), you are already behind schedule.
Dead-letter queues: making failures visible
Some messages cannot be processed no matter how many times they are retried. A malformed message that fails JSON parsing. A metric referencing a run_id that was deleted between enqueue and processing. A downstream service that is permanently unavailable. After N delivery attempts (configurable per queue), the broker moves the message to a dead-letter queue (DLQ). This is not deletion — it is parking in a place where humans or monitoring can inspect it.
Without a DLQ, a poison message that throws an exception on every processing attempt will loop forever, blocking the consumer or consuming retries. With a DLQ, the poison message exits the main flow after N retries, the consumer keeps processing the rest of the queue, and an engineer can inspect the DLQ at leisure to fix the root cause.
Teams discover the DLQ contains 40,000 messages six months after launch. A DLQ that grows without alarm is just a slow data sink. Alert when the DLQ is non-empty. Define a playbook: what does a DLQ message mean? Who owns it? What do you do with a metric you can't write because the run was deleted — discard it, re-create the run, or escalate? The DLQ is only useful if you have a documented response.
Retry strategies: exponential backoff with jitter
Retrying immediately after a failure hits the same broken thing at the same moment. The standard pattern is exponential backoff with jitter: wait a random interval selected from [0, min(cap, base × 2^attempt)] seconds before each retry. The jitter prevents N consumers from retrying simultaneously (a retry thundering herd), which would amplify the very overload that caused the failure. AWS, GCP, and Azure all document this pattern explicitly in their SDK retry guides.
import time, random def backoff_delay(attempt, base=1.0, cap=60.0): # full jitter: uniform [0, min(cap, base * 2**attempt)] ceiling = min(cap, base * (2 ** attempt)) return random.uniform(0, ceiling) def process_with_retry(msg, max_attempts=3): for attempt in range(max_attempts): try: process_message(msg) return # success except TransientError: if attempt < max_attempts - 1: time.sleep(backoff_delay(attempt)) except PoisonError: break # non-retryable — send to DLQ immediately broker.nack(msg['id'], dlq=True)
TRACELABTraceLab's ingest queue: what changes and what doesn't
TraceLab v2 has a synchronous ingest endpoint. Here is the queue-based architecture we add as part of Stage 4's TraceLab v3 milestone. The key design decisions:
- The HTTP POST endpoint validates and enqueues; it returns
202 Acceptedwith amessage_id. The client can poll/metrics/{message_id}/statusif it needs to confirm processing. - Workers consume from the queue using a thread pool (not async — the writes are I/O-bound but CPU-simple; a thread pool is simpler and avoids async coloring the whole codebase).
- The
metricstable gets a unique constraint on(run_id, key, ts_ms). Every worker write isINSERT OR IGNORE. Redelivery is harmless. - The DLQ is a separate DB table
dlq_messageswith a JSON payload andfailed_attimestamp — no external broker dependency for TraceLab's scale.
import queue, threading, time, json from tracelab.db import get_db # in-process queue; swap for Redis/RabbitMQ when multi-process needed _q = queue.Queue(maxsize=10_000) # bounded — backpressure via Full exception _MAX_RETRIES = 3 def enqueue(run_id, key, value, ts_ms, msg_id): try: _q.put_nowait({ 'run_id': run_id, 'key': key, 'value': value, 'ts_ms': ts_ms, 'msg_id': msg_id, 'attempts': 0 }) except queue.Full: raise RuntimeError("Ingest queue full — apply backpressure to callers") def _worker(): db = get_db() while True: msg = _q.get() try: # idempotent write: unique constraint on (run_id, key, ts_ms) db.execute( "INSERT OR IGNORE INTO metrics (run_id,key,value,ts_ms,msg_id) VALUES(?,?,?,?,?)", (msg['run_id'], msg['key'], msg['value'], msg['ts_ms'], msg['msg_id']) ) db.commit() except Exception as exc: msg['attempts'] += 1 if msg['attempts'] < _MAX_RETRIES: time.sleep(0.1 * 2 ** msg['attempts']) _q.put(msg) # re-enqueue for retry else: _send_to_dlq(msg, str(exc)) finally: _q.task_done() def start_workers(n=4): for _ in range(n): t = threading.Thread(target=_worker, daemon=True) t.start()
This design deliberately uses Python's standard-library queue.Queue rather than an external broker. For TraceLab's scale (one to a few robots, one team), an external broker like RabbitMQ or Redis Streams is over-engineering. The queue is bounded (backpressure via queue.Full), workers are daemon threads that die with the process, and the DLQ is a database table. When TraceLab needs to survive a process restart with messages intact — the moment the queue is "real" infrastructure — it graduates to an external broker. Not before.
In Lesson 3-2 we gave every HTTP mutation request an Idempotency-Key header that the server stored to make retries safe. That key becomes the msg_id here: the producer generates it, includes it in the queue message, and the consumer's unique constraint on msg_id means the second delivery is a no-op. One concept, two layers, same invariant.
WEAKNESSWhere this bites
The most common production failure mode: the queue is declared without a maxsize, so during an incident the producer keeps enqueueing, the queue grows to millions of items, the process hits its memory limit, and crashes — taking the entire queue with it. Now you have lost hours of data and the service is down. Always bound your queues. The right maxsize should be derived from how much memory you can afford and how much latency the caller can tolerate when the queue is full, not from "infinity sounds safe."
The most common correctness failure: deploying at-least-once delivery (the only practical choice) with consumers that were designed for exactly-once delivery. The team says "we tested it and there are no duplicates." There are no duplicates in development because nothing crashes cleanly in development. In production, the first worker restart during a deploy creates a batch of duplicates that silently inflates every metric. Treat every consumer as if it will see each message twice, from day one, regardless of how rarely it actually happens.
Queues defer work; they do not eliminate it. If your consumer writes take 400 ms each and your producer generates 10 messages per second, a queue gives you time to notice — it does not solve the problem. The queue will reach its bound in (maxsize / net deficit rate) seconds and then block the producer anyway. The correct fix is to make the consumer faster or add more consumers. Queues buy you time for an incident response; they are not a substitute for capacity planning.
KEY INSIGHTHow this connects
- Idempotency keys (3.2) — the HTTP-level idempotency key introduced there is the same concept applied here as consumer-level deduplication. The debt from that lesson pays off directly.
- Caching (4.3) — the cache-invalidation call removed from the synchronous ingest path is now a deferred consumer step. The two lessons share the same ingest refactor.
- Distributed Systems Reality (4.5) — next. Queues are one distributed component; Lesson 4.5 generalizes: network partitions, consensus, the CAP theorem, and the full failure-mode taxonomy that queues introduce.
Everything in this lesson is a consequence of one fact: distributed work means the producer and consumer run in different processes, possibly on different machines, and either can fail at any moment. Once you accept that, idempotency and at-least-once delivery stop being optional polish and start being the only safe default.
GLOSSARYKey terms
- At-least-once delivery
- A delivery guarantee: the broker redelivers until the consumer acks. Every message is delivered one or more times; requires idempotent consumers to prevent duplication effects.
- At-most-once delivery
- The broker fires and forgets: no ack, no retry. Messages may be lost on consumer crash. Suitable for non-critical, high-throughput, lossy streams (metrics sampling, logs).
- Idempotent consumer
- A message consumer whose processing effect is the same whether it receives a message once or many times. Usually implemented with unique constraints and INSERT OR IGNORE semantics.
- Backpressure
- The signal propagated upstream when a queue or buffer is full, telling producers to slow down or wait. Prevents unbounded memory growth at the cost of producer latency.
- Dead-letter queue (DLQ)
- A secondary queue where messages are moved after exceeding their maximum retry count. Allows the main queue to keep flowing while preserving failed messages for inspection.
- Poison message
- A message that causes the consumer to fail on every delivery attempt — malformed payload, referencing deleted data, triggering a bug. Without a DLQ, a poison message stalls the entire queue.
- Exponential backoff with jitter
- A retry delay strategy: wait a random duration drawn from [0, min(cap, base × 2^attempt)]. Prevents retry thundering herds where many consumers hit a recovering service simultaneously.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Why does at-least-once delivery require idempotent consumers, and what does "idempotent" mean concretely for a database write?›
At-least-once delivery redelivers a message whenever a consumer crashes before acking. Without idempotency, the second delivery causes a second effect — a duplicate DB row, a duplicate charge, a duplicate email. Concretely for a metric write: the table has a unique constraint on (run_id, key, ts_ms) and the INSERT uses INSERT OR IGNORE. The second delivery attempts the same INSERT, the unique constraint fires, the row is skipped — the DB state is identical to after the first delivery. That's idempotency: same input, same outcome, regardless of how many times you apply it.
2What is backpressure, and why is an unbounded queue dangerous even before it fills?›
Backpressure is the signal that flows upstream when a buffer is full — it tells producers to slow down. An unbounded queue never sends this signal. The producer keeps enqueuing at full speed; the queue grows silently in memory; everything appears healthy on standard error/latency dashboards; and then the process runs out of memory and crashes, losing all queued messages at once. Bounded queues with a measured maxsize force producers to confront overload while the queue is still manageable and recoverable — not after it has become a memory bomb.
3Why is "exactly-once delivery" generally not achievable when the consumer writes to an external database, and what is the practical substitute?›
Exactly-once would require "mark message consumed in broker" and "apply effect in database" to be a single atomic operation across two independent systems. That requires a two-phase commit (2PC) across them. 2PC is slow, fragile, and rarely supported between a message broker and an arbitrary database. The practical substitute: use at-least-once delivery (which the broker can provide alone, without cross-system coordination), and make every consumer effect idempotent — so duplicate deliveries produce the same state as a single delivery. The outcome is effectively equivalent to exactly-once at the data level.
4A poison message is killing your consumer loop. What happens without a DLQ, and what happens with one?›
Without a DLQ: the consumer tries to process the message, fails, nacks, the broker redelivers immediately, the consumer fails again, and so on indefinitely. Either the consumer loops forever on one message (blocking everything behind it) or the retry mechanism burns all its attempts and discards the message — silently losing data. With a DLQ: after N configured retries, the broker moves the message to the DLQ and the consumer continues processing the rest of the queue. The message is preserved for inspection, a monitoring alert fires on DLQ non-empty, and the engineering team can investigate the root cause without any other messages being held hostage.