ProjekinAja. Software Engineering Stage 4 · Architecture Lesson 4.4
● Stage 4 · Lesson 4.4

Queues and Async Work: Decoupling in Time

The ingest path must never block on the slow work — backpressure, at-least-once delivery, idempotent consumers, and dead-letter queues.

⏱ ~22 min read 🏗 Architecture decision 🔥 Failure-mode heavy

🎯 By the end you'll understand

Why a synchronous ingest path breaks under bursty load. What at-least-once, at-most-once, and exactly-once delivery actually mean and which is achievable without distributed transactions. How idempotency keys (introduced in Lesson 3-2 for HTTP) pay off here as consumer idempotency. What backpressure is, how queues can hide overload until they explode, and how a dead-letter queue makes the invisible visible. Where to draw the line between queuing and over-engineering — and what TraceLab's ingest module looks like once a queue is in front of it.

KEY INSIGHTA queue is a promise to do work later

The robot's telemetry burst at 10 kHz should never wait on your database write, your metric rollup, or your alert evaluation. A queue lets the sender say "noted" and move on — but it shifts all the hard problems downstream.

The synchronous model feels natural: a client sends a request, the server does the work, the server replies. It breaks the moment the work takes longer than the client can afford to wait, or arrives faster than the worker can process. Message queues are the standard answer. They decouple the ingest rate from the processing rate. They buffer bursts. They let workers scale independently. But they are not free: you trade synchronous simplicity for a new class of problems — delivery guarantees, duplicates, ordering, poison messages, and the seductive lie that an unbounded queue means you're keeping up.

🔑 Hold onto this

A queue decouples producer from consumer in time: the producer enqueues and moves on; the consumer processes when ready. This is powerful, but it re-introduces every hard problem distributed systems normally hide: what if the consumer crashes mid-message? The broker must redeliver — which means the consumer will see the same message twice. The only safe response is to make processing idempotent: applying the same message twice produces the same state as applying it once. The idempotency key you applied to HTTP in Lesson 3-2 is the exact same idea, carried into the queue.

HOW IT WORKSWhy a synchronous ingest path breaks

TraceLab's telemetry ingest endpoint currently does this synchronously:

ingest.py (synchronous — before queuing)
def post_metric(run_id, key, value):
    # 1. validate
    _validate(run_id, key, value)      # ~1 ms
    # 2. persist to DB
    db.insert(run_id, key, value)       # ~8 ms p50, 400 ms p99
    # 3. update rollup
    cache.invalidate(run_id)           # ~2 ms + cache miss cascade
    # 4. check alert rules
    evaluate_alerts(run_id, key, value) # ~15 ms, CPU-bound
    return "ok"

At 10 RPS this is fine. A robot bursts to 400 RPS for three seconds — 1,200 requests. The DB is the bottleneck at ~100 concurrent writes before latency degrades. The endpoint latency goes from 26 ms to several hundred milliseconds. Clients time out and retry, doubling the load. This is a thundering-herd collapse caused by coupling ingest throughput to storage throughput.

The queue fix: post_metric validates and enqueues. It returns 202 Accepted — acknowledged, not done — in under 2 ms. A separate pool of workers drains the queue at a controlled rate. The burst is absorbed by the queue depth; latency stays flat for the caller; the DB sees a steady, bounded write rate regardless of producer spikes.

1

Producer — fast path only

Validates the message and enqueues it. Returns immediately. No DB writes, no cache operations, no alert evaluation. The caller gets confirmation that the message is durably queued, not that it is processed.

2

Broker — durable buffer

Stores messages until a consumer acknowledges them. If the broker crashes before a consumer acks, the message is redelivered on broker restart. This is the source of at-least-once semantics: the broker guarantees you'll see it, but may deliver it more than once.

3

Consumer — slow path, controlled concurrency

Pulls messages, does the expensive work (DB write, rollup, alerts), then acks. If it crashes between processing and acking, the broker redeliveries. The consumer must handle this gracefully — which means idempotency.

Robot fleet producer HTTP API producer Message Broker [msg0][msg1][msg2]→ durable queue + ack worker-0 worker-1 worker-2 DB + Cache controlled write rate Dead-Letter Queue (DLQ) after N retries
Queue-based ingest: producers enqueue and return 202 immediately; workers drain at a controlled rate; failed messages route to a DLQ instead of disappearing.

CORE CONCEPTDelivery semantics: the three promises

When a broker says a message is delivered, it makes one of three promises. Only one of them is achievable without distributed transactions, and understanding why is essential before you can reason about correctness.

SemanticWhat it meansHow it breaksAchievable without 2PC?
At-most-once The broker fires and forgets. No ack, no retry. Best throughput. Worker crashes before processing? Message is gone forever. Yes — the default when you don't ack.
At-least-once The broker holds the message until the consumer acks. On timeout or crash, it redelivers. Consumer processes but crashes before acking. Same message processed twice. This is the normal operating mode of every real queue. Yes — ack after processing. Requires idempotent consumers.
Exactly-once Every message processed exactly once, even across crashes. Requires atomic: "mark message consumed" + "apply effect." That's a two-phase commit across two systems. No — not in the general case. Kafka's exactly-once works only within Kafka itself. External effects cannot be made exactly-once without idempotent writes.
⚠ "Exactly-once" is a marketing claim

Kafka's transactional producer/consumer can give exactly-once within the Kafka log. But the moment you write to a database, call an external API, or send an email, you're outside the Kafka transaction. A crash after the DB write but before the Kafka commit means the message is replayed — which hits the DB again. The only general solution is: use at-least-once delivery + idempotent consumers. This is not a limitation to route around; it is the correct architecture.

The practical upshot: design all consumers as if they will receive every message at least twice. For TraceLab's metric ingest, that means a metric write with the same (run_id, key, timestamp) triple is a no-op if it already exists — the idempotency key pattern from Lesson 3-2, applied to the database write.

consumer.py — idempotent metric write
def process_message(msg):
    # idempotency key = (run_id, metric_key, timestamp_ms)
    # INSERT OR IGNORE: if row already exists, silently skip
    db.execute(
        """INSERT OR IGNORE INTO metrics
              (run_id, key, value, ts)
           VALUES (?, ?, ?, ?)""",
        (msg['run_id'], msg['key'], msg['value'], msg['ts'])
    )
    # ack ONLY after successful write
    broker.ack(msg['id'])
    # if we crash here: broker redelivers, INSERT OR IGNORE skips duplicate → correct
💡 Ack position matters enormously

Acking before processing (fire-and-forget) gives at-most-once: crash after ack = lost. Acking after processing gives at-least-once: crash before ack = redelivery = possible duplicate. The idempotency guarantee is what turns "at-least-once delivery + idempotent consumer" into the practical equivalent of exactly-once outcomes, without distributed transactions.

SEE ITQueue / backpressure simulator

The widget below simulates TraceLab's ingest queue over 60 ticks of simulated time. Tune producer burst size, consumer count, and retry behavior; then crash a consumer mid-run and watch redelivery create duplicate metrics — until you toggle idempotency on.

⚙ Interactive · Queue & backpressure simulator

Each tick: producers enqueue a burst; each live consumer processes one message per tick. Consumer crash at tick 20 (if enabled). Idempotency toggle controls whether duplicates land in the DB or are silently skipped.

QUEUE DEPTH
0
DB ROWS (total writes)
0
DUPLICATES IN DB
0
DLQ DEPTH
0
MSGS PROCESSED
0
TICK
0 / 60
Queue depth
0
DLQ
0
DB duplicates
0
Run the simulation to see the event log.

Configure above and run. Watch queue depth spike when burst > consumers × processing rate. Enable crash to see redelivery; toggle idempotency to see duplicates appear and disappear.

Key observations to take away: with idempotency off and a crash enabled, the DB accumulates duplicates for every redelivered message — data corruption that is invisible until someone notices their loss metrics are double. With idempotency on, the second write to the same (run_id, key, ts) is silently skipped; the DB row count is correct. The DLQ fills when a poison message exceeds its retry limit — it does not drain the queue, which is the point: the rest of the stream keeps flowing.

IN PRACTICEIdempotent consumers: the correct mental model

An idempotent operation produces the same result no matter how many times you apply it. The classic examples from math: |x| (absolute value), x AND x = x. In distributed systems: a DB write guarded by a unique constraint, a cache SET (overwriting with the same value is harmless), a DELETE that returns success whether or not the row existed.

The non-examples that will hurt you: appending to a list, incrementing a counter, sending an email. If your consumer appends a row on every delivery and gets redelivered, you get a duplicate row. The fix is to structure the write so re-applying it has no additional effect:

EffectNon-idempotent formIdempotent form
Write a metric INSERT INTO metrics ... INSERT OR IGNORE INTO metrics ... with a unique constraint on (run_id, key, ts)
Update a count UPDATE runs SET count = count + 1 Replace event-derived count with a snapshot derived from canonical events: SELECT COUNT(*) FROM events WHERE run_id=?
Send a notification send_email(user, subject) Gate on a deduplication table: INSERT OR IGNORE INTO sent_notifications (msg_id) first; email only if insert succeeded.
Charge a customer stripe.charge(amount) Pass an idempotency key to Stripe: stripe.charge(amount, idempotency_key=msg_id) — Stripe stores the result and returns the cached response on replay.
📖 Idempotency key vs deduplication key

An idempotency key is a client-supplied token that says "this is the same logical operation." The server stores the key and returns the cached result on replay. Introduced in Lesson 3-2 for safe HTTP retries. A deduplication key is the same concept on the broker side: if a producer sends the same message ID twice (e.g., after a network hiccup), the broker drops the second copy before it ever reaches a consumer. Both protect against duplication; they operate at different layers.

CORE CONCEPTBackpressure and dead-letter queues

A queue absorbs bursts. But what happens when the producer is systematically faster than the consumer? The queue grows. If the queue is unbounded, it grows until memory is exhausted and the process crashes — losing everything in the queue. If the queue is bounded, producers must eventually block or drop. Both are better outcomes than silent accumulation, but neither is invisible. Backpressure is the mechanism by which a full queue signals to its upstream that they need to slow down or wait.

Real systems choose one of three responses to a full queue:

💡 Queue depth is your leading indicator

In a well-instrumented system, the queue depth metric tells you about overload before it becomes an outage. A steadily growing queue at 02:00 on a Tuesday is not "fine" just because nothing has crashed yet — it is the overload in slow motion. Alert on queue depth, not just errors. A good rule: if the queue depth exceeds (consumer throughput × target latency budget), you are already behind schedule.

Dead-letter queues: making failures visible

Some messages cannot be processed no matter how many times they are retried. A malformed message that fails JSON parsing. A metric referencing a run_id that was deleted between enqueue and processing. A downstream service that is permanently unavailable. After N delivery attempts (configurable per queue), the broker moves the message to a dead-letter queue (DLQ). This is not deletion — it is parking in a place where humans or monitoring can inspect it.

Without a DLQ, a poison message that throws an exception on every processing attempt will loop forever, blocking the consumer or consuming retries. With a DLQ, the poison message exits the main flow after N retries, the consumer keeps processing the rest of the queue, and an engineer can inspect the DLQ at leisure to fix the root cause.

⚠ A DLQ is a quarantine, not a drain

Teams discover the DLQ contains 40,000 messages six months after launch. A DLQ that grows without alarm is just a slow data sink. Alert when the DLQ is non-empty. Define a playbook: what does a DLQ message mean? Who owns it? What do you do with a metric you can't write because the run was deleted — discard it, re-create the run, or escalate? The DLQ is only useful if you have a documented response.

Retry strategies: exponential backoff with jitter

Retrying immediately after a failure hits the same broken thing at the same moment. The standard pattern is exponential backoff with jitter: wait a random interval selected from [0, min(cap, base × 2^attempt)] seconds before each retry. The jitter prevents N consumers from retrying simultaneously (a retry thundering herd), which would amplify the very overload that caused the failure. AWS, GCP, and Azure all document this pattern explicitly in their SDK retry guides.

retry.py — exponential backoff with jitter
import time, random

def backoff_delay(attempt, base=1.0, cap=60.0):
    # full jitter: uniform [0, min(cap, base * 2**attempt)]
    ceiling = min(cap, base * (2 ** attempt))
    return random.uniform(0, ceiling)

def process_with_retry(msg, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            process_message(msg)
            return  # success
        except TransientError:
            if attempt < max_attempts - 1:
                time.sleep(backoff_delay(attempt))
        except PoisonError:
            break  # non-retryable — send to DLQ immediately
    broker.nack(msg['id'], dlq=True)

TRACELABTraceLab's ingest queue: what changes and what doesn't

TraceLab v2 has a synchronous ingest endpoint. Here is the queue-based architecture we add as part of Stage 4's TraceLab v3 milestone. The key design decisions:

tracelab/ingest_queue.py — v3 sketch
import queue, threading, time, json
from tracelab.db import get_db

# in-process queue; swap for Redis/RabbitMQ when multi-process needed
_q = queue.Queue(maxsize=10_000)  # bounded — backpressure via Full exception
_MAX_RETRIES = 3

def enqueue(run_id, key, value, ts_ms, msg_id):
    try:
        _q.put_nowait({
            'run_id': run_id, 'key': key, 'value': value,
            'ts_ms': ts_ms, 'msg_id': msg_id, 'attempts': 0
        })
    except queue.Full:
        raise RuntimeError("Ingest queue full — apply backpressure to callers")

def _worker():
    db = get_db()
    while True:
        msg = _q.get()
        try:
            # idempotent write: unique constraint on (run_id, key, ts_ms)
            db.execute(
                "INSERT OR IGNORE INTO metrics (run_id,key,value,ts_ms,msg_id) VALUES(?,?,?,?,?)",
                (msg['run_id'], msg['key'], msg['value'], msg['ts_ms'], msg['msg_id'])
            )
            db.commit()
        except Exception as exc:
            msg['attempts'] += 1
            if msg['attempts'] < _MAX_RETRIES:
                time.sleep(0.1 * 2 ** msg['attempts'])
                _q.put(msg)   # re-enqueue for retry
            else:
                _send_to_dlq(msg, str(exc))
        finally:
            _q.task_done()

def start_workers(n=4):
    for _ in range(n):
        t = threading.Thread(target=_worker, daemon=True)
        t.start()

This design deliberately uses Python's standard-library queue.Queue rather than an external broker. For TraceLab's scale (one to a few robots, one team), an external broker like RabbitMQ or Redis Streams is over-engineering. The queue is bounded (backpressure via queue.Full), workers are daemon threads that die with the process, and the DLQ is a database table. When TraceLab needs to survive a process restart with messages intact — the moment the queue is "real" infrastructure — it graduates to an external broker. Not before.

💡 The idempotency key from Lesson 3-2 pays off here

In Lesson 3-2 we gave every HTTP mutation request an Idempotency-Key header that the server stored to make retries safe. That key becomes the msg_id here: the producer generates it, includes it in the queue message, and the consumer's unique constraint on msg_id means the second delivery is a no-op. One concept, two layers, same invariant.

WEAKNESSWhere this bites

✗ Unbounded queues that hide overload until explosion

The most common production failure mode: the queue is declared without a maxsize, so during an incident the producer keeps enqueueing, the queue grows to millions of items, the process hits its memory limit, and crashes — taking the entire queue with it. Now you have lost hours of data and the service is down. Always bound your queues. The right maxsize should be derived from how much memory you can afford and how much latency the caller can tolerate when the queue is full, not from "infinity sounds safe."

✗ Consumers that are not idempotent

The most common correctness failure: deploying at-least-once delivery (the only practical choice) with consumers that were designed for exactly-once delivery. The team says "we tested it and there are no duplicates." There are no duplicates in development because nothing crashes cleanly in development. In production, the first worker restart during a deploy creates a batch of duplicates that silently inflates every metric. Treat every consumer as if it will see each message twice, from day one, regardless of how rarely it actually happens.

✗ Using a queue to avoid fixing the slow consumer

Queues defer work; they do not eliminate it. If your consumer writes take 400 ms each and your producer generates 10 messages per second, a queue gives you time to notice — it does not solve the problem. The queue will reach its bound in (maxsize / net deficit rate) seconds and then block the producer anyway. The correct fix is to make the consumer faster or add more consumers. Queues buy you time for an incident response; they are not a substitute for capacity planning.

KEY INSIGHTHow this connects

  • Idempotency keys (3.2) — the HTTP-level idempotency key introduced there is the same concept applied here as consumer-level deduplication. The debt from that lesson pays off directly.
  • Caching (4.3) — the cache-invalidation call removed from the synchronous ingest path is now a deferred consumer step. The two lessons share the same ingest refactor.
  • Distributed Systems Reality (4.5) — next. Queues are one distributed component; Lesson 4.5 generalizes: network partitions, consensus, the CAP theorem, and the full failure-mode taxonomy that queues introduce.

Everything in this lesson is a consequence of one fact: distributed work means the producer and consumer run in different processes, possibly on different machines, and either can fail at any moment. Once you accept that, idempotency and at-least-once delivery stop being optional polish and start being the only safe default.

GLOSSARYKey terms

At-least-once delivery
A delivery guarantee: the broker redelivers until the consumer acks. Every message is delivered one or more times; requires idempotent consumers to prevent duplication effects.
At-most-once delivery
The broker fires and forgets: no ack, no retry. Messages may be lost on consumer crash. Suitable for non-critical, high-throughput, lossy streams (metrics sampling, logs).
Idempotent consumer
A message consumer whose processing effect is the same whether it receives a message once or many times. Usually implemented with unique constraints and INSERT OR IGNORE semantics.
Backpressure
The signal propagated upstream when a queue or buffer is full, telling producers to slow down or wait. Prevents unbounded memory growth at the cost of producer latency.
Dead-letter queue (DLQ)
A secondary queue where messages are moved after exceeding their maximum retry count. Allows the main queue to keep flowing while preserving failed messages for inspection.
Poison message
A message that causes the consumer to fail on every delivery attempt — malformed payload, referencing deleted data, triggering a bug. Without a DLQ, a poison message stalls the entire queue.
Exponential backoff with jitter
A retry delay strategy: wait a random duration drawn from [0, min(cap, base × 2^attempt)]. Prevents retry thundering herds where many consumers hit a recovering service simultaneously.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Why does at-least-once delivery require idempotent consumers, and what does "idempotent" mean concretely for a database write?›

At-least-once delivery redelivers a message whenever a consumer crashes before acking. Without idempotency, the second delivery causes a second effect — a duplicate DB row, a duplicate charge, a duplicate email. Concretely for a metric write: the table has a unique constraint on (run_id, key, ts_ms) and the INSERT uses INSERT OR IGNORE. The second delivery attempts the same INSERT, the unique constraint fires, the row is skipped — the DB state is identical to after the first delivery. That's idempotency: same input, same outcome, regardless of how many times you apply it.

2What is backpressure, and why is an unbounded queue dangerous even before it fills?›

Backpressure is the signal that flows upstream when a buffer is full — it tells producers to slow down. An unbounded queue never sends this signal. The producer keeps enqueuing at full speed; the queue grows silently in memory; everything appears healthy on standard error/latency dashboards; and then the process runs out of memory and crashes, losing all queued messages at once. Bounded queues with a measured maxsize force producers to confront overload while the queue is still manageable and recoverable — not after it has become a memory bomb.

3Why is "exactly-once delivery" generally not achievable when the consumer writes to an external database, and what is the practical substitute?›

Exactly-once would require "mark message consumed in broker" and "apply effect in database" to be a single atomic operation across two independent systems. That requires a two-phase commit (2PC) across them. 2PC is slow, fragile, and rarely supported between a message broker and an arbitrary database. The practical substitute: use at-least-once delivery (which the broker can provide alone, without cross-system coordination), and make every consumer effect idempotent — so duplicate deliveries produce the same state as a single delivery. The outcome is effectively equivalent to exactly-once at the data level.

4A poison message is killing your consumer loop. What happens without a DLQ, and what happens with one?›

Without a DLQ: the consumer tries to process the message, fails, nacks, the broker redelivers immediately, the consumer fails again, and so on indefinitely. Either the consumer loops forever on one message (blocking everything behind it) or the retry mechanism burns all its attempts and discards the message — silently losing data. With a DLQ: after N configured retries, the broker moves the message to the DLQ and the consumer continues processing the rest of the queue. The message is preserved for inspection, a monitoring alert fires on DLQ non-empty, and the engineering team can investigate the root cause without any other messages being held hostage.

Next: Distributed Systems Reality — the full failure-mode taxonomy →