ProjekinAja. Software Engineering Stage 5 · Shipping & Running Lesson 5.7
● Stage 5 · Lesson 5.7

Observability: Logs, Metrics, Traces

Instrumenting so that 2 a.m.-you can answer questions current-you hasn't thought of — and dogfooding the very product TraceLab is.

⏱ ~22 min read 🔭 Three-pillar observability 🔧 TraceLab v4 milestone

🎯 By the end you'll understand

What logs, metrics, and traces each can and cannot answer — and why you need all three. Structured logging with correlation IDs (and the cardinality trap that sinks your log budget). RED and USE as metric shapes for services and resources. Why p99 matters and means can silently lie. How a distributed trace waterfall exposes N+1 queries and serial-when-parallel spans that no metric or log alone can reveal. And you'll instrument TraceLab with the exact primitives it is supposed to teach — dogfooding as a design constraint.

KEY INSIGHTObservability is not monitoring — it is the ability to ask new questions

Monitoring tells you that something is wrong. Observability tells you why — even for failures you did not predict before they happened.

A monitoring dashboard with twelve graphs is not observability. It is twelve specific questions asked in advance. If production fails in a way that touches a thirteenth dimension, the dashboard is silent while users scream. Real observability means your system emits enough raw signal that you can explore its state after the fact — drilling from a metric anomaly into the logs for that time window, correlating those logs to a specific trace, and finding the one slow DB call that is the actual cause.

🔑 Hold onto this

The three signals are not interchangeable. Logs record events — discrete things that happened, with context. Metrics record state over time — numbers you can aggregate, graph, and alert on. Traces record causality — the parent-child call graph for a single request as it crosses service boundaries. A production incident almost always requires all three: a metric tells you when latency spiked, a log tells you what request hit the spike, and a trace tells you which span inside that request was responsible. Knowing which pillar answers which question is the skill this lesson builds.

CORE CONCEPTThe three signals and what each can and cannot answer

SignalWhat it recordsCan answerCannot answer (alone)
Logs Discrete events with key-value context: timestamp, level, message, request ID, user ID, error detail What happened to this one request? What was the error message? What was the payload that triggered the bug? How often is this happening? Is it getting worse? Which service in a distributed call chain caused it?
Metrics Numeric measurements over time: counters, gauges, histograms. Aggregated and cheap to store. How many requests per second? What is the error rate? Is memory growing? How does p99 latency compare to last week? Why is this specific request slow? What was in the payload? Which user was affected?
Traces The causal tree of a single request: span per operation, parent-child relationships, timing per hop, cross-service propagation via trace headers Where did this request actually spend its time? Did the DB query hit the N+1 path? Is a fan-out supposed to be parallel but running serially? What is the aggregate error rate? Is this pattern common or a one-off? What did the user see?

The practical upshot: metrics get you to the time window, logs get you to the request, traces get you to the span. Each narrows the search space for the next. Skipping any one pillar means your incident triage will, at some point, stall in a place only that pillar could have resolved.

HOW IT WORKSStructured logs and correlation IDs

The printf era of logging — print(f"Processing {run_id}") scattered across a codebase — was not without merit in a single-process world. It fails as soon as two concurrent requests interleave their output, as soon as you need to query across log fields, and as soon as a machine needs to parse what a human typed. Structured logging is the fix: emit JSON (or logfmt) with explicit fields, and let downstream tools index them.

bad_logging.pydon't do this
# Unstructured: grep-only, no machine-readable fields, interleaved in concurrent requests
print(f"Starting ingestion for run {run_id}")
print(f"Inserted 47 rows")
print(f"ERROR: database timeout after 30s")
structured_logging.pydo this
import logging, json, uuid

class JSONFormatter(logging.Formatter):
    def format(self, record):
        d = {
            "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S.%fZ"),
            "level": record.levelname,
            "msg": record.getMessage(),
        }
        if hasattr(record, "extra"):
            d.update(record.extra)
        return json.dumps(d)

# In request handler — attach correlation ID once, carry everywhere:
def handle_ingest(run_id: str):
    correlation_id = str(uuid.uuid4())
    log = logging.getLogger("tracelab")
    extra = {"run_id": run_id, "correlation_id": correlation_id}

    log.info("ingestion.started", extra={"extra": extra})
    rows = do_insert(run_id)
    log.info("ingestion.complete", extra={"extra": {"rows": rows, **extra}})
    # Now you can filter: {"correlation_id": "abc-123"} and get the whole story.

The correlation ID is the load-bearing field. When a single user request fans out across an API server, a Celery worker, and three database calls, each of those processes writes its own logs — but if every entry carries the same correlation ID, you can reconstruct the whole story in any log aggregation tool by filtering on that one field. Without it, logs from concurrent requests interleave into noise.

⚠ The cardinality trap

High-cardinality fields in structured logs are fine — user IDs, request IDs, even full SQL queries. High-cardinality label sets on metrics are budget-breaking. If you add user_id as a Prometheus label, you get one time series per user. At 100k users that is 100k time series — metric storage bloats, query latency spikes, and your Prometheus instance OOMs. The rule: logs absorb high cardinality; metrics stay low-cardinality (service, endpoint, status code, not individual IDs).

In Python, the standard logging module is sufficient; structlog adds processor chains if you want context binding without threading locals. What matters is the output format: write JSON to stdout, let your container runtime or log shipper (Fluent Bit, Vector) forward it to your aggregation backend (Loki, Elasticsearch, CloudWatch Logs). The service should not know or care where logs go.

CORE CONCEPTMetrics: RED, USE, and why percentiles tell the truth

Two complementary mental models cover most of what you need to monitor in a web service stack. Neither is new or contested — they are just useful shapes for asking "what numbers should I record?"

RED

RED — for services (request-oriented)

Rate (requests/sec), Errors (error rate or count), Duration (latency distribution). Introduced in the context of microservices monitoring. If you have these three for every service endpoint, you can answer: is the service healthy? is it slow? is it failing? The model fits any system that handles discrete requests: HTTP APIs, gRPC services, background workers, even database queries.

USE

USE — for resources (capacity-oriented)

Utilization (fraction of time the resource is busy), Saturation (backlog or queue depth), Errors. Formalized by Brendan Gregg. Fits non-request resources: CPUs, disks, memory, network interfaces, connection pools, thread pools. A CPU at 95% utilization is a warning; a CPU at 95% utilization with a scheduler run-queue depth of 20 is a problem now.

Percentiles over averages

The most common metric mistake after "not recording metrics at all" is averaging latency. Consider TraceLab's ingestion endpoint. Your mean response time is 120 ms. Sounds fine. But your p99 is 4,200 ms, because 1% of requests hit the N+1 query path and fire forty-seven sequential DB queries. The average hides the bad tail entirely. The users in that 1% experience the product as broken. The mean is where truth goes to die.

request count latency (ms) mean 120ms p99 4,200ms 50 150 250 400 4,200
The mean latency (120 ms) sits in the bulk of the distribution and looks fine. The p99 (4,200 ms) is in the sparse tail — the users there see a broken product. Averaging hides the tail entirely.

Record latency as a histogram (Prometheus Summary or Histogram) and query p50, p95, p99, and p999 separately. The Prometheus convention is to expose a histogram at the service boundary; your alerting rules fire on p99 > some threshold, not on average. For SLOs (covered in Lesson 5.8), the SLI is almost always defined over a percentile.

💡 p99 is where users live

A common rule of thumb: track p50 (typical user), p95 (bad-luck user), p99 (very-bad-luck user), and p999 (your next incident report). If p50 is fine and p99 is broken, you have a bi-modal distribution — two different code paths, one fast and one slow. That is diagnostic information a mean erases completely. TraceLab's N+1 query is exactly this pattern: most runs hit the indexed path (fast), a few trigger the full-scan path (slow).

HOW IT WORKSDistributed traces: the call graph you were missing

A trace is a tree of spans. A span records: service name, operation name, start time, duration, status, and any attached key-value attributes. When an API call hands off to a queue, which a worker consumes, which queries a database, each step creates a child span. The whole tree shares a trace ID — a random 128-bit identifier propagated via HTTP headers (the W3C traceparent header, or the older B3 format). When the trace is complete, a tracing backend (Jaeger, Tempo, Honeycomb, AWS X-Ray) assembles the spans into a waterfall diagram.

0 ms 240 ms 480 ms API /ingest 480ms queue.publish 10ms worker.ingest 450ms db.query ×47 … (47 total) ← N+1 pattern auth.verify storage.write 38ms serial queries, should be one batch trace_id: a3f82c… · 480ms total · 52 spans
A trace waterfall for one TraceLab ingestion request. The worker span consumes 450 of the 480 ms, and the span breakdown immediately shows 47 sequential db.query spans — the N+1 pattern. No metric or log alone would show you this structure; only the trace does.

Two failure modes that only a trace can expose reliably:

📖 OpenTelemetry (OTel) is the standard

OpenTelemetry is the CNCF-backed open standard for instrumentation APIs, SDKs, and a vendor-neutral collector. You instrument once with the OTel SDK, then route telemetry to any backend (Jaeger, Tempo, Honeycomb, Datadog, Lightstep) by changing a config line. Avoid vendor-specific SDKs for instrumentation — you will switch backends someday. The opentelemetry-sdk Python package with opentelemetry-instrumentation-fastapi gives you automatic span creation for every HTTP endpoint with about ten lines of setup code.

SEE ITIncident triage console

A seeded TraceLab fault is live. Use the three panels to work through the canonical triage sequence: first check metrics to establish when and how bad, then filter logs by correlation ID to find the exact failing request, then walk the trace waterfall to localize the span responsible. Five possible faults are seeded; the N+1 query and the serial-when-parallel span are among them. Triage is timed — a clock runs once you start querying. Three pillars, one answer.

🚨 Incident Triage Console — TraceLab Production

Step through: Metrics → Logs → Trace. Each pillar narrows the search. Find the fault and submit your diagnosis.

Triage time: 00:00
① Metrics ② Logs ③ Trace ④ Diagnose
RED — API Gateway /ingest
USE — Worker Pool

Start in Metrics to see the shape of the anomaly, then drill into Logs, then Trace.

Notice what each pillar contributed. Metrics showed you when and how much — latency up, error rate up, worker saturation at 100%. The logs let you filter to a specific correlation ID and see the error message. The trace gave you the span-level breakdown: forty-seven serial db.query spans, or two independent calls running sequentially when they should run in parallel. Neither logs nor metrics alone would give you the span structure. That is exactly why all three pillars exist.

TRACELABInstrumenting TraceLab with what it teaches

TraceLab's Stage-5 milestone is called "shipped and survived." For observability, that means the system monitoring experiment runs must itself emit structured logs, RED metrics, and traces — dogfooding the primitives it is supposed to teach. This closes the loop: TraceLab is both an experiment-tracking tool and its own first production workload.

tracelab/telemetry.pyTraceLab v4 — observability bootstrap
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
import logging, json, os

# ── Structured JSON logger ─────────────────────────────────────────
class _JSON(logging.Formatter):
    def format(self, r):
        d = {"ts": self.formatTime(r, "%Y-%m-%dT%H:%M:%S.%fZ"),
             "level": r.levelname, "logger": r.name, "msg": r.getMessage()}
        if hasattr(r, "ctx"): d.update(r.ctx)
        return json.dumps(d)

def get_logger(name: str) -> logging.Logger:
    log = logging.getLogger(name)
    if not log.handlers:
        h = logging.StreamHandler(); h.setFormatter(_JSON())
        log.addHandler(h); log.setLevel(logging.INFO)
    return log

# ── Tracer & Meter ─────────────────────────────────────────────────
def setup_telemetry(service_name: str = "tracelab"):
    endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")

    tp = TracerProvider()
    tp.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint)))
    trace.set_tracer_provider(tp)

    reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint=endpoint))
    mp = MeterProvider(metric_readers=[reader])
    metrics.set_meter_provider(mp)

# ── RED instruments for the ingestion endpoint ─────────────────────
tracer  = trace.get_tracer("tracelab.ingest")
meter   = metrics.get_meter("tracelab.ingest")
ingest_requests  = meter.create_counter("ingest_requests_total")
ingest_errors    = meter.create_counter("ingest_errors_total")
ingest_duration  = meter.create_histogram("ingest_duration_ms",
                       unit="ms", description="Ingest request latency")
tracelab/ingest.pyTraceLab v4 — instrumented ingest handler
import time, uuid
from opentelemetry import trace
from .telemetry import tracer, ingest_requests, ingest_errors, ingest_duration, get_logger

log = get_logger("tracelab.ingest")

async def handle_ingest(run_id: str, payload: dict):
    correlation_id = str(uuid.uuid4())
    ctx = {"run_id": run_id, "correlation_id": correlation_id}
    t0 = time.monotonic()

    with tracer.start_as_current_span("ingest") as span:
        span.set_attributes({"run_id": run_id, "correlation_id": correlation_id})
        log.info("ingest.started", extra={"ctx": ctx})
        ingest_requests.add(1, {"endpoint": "ingest"})

        try:
            # N+1 debt repaid: batch-load run + metrics in one query
            rows = await batch_insert_run(run_id, payload)
            ms = (time.monotonic() - t0) * 1000
            ingest_duration.record(ms, {"endpoint": "ingest"})
            log.info("ingest.complete", extra={"ctx": {**ctx, "rows": rows, "ms": round(ms, 1)}})

        except Exception as e:
            ingest_errors.add(1, {"endpoint": "ingest"})
            span.record_exception(e)
            log.error("ingest.failed", extra={"ctx": {**ctx, "error": str(e)}})
            raise

Three things worth noting in this code. The correlation_id is attached to both the log entries and the span attributes — when you see a log error, you can use the correlation ID to find the trace and vice versa. The histogram records every request duration, not just errors, so you can compute percentiles across the fleet. And the span wraps the entire operation: any child spans (a future DB call, an S3 write) will automatically be children of the ingest span, building the waterfall without extra wiring.

The debt ledger for TraceLab is almost clear. The N+1 query you found in the trace widget above is exactly the pattern batch_insert_run is designed to avoid — a JOIN replacing a loop of individual selects. The serial-when-parallel fan-out is handled by wrapping independent awaitable calls with asyncio.gather(). These are Stage-3 and Stage-4 debts being repaid with Stage-5 tooling that makes the repayment verifiable in production.

WEAKNESSWhere observability goes wrong

✗ printf logs are not "good enough" until they suddenly aren't

Every engineer has a production incident where the logs are a stream of interleaved lines from five concurrent requests, the timestamp granularity is one second, and the error message says "unexpected None" with no context. The timed difference between structured and unstructured logging is zero at write time and enormous at 2 a.m. The cost of switching from print() to log.info("event", extra={"ctx": ...}) is measured in minutes. The cost of not switching is measured in incident hours. There is no real argument for unstructured logs in a production system.

✗ Means are where truth goes to die

The p99 latency lesson from this lesson is not abstract. A real pattern: a dashboard shows "average response time: 180 ms, well within SLA." The SLA is defined as p99 < 500 ms. Nobody has plotted the p99. The p99 is 2,100 ms. The SLA is being violated daily, for 1% of users, and nobody knows. Alerting on average latency is not alerting on user experience. Change your dashboards and alerts to use histograms with explicit percentile queries before you write your SLOs.

✗ The N+1 query is invisible until you add tracing

An N+1 query in an ORM is one of the easiest performance bugs to introduce and one of the hardest to find without traces. Your logs show "query executed." Your metric shows elevated latency. Nowhere does it say "you fired 47 queries where 1 would do." The only way to see the query fan-out structure is a trace waterfall. This is not a pathology of ORMs specifically — any system that makes remote calls in a loop has this failure mode. Distributed tracing is the only observability primitive that makes the call graph visible.

KEY INSIGHTHow this connects

  • CD: Releasing Without Fear (5.4) — the canary and feature-flag mechanics you learned there are what make observability actionable: a spike in p99 fires an automatic rollback only if the release pipeline watches the right SLI. Shipping without observability is driving blind; shipping without a rollback is driving blind with no brakes.
  • SLOs, Error Budgets, and Alerts That Don't Cry Wolf (5.8) — the next lesson uses the histogram metrics you set up here to define SLIs and SLOs. The p99 latency you instrument today becomes the SLI definition tomorrow. Observability and SLOs are not separate systems — the SLO is a policy over the telemetry.
  • Inside the Server: The Request Lifecycle (3.3) — the middleware and handler layers you stepped through there are the exact layers traces instrument. Each middleware hop becomes a span; the time hidden in serialization or authentication is invisible until you add child spans to those layers.
  • Queues and Async Work: Decoupling in Time (4.4) — the queue consumer that dequeues a message and processes it asynchronously is exactly where a trace ID must cross a process boundary via message metadata. Without propagating the trace header through the queue, the worker's spans are orphaned and the waterfall breaks at the queue boundary.

Observability is the last lesson before the SLO lesson because you need the telemetry before you can set policies over it. The Stage-5 arc ends with a scripted outage (Lesson 5.9) — the postmortem you write will lean entirely on the signals you set up here. Instrument first; incidents will test the instrumentation.

GLOSSARYKey terms introduced

Structured logging
Emitting log entries as machine-readable key-value records (JSON, logfmt) instead of free-form strings, enabling indexing, filtering, and aggregation by field.
Correlation ID
A unique identifier attached to a request at its entry point and propagated to every downstream service and log entry, allowing all log lines for one request to be reassembled after the fact.
RED metrics
Rate, Errors, Duration — the three metrics that characterize the health of any request-oriented service or endpoint.
USE metrics
Utilization, Saturation, Errors — the three metrics that characterize the health of any resource (CPU, memory, connection pool, thread pool). Formalized by Brendan Gregg.
Percentile (p99)
The latency value below which 99% of requests complete. Unlike the mean, percentiles preserve tail behavior and are the correct measurement for user-facing SLOs.
Span
A named, timed operation within a trace. Spans form a parent-child tree; each span records service, operation, start time, duration, status, and key-value attributes.
Trace waterfall
The visual representation of a distributed trace as a Gantt chart, showing spans as horizontal bars positioned by start time and width by duration, making serial vs parallel execution immediately visible.
OpenTelemetry (OTel)
A CNCF standard for instrumentation APIs, SDKs, and a vendor-neutral collector protocol, allowing services to emit logs, metrics, and traces without coupling to any specific observability backend.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1You see a latency spike in your metrics. What does each pillar tell you, in sequence?›

The metric tells you when the spike started, how large it is, and which endpoint is affected — it narrows the time window and points at the symptom. A log search on that time window, filtered by correlation ID, tells you which requests were slow and what was in their error or warning messages — it narrows from aggregate to individual request. The trace for one of those slow requests tells you which span caused the slowness — the exact operation (a DB query, a remote call, a computation) and how long it took relative to others. You need all three because each narrows what the next can query.

2Why is recording average latency in a metric insufficient, and what should you record instead?›

The average aggregates all requests into one number and is dominated by the bulk of fast requests. A small fraction of slow tail requests (p99, p999) can be severely impacted without moving the mean visibly — those users experience the product as broken while the dashboard looks healthy. Record latency as a histogram and query explicit percentiles: p50 (typical), p95, p99, p999. SLOs should be defined over percentiles, not averages, because percentile behavior is what users actually experience.

3What is the cardinality trap in metrics, and why is it not a problem in logs?›

Metric backends like Prometheus store one time series per unique label combination. Adding high-cardinality labels (user ID, request ID, trace ID) to metrics creates millions of time series — memory and query costs scale with the cardinality, often fatally. Logs, by contrast, are append-only records indexed by field at query time; a high-cardinality field like user_id in a JSON log entry costs nothing at write time and is indexed for exact-match queries. The rule: metrics stay low-cardinality (service, endpoint, status code bucket); logs absorb high cardinality.

4Which observability primitive reveals an N+1 query pattern, and why can't logs or metrics do it alone?›

A distributed trace reveals the N+1 pattern. Logs record that queries executed but not their call-graph structure. Metrics record aggregate query counts and durations but not how many queries belonged to one logical request. Only a trace, which preserves the parent-child span structure, shows that one logical ingest operation fired 47 child db.query spans in sequence — making the pattern visually obvious in the waterfall. Without tracing, N+1 bugs can persist for months behind metrics that say "DB latency elevated" without explaining why.

Next: turning your histogram into an SLO with an error budget that makes alerts meaningful →