Debugging Firmware: Logs, JTAG, Logic Analyzers, and the Scientific Method
The bug is not in the last thing you changed. It is in the last assumption you made.
🎯 By the end you'll understand
How to debug something you cannot reproduce, by treating a symptom as a set of competing hypotheses and choosing the evidence that eliminates the most of them per minute spent. Which evidence sources an ESP32-C3 gives you for free — the reset-reason register, the ROM banner, a core dump, the stack high-water mark — and which ones perturb the very timing you are trying to observe. Why a logic analyser and a spare GPIO beat a log line for anything to do with timing, and why adding logging sometimes makes a bug disappear. How to attach GDB over the C3's native USB with no external probe, and what deep sleep does to that session. And why a watchdog is a detector rather than a fix, so that "the watchdog restarts it" is a description of a defect rather than a solution.
KEY INSIGHTDebugging is hypothesis elimination, not searching
Three tags out of twenty-five reset about once a day. The team spent a week reading the state machine, added logging to every transition, and could not reproduce it on the bench. With the extra logging the resets became rarer, which everyone took as progress. It was not progress: the log lines had slowed the wake window enough to change when the radio transmitted. The answer was one function call away the whole time —esp_reset_reason()returnedESP_RST_BROWNOUT, and the battery on those three units was the oldest in the batch.
Firmware bugs are rarely found by reading code, because the code is the thing you already believe. They are found by asking what else would have to be true, and then measuring it. That reframing changes what you do first: instead of "where in my code is the mistake", you ask "what are the four things that could produce this symptom, and what single measurement would eliminate three of them?" A week of code reading eliminates nothing, because reading confirms whatever you expected. One register read eliminated four hypotheses in ten seconds. The skill this lesson teaches is not a tool; it is the habit of choosing the next action by how much it would change your mind.
Before you touch anything, write down the competing explanations — three or four is usually enough — and for each one, the observation it predicts that the others do not. Then pick the cheapest measurement that separates them. Evidence that is consistent with every hypothesis on your list is not evidence, however much of it you collect; a log line that says "entering SAMPLE" when all four candidates enter SAMPLE has told you nothing at the cost of three milliamps and a timing change.
CORE CONCEPTThe method, in five steps you can do at a bench
State the symptom precisely, with numbers
Not "it crashes sometimes" but "three units out of twenty-five reset roughly once per 24 hours; the other twenty-two have not reset in nine days". The rate, the population, and the exceptions are all data. "Sometimes" is the enemy: it hides the fact that 22 units disagree with your theory.
List the hypotheses, all of them, before choosing one
For a reset: brown-out, task watchdog, interrupt watchdog, a panic from bad memory, or an external power interruption. Five candidates, each with a different signature. Writing them down stops you from investigating your favourite for a week.
Choose the measurement that discriminates most per minute
The reset-reason register separates all five in ten seconds. A logic analyser on the I2C bus separates none of them and takes twenty minutes to set up. Rank actions by how much of the hypothesis space they cut, not by how comfortable you are with the tool.
Predict before you look
Say out loud what each hypothesis predicts the measurement will show. A prediction you would have accepted either way is not a test. This is also the step that catches the situation where all your hypotheses predict the same thing — which means the real cause is not on your list yet.
Change one thing, and be able to undo it
The fix is a hypothesis too. If it works, prove it by putting the bug back: a fix you cannot reverse to reproduce the failure is a coincidence you have adopted. On a 25-unit pilot, that means version control on firmware, a note in the bug log, and the willingness to reflash one unit with the old build.
On an ESP32-C3, the first question after any unexplained restart has one function: esp_reset_reason(). Its answer routes the entire investigation, and Beacon records it in the buglog partition at every boot so the answer survives the reboot that would otherwise erase it.
COMPARISONEvidence, ranked by what it costs you
Every tool has three properties that matter: what it can tell you, how long it takes to set up, and how much it changes the system while measuring. The third column is the one people ignore and the one that decides whether you can find a timing bug at all.
| Evidence | Answers | Setup | Perturbs the system? |
|---|---|---|---|
esp_reset_reason() + the ROM banner | Why the chip restarted, and whether software was involved at all | Seconds | No |
| The buglog partition (3.2) | What the previous boot was doing: last state, last error, retry counts, recoveries | Already there | Negligible — one small write per boot |
| GPIO toggle + oscilloscope | Exactly how long a function, an ISR, or a wake window takes, with its distribution | Minutes | About 200 ns per toggle. Effectively no. |
| Logic analyser on a bus | Ground truth for I2C, SPI, UART: what was really on the wire, ack by ack (3.5) | Ten to twenty minutes | No |
| Current profile (µA meter or shunt + scope) | Whether the device slept, how long each wake really was, how many wakes happened | Ten minutes | No, if the shunt is small |
| Core dump + backtrace | Where a panic happened, with the call stack and task states | Minutes, once enabled | Only when it fires |
| GDB over USB-JTAG | Anything, interactively: breakpoints, watchpoints, memory inspection | Minutes | Yes, profoundly — stopping the core changes every timing relationship, and deep sleep ends the session |
| Extra log lines | What you thought to print, where you thought to print it | Seconds | Yes: milliseconds per line on a console, plus flash and power (3.4) |
Read the bottom two rows together with the third. A GPIO toggle and a scope answer "how long did this take" with a resolution of nanoseconds and a perturbation of nanoseconds; a log line answers the same question with a resolution of whatever you printed and a perturbation of milliseconds. For anything involving timing, races, interrupt duration, or bus behaviour, the instrument is the pin — and Beacon's board reserves a test pad for exactly this (6.3). The debugger is unmatched for logic bugs you can reproduce on demand, and nearly useless for a device whose defining behaviour is sleeping for sixty seconds at a time.
SEE ITBug triage board: three symptoms from Beacon's pilot
Pick a symptom from the pilot. The candidate causes are listed with the evidence that would separate them; your job is to choose measurements that eliminate candidates, not measurements that confirm your favourite. Every action costs bench time, and one of them changes the system while measuring. When you are confident, commit to a cause — and the board will tell you whether you had actually earned it.
Two lessons are built into the board. First, the cheapest action is usually the most discriminating, and it is almost never the one people reach for: the reset-reason register costs two minutes and eliminates most of a symptom's candidates, while the logic analyser costs twenty minutes and, for a reset symptom, eliminates nothing. Second, try the verbose rebuild on symptom A. It costs fifteen minutes, tells you almost nothing you did not know, and — because the extra console writes lengthen the wake window — makes the symptom rarer, which feels like progress and is the exact mechanism by which teams lose a week.
IN PRACTICELogging that pays for itself
Logging is not free and it is not neutral, so it has to earn its place. On a battery device the price list is concrete: time (a console write is milliseconds, and on the 45 ms wake window that is a large fraction), flash (every format string is bytes of .rodata, and Lesson 3.2 showed a build that overflowed its partition on log strings alone), and power (time awake at 40 mA, once per minute, forever).
# sdkconfig: compile the noise out of the release build entirely. CONFIG_LOG_DEFAULT_LEVEL_INFO=y CONFIG_LOG_MAXIMUM_LEVEL_INFO=y # DEBUG/VERBOSE calls are removed by the compiler: # no code, no format strings, no flash cost at all // In the source, keep the expensive detail behind the levels you stripped: ESP_LOGI(TAG, "%s + %s", ST[s], EV[e]); // ships: one line per transition ESP_LOGD(TAG, "i2c rx %02x %02x %02x", b[0],b[1],b[2]); // dev only: gone in release // Runtime override for one component, without a rebuild: esp_log_level_set("beacon", ESP_LOG_DEBUG); // only works if the level // was not compiled out
That last comment is the trap worth knowing: CONFIG_LOG_MAXIMUM_LEVEL removes the calls at compile time, so esp_log_level_set() cannot bring them back. It is the right trade for a shipped battery product and the wrong one for a development build, which is why the two builds differ in exactly that line.
What Beacon actually logs. One line per state transition — state, event, next state — because that single line reconstructs the whole session (3.7). The reset reason and the previous state at every boot. Driver failures with their error name and a count. Bus recoveries. Everything else is at DEBUG and is not in the shipped binary. The result is a log you can read from a serial capture and a wake window that stays inside its budget. The rule of thumb: log decisions, not data. "SAMPLE + SAMPLE_READY ⇒ ADVERTISE" tells you what the firmware chose; printing the six raw bytes tells you what you could have read off a logic analyser without spending any power.
Any bug that involves timing — a race, an ISR duration, a bus that is nearly too slow, a supply that sags at a particular instant — can be moved or hidden by the delay a log line adds. The signature is unmistakable once you know it: the bug becomes rarer when you add logging and returns when you remove it, and somebody eventually proposes leaving the logging in "because it works". That is not a fix, it is a timing coincidence you have shipped, and it will unfix itself when a compiler upgrade changes the code layout or a colder day slows a rise time. When you see that signature, stop logging and start measuring: a GPIO toggle at the top and bottom of the suspect region, a scope, and the distribution of durations. The measurement that does not perturb is the only one that can be trusted here.
EXAMPLEJTAG over the C3's own USB, and what deep sleep does to it
One of the genuinely good things about the ESP32-C3 is that the debug probe is inside the chip. The native USB Serial/JTAG peripheral on GPIO18/19 (3.3) presents both a serial console and a JTAG debug interface over the same connector, so a single USB-C cable gives you flashing, monitoring, and a full GDB session with no external hardware.
$ idf.py openocd # uses the built-in USB-JTAG config for this target $ idf.py gdb # attaches, loads symbols from the ELF (gdb) break sht40_fetch # hardware breakpoints: a small, fixed number (gdb) watch app.sht.fail_count # a watchpoint traps the WRITE — who changed it, and when (gdb) bt # backtrace: the call chain that got you here (gdb) info threads # every FreeRTOS task, its state and its stack (3.8) (gdb) p *app.sht # read a struct out of a running device # Postmortem instead of interactive — for a fault you cannot catch live: $ idf.py coredump-info # decode a stored/emitted dump into a backtrace
Three honest limitations, because a debugger is not a universal tool.
- Deep sleep ends the session. The digital core, and the USB Serial/JTAG peripheral with it, is powered down. You cannot single-step through a sleep, and after every wake you are attaching to what is effectively a freshly booted chip. For a device whose whole architecture is "sleep for sixty seconds", the debugger is a tool for the 45 milliseconds and the buglog is the tool for the rest.
- Stopping the core stops the world. Halting at a breakpoint does not pause the BLE controller's view of time, the sensor's conversion, or the I2C peripheral's timeout. You will resume into a system that has moved on, and the resulting errors are artefacts of the debugger, not of your bug. This is why a breakpoint in an ISR is usually a bad idea and a GPIO toggle is usually a good one.
- Hardware breakpoints are a scarce resource. The chip has a small fixed number, and the stack-overflow watchpoint option from Lesson 3.8 consumes one of them. When the debugger declines to set another one, that is the reason.
Where GDB is unbeatable: a reproducible logic bug, a corrupted variable (a watchpoint answers "who wrote this" in one step, where logging can only tell you "it was wrong by the time I printed it"), and reading the state of every task after a hang. Where it is worth almost nothing: an intermittent field fault on a sleeping device, which is what the reset reason, the buglog, and a current profile are for.
DEFINITIONWatchdogs are detectors, not fixes
A watchdog is a timer that resets the chip if software stops proving it is alive. ESP-IDF gives you three, and they detect genuinely different failures — which is exactly what makes them valuable as evidence.
| Watchdog | Fires when | Tells you | Cannot see |
|---|---|---|---|
| Task watchdog | A monitored task — the idle task by default — has not run within the timeout (5 s by default) | A task is hogging the CPU or is deadlocked; the message names it (3.8) | A task that is blocked politely and correctly on an empty queue |
| Interrupt watchdog | Interrupts have been disabled for longer than its timeout, in the hundreds of milliseconds | A critical section or an ISR that ran far too long (3.4) | An ISR that is merely 3 ms long and fires constantly |
| RTC watchdog | The very early boot path stalls, before the app is running | A bootloader or flash problem | Anything after the application starts |
The blind spot in the first row is the important one, and Lesson 3.7 named it: a state machine stranded in ADVERTISE, waiting for an event that already went somewhere else, is perfectly healthy from the task watchdog's point of view. The loop is running, the queue receive is blocking, the idle task gets plenty of time. The device is doing nothing at 40 mA and no watchdog on the chip can tell. That is why Beacon adds an application-level timeout — five seconds in any state with no event — which is a watchdog for the logic rather than for the scheduler.
A watchdog reset is a defect report, not a recovery. Each one costs a lost report, a full boot's worth of energy, and — on a device that keeps state in RTC memory — potentially a lost sequence number. Shipping a product that resets daily and calling it self-healing means shipping a bug you have decided not to find, and the same defect will eventually appear in a state where the reset does not help. The correct response is to record the reason, count the events per unit, and treat any non-zero count in the pilot as a work item. The feeding pattern matters too: a watchdog fed from a timer callback, or from a loop that runs regardless of whether the application is making progress, proves only that the timer works. Feed it from the place that would stop if the device stopped being useful.
BEACONFirmware v0, and the bug log that comes with it
This lesson completes Stage 3's milestone. Beacon firmware v0 runs on an ESP32-C3 dev kit with drivers for both sensors, the application state machine, motion-interrupt wake, structured logging, and a watchdog — all verified on the bench with a logic-analyser trace of each bus. The artefact that matters as much as the code is the bug log: a plain file in the repository, one entry per defect, written while the memory is fresh.
| # | Symptom | Hypotheses considered | Discriminating evidence | Root cause and fix |
|---|---|---|---|---|
| 1 | Tag never sleeps after the first motion event | FSM stranded; wake source misconfigured; interrupt latch | Current profile: flat 40 mA, no sleep at all; INT1 measured high with a meter | The LIS2DH12 latch was not cleared before arming the wake, so the level-high wake source re-triggered immediately. Read INT1_SRC in the SLEEP entry action (3.4, 3.7). |
| 2 | Every fourth report reads −45.00 °C | Sensor failing; bus too fast; missing conversion delay | Logic analyser: the read was issued 0.4 ms after the command, and the sensor NACKed | A vTaskDelay(pdMS_TO_TICKS(9)) that evaluates to zero ticks at the 100 Hz tick (3.3). Replaced with an 8.5 ms esp_timer one-shot; the CRC check that caught it stays (3.6). |
| 3 | Occasional panic naming the cache, only when the kit is being handled | Bad RAM; stack overflow; flash write during an interrupt | Reset reason PANIC plus a core-dump backtrace inside the motion handler | The handler was in flash and an edge landed inside an NVS commit with the cache disabled. Added IRAM_ATTR and ESP_INTR_FLAG_IRAM (3.4). |
| 4 | Bus dead after an unplanned reset; power cycle fixes it | Slave stuck mid-byte; short; driver state | SDA read low with SCL high at boot, before any transaction | A slave holding SDA after a reset mid-transfer. Added the boot-time stuck check and the nine-clock recovery, and counted the events (3.5). |
Every one of those entries has the same shape, and that shape is the deliverable: symptom, alternatives, the measurement that chose between them, and the fix. Written down, four entries become a checklist for the Rev A bring-up in Stage 6 and the beginning of the post-mortem in 6.9. Not written down, they become four things one person remembers and nobody else can act on.
(1) esp_reset_reason() and the last FSM state are written to the buglog partition at every boot, so a field unit brings its history home. (2) The shipped build compiles out DEBUG and VERBOSE entirely; INFO logs one line per state transition, driver failures with counts, and bus recoveries. (3) A test pad is toggled at the start and end of each wake window so a scope can measure the real active time — the number R1's whole battery budget rests on. (4) The task watchdog is enabled, and the application adds its own five-second stranded-state timeout for the failure the watchdog cannot see. (5) Watchdog resets, brown-outs, bus recoveries, and dropped events are counted per unit and read out during the pilot (6.7). (6) A saved analyser capture of a healthy wake window is committed next to the firmware as the reference to compare against.
WEAKNESSWhere this bites
Swapping a part, raising a timeout, adding a delay, reflashing — each of these can make a symptom go away without teaching you anything, and each one contaminates the experiment for everybody who looks next. The specific damage is that you can no longer tell which change mattered, so the bug's real cause stays on the loose and reappears under a different symptom. Change one thing at a time, write down what you predicted, and be able to put it back. If a fix works, prove it by reintroducing the bug: a fix you cannot reverse to reproduce the failure is a coincidence, not an explanation.
The most common wasted week is spent gathering data that every hypothesis predicts. A log line at the top of SAMPLE when all four candidates reach SAMPLE; a scope trace of a signal nobody doubts; a second unit that behaves identically to the first. It feels productive because output accumulates, and it eliminates nothing. The discipline is to ask, before the measurement, which candidate would this rule out? If the honest answer is "none", do something else — and if no available measurement discriminates, your list of hypotheses is incomplete, which is itself a finding worth acting on.
A log line says what your code thought it did. A logic analyser says what actually happened on the pins. When they disagree, the wire is right, and the gap between them is usually the bug: the transaction that "succeeded" but never left the peripheral, the interrupt that "fired" 3 ms after the edge, the pin you "set high" that the pad never drove because its clock was gated (3.1). This is not an argument against logging; it is an argument for knowing which instrument answers which question. Logs are for decisions, scopes and analysers are for physics, and any question containing the words "how long" or "when exactly" belongs to the second category.
KEY INSIGHTHow this connects
- Serial Buses: UART, I2C, and SPI (3.5) — the analyser trace you now know how to read is the ground truth this lesson keeps deferring to, and the stuck-bus check is entry number four in the bug log.
- State Machines and Event-Driven Firmware (3.7) — the transition log is the single most valuable thing Beacon prints, and the stranded-state timeout is the watchdog for the failure no hardware watchdog can see.
- FreeRTOS Essentials (3.8) — the task watchdog, stack high-water marks, and
info threadsin GDB are the evidence sources for everything scheduling-related. - Batteries: Chemistry, Capacity, and the Lies of mAh (4.1) — next stage: the current profile you used here as evidence becomes the measurement the entire six-month requirement is built on.
- Field Data and the 30-Day Pilot (6.7) — the counters and the buglog partition defined here are what the pilot actually reads, and the bug log is where the post-mortem begins.
Stage 3 ends with firmware that works and, more importantly, firmware that can explain itself: a state machine whose every transition is logged, drivers that say when they are unsure, an interrupt path with a measured budget, and a device that records why it restarted. Stage 4 takes that firmware and asks the question the canon has been holding over it since Lesson 1.9 — can this thing really run for six months on one cell — and this time the answer has to be measured rather than argued.
GLOSSARYKey terms introduced
- Discriminating evidence
- A measurement whose outcome differs between competing hypotheses. Evidence consistent with all of them, however much of it you gather, tells you nothing.
- Reset reason
- A register, read via
esp_reset_reason(), that says why the chip last restarted: power-on, brown-out, task watchdog, interrupt watchdog, panic, deep-sleep wake. The cheapest discriminating measurement in ESP32 firmware. - Core dump
- A snapshot of memory and task state captured when the firmware panics, decoded on the host into a backtrace. Postmortem evidence for faults you cannot catch live.
- Heisenbug
- A defect whose behaviour changes when observed — classically, one that becomes rarer when logging is added because the added milliseconds shift the timing. The signature that tells you to stop logging and start measuring.
- Watchpoint
- A hardware trap on a write to a specific address, so the debugger stops at the instruction that corrupted a variable rather than after the fact. Scarce: the chip has only a few.
- Task watchdog
- Resets the chip if a monitored task has not run within the timeout. Detects hung and starved tasks; cannot see a state machine correctly waiting for an event that will never arrive.
- Interrupt watchdog
- Resets the chip if interrupts have been disabled for too long, catching an over-long critical section or handler. A different failure from the task watchdog, and a different fix.
- Bug log
- A file in the repository with one entry per defect: symptom, hypotheses considered, the evidence that discriminated, root cause, fix. The artefact that turns one person's debugging into the team's knowledge.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1A colleague reports that adding log lines to the wake path made a daily reset "mostly go away", and proposes shipping the extra logging. What do you say, and what do you do instead?›
That is the signature of a timing-sensitive bug, not a fix. Console writes cost milliseconds — roughly 87 µs per byte at 115200 baud — so the added logging changed when the radio transmits, when the I2C transaction happens, and how long the wake window lasts. The defect is still there; it has been moved. Shipping it means shipping a timing coincidence that will unfix itself when a compiler upgrade changes code layout, a colder day slows a rise time, or a battery ages. Instead: read esp_reset_reason(), which distinguishes brown-out, task watchdog, interrupt watchdog, panic and external reset in seconds and costs nothing. If the answer is brown-out, the investigation moves to hardware — scope the 3.3 V rail during a transmit burst on a unit with an old cell — and no amount of firmware reading would have found it. And note the extra evidence hiding in the report: three units out of twenty-five, which points at something that varies between units rather than at the code they all share.
2A tag stops reporting after a few hours. You have twenty minutes. Rank these actions and justify the ranking: attach GDB, capture the I2C bus with an analyser, read the buglog partition, add debug logging and reflash.›
Read the buglog first: it costs a minute, does not perturb anything, and it already contains the reset reason, the last state the machine was in, driver failure counts and bus-recovery counts — which distinguishes "the chip reset" from "the machine stranded" from "the bus is stuck" immediately. Second, capture the bus with the analyser, because if the buglog points at I2C then the wire is the ground truth: you will see whether the master ever issued a START, whether SDA is being held low by a slave stuck mid-byte, and whether the address was acknowledged. Third, GDB — useful only if the fault is reproducible while attached, and this device sleeps for sixty seconds at a time, which ends the debug session at every cycle. Last, debug logging and a reflash: it is the slowest option, it destroys the failed state you were about to examine, and it perturbs the timing that may well be the cause. The general ranking is by discrimination per minute, adjusted for how much the tool disturbs what you are measuring.
3Why can the task watchdog not detect Beacon's stranded-state failure, and what does the firmware add to cover the gap?›
Because nothing is hung. The application task is blocked on xQueueReceive, which is exactly what a well-behaved task does when it has nothing to do; the idle task therefore runs normally and feeds the watchdog. The task watchdog detects a task that will not yield, and a stranded state machine yields perfectly — it is simply waiting for an event that was delivered to a state that discarded it. The device is not crashed and not working, and on battery power it burns 40 mA until the cell is flat, about a day for a 1000 mAh cell. The cover is an application-level timeout: the queue receive uses a five-second timeout, and on expiry the firmware records the current state to the buglog and forces a transition to SLEEP. That is a watchdog for the logic rather than for the scheduler. It does not fix the missing transition — nothing does except filling in the table — but it converts a silent battery death with no evidence into one lost report and a log entry naming the state.
4You need to know whether Beacon's wake window is really the 45 ms the battery budget assumes. Describe the measurement, and explain why a log line at the start and end of the window is the wrong instrument.›
Toggle a spare GPIO high on entry to the wake path and low just before esp_deep_sleep_start(), then measure the pulse on an oscilloscope or logic analyser — ideally with persistence or statistics on, so you get the distribution across many wakes rather than one anecdote. The toggle costs roughly 200 ns and changes nothing else, and the scope resolves the edges to nanoseconds. Cross-check it against a current profile — a shunt and a scope, or a µA meter with enough bandwidth — which shows both the duration and the amplitude, and therefore the actual charge per wake that R1's arithmetic depends on. A log line is the wrong instrument for three reasons: it costs milliseconds, so it changes the very quantity being measured, and by a significant fraction of a 45 ms window; its resolution is limited to whatever timestamp you print; and it only measures the path you thought to instrument, whereas the current profile shows everything the device did, including the parts you forgot about. For any question of the form "how long" or "when exactly", the pin is the instrument and the log is a story about the pin.