Firmware in the Field: Logging, Telemetry, and Remote Diagnosis
You will not be in the room when it fails. Design the device to tell you what happened.
🎯 By the end you'll understand
Why the reset reason and the brown-out counter are the highest-value bytes in any embedded product, and how to read them on the ESP32-C3. How to budget a telemetry payload when you have eight bits and fifteen bits' worth of things worth knowing, and why over-specifying one field silently steals bits from another. How to keep an event log in memory that survives deep sleep and resets without wearing out flash, and how to read it back over a Bluetooth connection when a unit misbehaves. The three distinct diagnostic layers — device flags, fleet liveness, and fleet analytics — and the incidents that only one of them can ever see. What not to put in a broadcast that anyone in range can record. And which of Beacon's six pilot incidents its flags byte actually diagnosed, which was three fewer than anyone expected.
KEY INSIGHTA fleet where everything is OK is a fleet you cannot see
The default value of a status flag is zero, and zero usually means "fine." A fleet in which every unit reports fine is indistinguishable from a fleet in which every unit's status logic is broken — and from a fleet in which three units are not reporting at all.
Three of Beacon's twenty-five pilot units failed to start after a routine battery swap. They had been put into the ESP32-C3's ROM download mode by a technician who happened to be holding the button while clipping in the cell, and a chip in download mode does not run the application, does not advertise, and does not report anything at all. Every unit that was reporting reported fine. The dashboard showed twenty-two green rows and three rows that had simply stopped updating, and the only reason anyone noticed was that somebody scrolled down and looked at a "last seen" column.
There is a general principle hiding in that anecdote, and it is the one this lesson is built on. No amount of device-side telemetry can report an absence. A unit that has crashed, bricked, run out of battery, or wandered out of radio range says nothing, and "nothing" is exactly what a healthy sleeping unit also says between reports. Detecting that class of failure is structurally a job for the receiver, not the transmitter — and it is the layer that most fleet dashboards are missing, because building a table of the latest reading is easy and building a table of who has stopped reporting requires deciding what "stopped" means.
Field diagnosis happens at three layers, and each catches things the others structurally cannot. Device flags report what the device knows about itself — reset causes, error counts, wake sources — and cost bits in every packet forever. Fleet liveness reports who has gone quiet, which no device can report about itself. Fleet analytics compares units against each other, catching the unit whose readings are plausible but wrong. Design for one layer and you will be blind to two thirds of what happens to you.
HOW IT WORKSReset reason and brown-out: the cheapest telemetry there is
When an ESP32-C3 starts, hardware has already recorded why. ESP-IDF exposes it through esp_reset_reason(), and the value distinguishes a normal power-on from a software reset, a panic, each flavour of watchdog, a wake from deep sleep, and — the one that matters most for a battery product — a brown-out.
| Reset reason | What it means | What you do about it |
|---|---|---|
ESP_RST_POWERON | Power was applied. On Beacon this means a battery was fitted | Expected once per service event. Frequent power-ons in the field mean a connector problem, not a firmware problem |
ESP_RST_DEEPSLEEP | Woke from deep sleep. The normal case, once a minute | Nothing — but counting it separately is how you distinguish a normal cycle from a reboot |
ESP_RST_BROWNOUT | The supply fell below the brown-out detector's threshold | The single most informative code in the list. Latch it, count it, and enter the safe state from Lesson 6.6 |
ESP_RST_PANIC | The firmware hit an exception and the panic handler reset the chip | A bug. Count it and, if you can afford the flash, store the program counter |
ESP_RST_TASK_WDT / ESP_RST_INT_WDT / ESP_RST_WDT | A task stopped feeding the watchdog, an interrupt handler ran too long, or the hardware watchdog fired | Count consecutive occurrences and escalate — a watchdog that reboots into the same hang is a loop |
ESP_RST_SW / ESP_RST_EXT | A deliberate software restart, or an external reset signal | Expected after an over-the-air update; unexpected otherwise |
ESP_RST_UNKNOWN | The cause could not be determined | Treat as suspicious rather than benign. Recent IDF versions add more specific codes, including power-glitch and CPU-lockup causes — check the enumeration for your version |
Alongside the reset reason there is a second question worth a bit or two: why did we wake? On the C3, esp_sleep_get_wakeup_cause() distinguishes a timer wake from a GPIO wake after deep sleep. One constraint is worth knowing before you assign pins: deep-sleep GPIO wakeup on this part is available only on a limited set of RTC-capable pins, so the accelerometer's interrupt and any wake-capable button both have to live inside that set. Check the technical reference for your silicon revision before committing a pin — and note that "move the button off GPIO9," which Rev B needs anyway because GPIO9 is a boot-strapping pin, is constrained by this as well.
/* RTC SRAM: survives deep sleep AND resets, lost when the battery is removed. That is exactly the lifetime a "how has this deployment gone" counter wants. */ RTC_DATA_ATTR static uint16_t n_brownout, n_panic, n_wdt, n_boot; RTC_DATA_ATTR static uint8_t last_reset, last_wake; void telemetry_boot(void) { esp_reset_reason_t r = esp_reset_reason(); last_reset = (uint8_t)r; last_wake = (uint8_t)esp_sleep_get_wakeup_cause(); switch (r) { case ESP_RST_DEEPSLEEP: break; // the normal path case ESP_RST_BROWNOUT: n_brownout++; break; case ESP_RST_PANIC: n_panic++; break; case ESP_RST_TASK_WDT: case ESP_RST_INT_WDT: case ESP_RST_WDT: n_wdt++; break; case ESP_RST_POWERON: n_boot++; counters_reload_from_nvs(); break; default: break; } /* A power-on wipes RTC SRAM, so the running totals live in NVS as well — written once an hour, not once a minute (Lesson 6.6's wear arithmetic). */ } /* Deep-sleep housekeeping. Pin states are NOT retained across deep sleep unless the pad's hold latch is enabled first, so anything driving an external gate floats for the whole sleep interval. GPIO1 drives the battery divider's PMOS gate: hold it. Necessary, and — see Lesson 6.9 — not sufficient. GPIO10 (LED rail) needs no hold: a 100 kOhm pull-up already parks it off. */ static void sleep_now(uint32_t seconds) { gpio_hold_en(PIN_VDIV_EN); // GPIO1 gpio_deep_sleep_hold_en(); esp_sleep_enable_timer_wakeup((uint64_t)seconds * 1000000); esp_deep_sleep_start(); }
Two details in that listing pay for themselves repeatedly. The counters live in RTC SRAM, which the C3 keeps powered through deep sleep and through a reset but not through a battery removal — so a counter tells you about this deployment rather than about the device's whole history, which is usually what you want, and the long-term totals are mirrored into flash on a slow schedule. And the deep-sleep helper enables the pad hold latch before sleeping: on ESP32 parts, a GPIO's output state is not automatically retained through deep sleep, and anything that drives an external MOSFET gate will float unless it is held. Keep that one in mind. It is necessary housekeeping, it is what the divider's gate on GPIO1 gets — and in Lesson 6.9 it turns out not to be enough, for a reason that is about volts rather than about code.
CORE CONCEPTFourteen bytes, and how to spend the byte you have left
Beacon reports by non-connectable BLE advertising: three packets per burst, once a minute, and then straight back to deep sleep. There is no connection, no acknowledgement, and no second chance — whatever the gateway does not hear is gone. The report is a manufacturer-specific advertising data element: one length byte, one type byte of 0xFF, and twelve bytes of payload, fourteen bytes in total inside an advertising packet whose legacy limit is 31 bytes.
Look at the resolution asymmetries, because they are the most transferable idea here. Temperature is an int16 scaled by 100, giving hundredths of a degree, against a requirement of ±0.3 °C and a sensor whose typical accuracy is a fraction of that. Humidity is a single byte, one per cent, which is comfortably finer than the ±3 % requirement. Battery voltage is a uint16 in millivolts, resolving 1 mV — while the actual measurement, through a two-megohm divider into the C3's ADC, carries an uncertainty of roughly ±15 mV after calibration and considerably worse across temperature. Four or five of those sixteen bits are describing noise with great precision. None of that is a bug, and all of it is bits you could have spent on something you cannot currently see.
SEE ITDesign the telemetry — then replay six real incidents against it
Allocate Beacon's flag bits among candidate signals, choose whether the gateway watches for silence and whether the dashboard compares units against each other, and decide whether to keep an event log the device can be asked for. Then look at the six incidents from the pilot. Each one is scored: diagnosed from the gateway alone, narrowed to a short list, or invisible. Two of them cannot be diagnosed by any allocation of bits, and finding out which two is the point.
Three things worth doing deliberately. Spend all eight Rev A bits on the reset reason, the brown-out count, and the watchdog count — a defensible, conservative allocation — and notice that unit 14's over-the-air drain and the motion storms are both invisible, because you bought reasons for reboots on a device that was not rebooting. Then turn on liveness monitoring and watch the three download-mode lock-ups move from invisible to detected, with zero bits spent; that failure was never a payload problem. Finally, turn everything on and observe that the humidity drift is still not diagnosable from any flag — it needs a comparison between units, which is a property of the fleet rather than of any device in it.
IN PRACTICERing buffers that survive, and a way to ask for them
Flags in the advertising payload are a summary. When a unit misbehaves in a way a summary cannot explain, you want a history — and a history has to live somewhere that survives what happened.
Three storage tiers, three lifetimes
RAM is lost on every deep sleep, which for Beacon is once a minute; it is useless for anything spanning a report cycle. RTC SRAM — the C3 keeps a few kilobytes of it alive in deep sleep, and it also survives a reset — is the right home for an event ring buffer and for counters, and it has no wear limit at all. It is lost when the battery is removed, which is a feature: it makes the counters mean "since this deployment started." Flash, via NVS, survives everything and has an erase budget, so it holds the slowly-changing things: the identity, the configuration, and hourly-flushed running totals. Lesson 6.6's arithmetic is the reason for that split: an hourly flush is sixty times fewer erases than a per-minute one for exactly the same data.
A useful event record is small and structured. Beacon's is four bytes: an event code, a value, and the sequence number of the report it happened during. Forty-eight of them is one hour of history in 192 bytes of RTC SRAM, and the ring is written by exactly one function so there is one place to reason about ordering.
typedef struct __attribute__((packed)) { uint8_t ev; // EV_BROWNOUT, EV_I2C_ERR, EV_WAKE_MOTION, EV_OTA_ENTER, ... uint8_t aux; // event-specific: reset code, I2C address, wake pin uint16_t val; // battery mV, error count, whatever the event is about } evrec_t; // 4 bytes; 48 of them = 192 B of RTC SRAM = 1 hour /* Read-out: a connectable advertising window, opened only on request. Beacon normally advertises non-connectably, which costs less and cannot be interfered with. Once an hour it advertises connectably for 2 seconds so a technician standing next to it can pull the log — a deliberate, bounded hole in the "no connections" design, and it costs about 0.4 uA on average. */ static void maybe_open_service_window(uint32_t seq) { if ((seq % 60) != 0) return; ble_adv_connectable(2000); // 2 s window, GATT exposes the ring read-only } /* Cost check, so the decision is arithmetic and not taste: 2 s of connectable advertising once an hour at ~12 mA average = 2 * 12 mA / 3600 s = 6.7 uAh per hour = 6.7 uA... no: per hour it is (2/3600) * 12 mA = 6.7 uA of ADDED AVERAGE current. Too much. Once every 6 hours instead: 1.1 uA. Once a day: 0.28 uA. Pick the interval from the budget, not from convenience. */
That last comment block is doing real work, and it is the habit worth stealing. A "small" feature added to a sleeping device is not small until you have divided its energy by the reporting period. A two-second connectable window every hour costs about 6.7 µA of average current — seventeen per cent of Beacon's entire 40 µA budget, for a diagnostic feature used perhaps twice a year. The same window once a day costs 0.28 µA, under one per cent, and is just as useful because a technician who is standing next to a misbehaving unit can also press its button to request one. Every telemetry feature on a battery product needs this calculation attached to it, in the source, where the next person will see it.
The reflex from server work is to log strings. On a device with kilobytes of retained memory and a fourteen-byte report, strings are unaffordable and, worse, unstructured — you cannot aggregate them across a fleet without parsing. A four-byte record with a numeric event code costs a hundredth of what a formatted line costs, sorts and counts trivially, and forces you to decide in advance what the interesting events are. Keep the human-readable strings in the host-side decoder, where they are free, and version the event-code table alongside the firmware version that is already in the payload.
COMPARISONThree layers, and what only each one sees
The widget above makes this concrete; here it is as a rule you can apply to any fleet product.
| Layer | Lives where | Costs | Sees uniquely | Structurally blind to |
|---|---|---|---|---|
| Device flags | In the payload, every report | Bits, forever, on every packet | Why the device reset, what error it just had, why it woke, what it is about to do | Anything that stops the device transmitting; anything requiring comparison with another unit |
| On-device event log | RTC SRAM, flushed to NVS | Retained memory; energy for read-out | The sequence of events leading to a failure — the thing a single flag can never convey | Anything that happens before the log exists, or on a unit you cannot reach |
| Fleet liveness | Gateway and dashboard | A "last seen" column and a threshold | Units that have stopped reporting: bricked, flat, out of range, stolen | Why. Liveness detects absence and cannot explain it |
| Fleet analytics | Dashboard, over history | Storage and a comparison rule | The unit whose readings are plausible but wrong; slow drift; a whole batch behaving differently | Fast, single-unit events; anything where you have no comparable neighbour |
The cheapest of those four is fleet liveness, and it is the one most often missing. It requires no device change, no bits, and no energy: a column recording when each unit was last heard from, and a rule for how long silence is allowed. Beacon's reporting period is one minute, packets are lost routinely in a real building, and the pilot's overall delivery rate was well under a hundred per cent — so the threshold has to tolerate normal loss. Five consecutive misses is a defensible starting point, and the right way to set it is from the observed distribution of gaps rather than from a round number. Get it wrong in the tight direction and the dashboard cries wolf until people stop looking, which is worse than not having it.
DEFINITIONWhat not to log, and what a broadcast really is
Beacon transmits in the clear, to nobody in particular, on a channel any phone can listen to. A person standing in the workshop with a scanner app sees every report from every unit, including the identifier. Two consequences follow, and both are easy to overlook when you are focused on making the thing work.
A static identifier is a tracker. An asset tag reports where an asset is, and an asset is usually attached to a person's work. A log of tag 4A2F's presence over a month is a log of who was in which part of the building when, and the fact that you never intended to collect that does not stop you from having collected it. If the deployment context makes that sensitive — and in a workplace it usually does — the mitigations are real but not free: rotate the identity using a resolvable private address so that only a gateway holding the identity resolving key can correlate reports, or reduce the reporting rate and the retention period so the data is less useful for tracking. Beacon uses a static address because the gateway is trivial and the pilot was internal, and that choice was written down as a choice.
A CRC is not a signature. The crc8 at the end of the payload detects corruption in transit. It does nothing whatsoever against someone transmitting a well-formed Beacon report with any values they like, because the algorithm is public and the data is unauthenticated. Anyone in range can inject a fake temperature reading, a fake battery level, or a fake unit. For an internal pilot that is an acceptable risk clearly stated; for a product where the data drives a decision, the payload needs a message authentication code with a per-device key, which costs bits you have already seen are scarce and a key-management story you do not currently have. Say which of those two situations you are in, out loud, in the documentation.
The specific failure to guard against: a diagnostic value added to the payload during development — a raw ADC count, an internal state number, a pointer's low byte — that nobody removes because it is only a byte and it is useful. It ships, it is broadcast continuously for the life of the product, and it is now part of a public interface that you cannot change without breaking every gateway. Payload fields are an API. Add them with the same reluctance you would add a public method, and keep the firmware version byte updated so a decoder can tell which layout it is looking at — Beacon has that byte, and it is the only reason the layout can ever change.
BEACONWhat Beacon's flags byte actually caught
Rev A's flags byte used six of its eight bits: low battery, an I2C error on the last sample, a "first report after a reset" marker, and three bits of reset reason. Two bits were left over, and the entire moved byte carried a single boolean. Here is how that allocation performed against the six pilot incidents.
| Incident | What the telemetry showed | Verdict |
|---|---|---|
| Unit 14 flat on day 4 (button held by the enclosure boss) | Battery millivolts falling far faster than any neighbour, and nothing else. No wake-source bit, no update-path bit | Narrowed only — "this unit is draining." The cause needed the unit on a bench |
| Three units silent after a battery swap (download mode) | Nothing. A chip in download mode never advertises | Found by a human noticing a stale "last seen" column. No allocation of bits could have done better |
| Unit 22, battery connector worked loose on day 11 | Nothing, then nothing. Identical signature to the above | Found the same way, two days later than it should have been |
| Unit 07 humidity reading 6–9 points high from day 12 | Plausible values, no error flag, no reset. The sensor was working perfectly and reporting a contaminated measurement | Found by a person eyeballing the dashboard and noticing unit 07 sitting above its neighbours — fleet analytics performed by eye |
| Two brown-out resets in week 4 | The reset-reason bits reported ESP_RST_BROWNOUT on the first report after each event | Diagnosed remotely, immediately, from three bits |
| Motion storms on the two CNC-bench units | The moved boolean was set on nearly every report — true, and useless, because it cannot distinguish "moved once" from "vibrated two hundred times" | Narrowed. The count came from the gateway inferring it from report timing, which worked by luck |
One incident in six was diagnosed remotely, and it was the one nobody would have predicted mattered. The three bits spent on the reset reason — the most conservative, least imaginative allocation available — turned out to be the only bits that earned their place, and they caught a mechanism that produced no visible symptom at all. Meanwhile the moved byte, which carries seven unused bits directly adjacent to a boolean that was not enough, is the clearest available illustration of a payload sized field-by-field rather than as a budget.
Three changes went onto the list with this evidence. First, put a saturating motion count in bits 1–3 of the moved byte: three bits, zero to seven-or-more events since the last report, in space that is already being transmitted. That turns the motion-storm incident from a two-day inference into a glance. Second, spend the flags byte's two spare bits on the wake source, which would have identified unit 14's failure on day one — the unit was waking from the button every single minute, and that is a two-bit fact. Third, and by far the cheapest, add a liveness rule to the gateway: no device change, no bits, no energy, and it converts the two most expensive incidents in the pilot from "somebody eventually noticed" into an alert. The full payload is still fourteen bytes; the only thing that changed is that the bits were allocated on purpose.
WEAKNESSWhere this bites
A zeroed structure, an uninitialised field, or a flag that is only ever set on the error path all produce the same result: a device that reports fine when its status logic never ran. The countermeasure is to make "healthy" a positive assertion rather than an absence. Include a heartbeat counter that must advance, a sequence number the gateway can check for gaps, and — the sharpest tool — a field whose expected value changes every report, so a stuck transmitter is visibly stuck. Beacon has a sequence byte for exactly this reason, and it is the reason a repeated-packet problem would be detectable at all.
It is natural to add a flag for every bug you just spent a week on, which is how a payload accumulates bits describing conditions that can no longer occur. The bits are permanent; the bugs are not. A better filter: add a signal if it distinguishes between two plausible future explanations for a symptom you expect to see. The reset reason passes that test — a silent reboot has several possible causes and three bits separate them. A flag for "the specific I2C timing bug from March" does not, because that bug is gone and its flag will read zero for the life of the product.
A two-second connectable advertising window every hour sounds like nothing and costs about 6.7 µA of average current — around seventeen per cent of Beacon's entire budget, for a feature used twice a year. A one-kilobyte flash write every minute sounds like nothing and costs the flash's whole erase budget in two years. On a device that spends 99.9 % of its life asleep, every recurring cost must be divided by the reporting period before it is judged, and the division belongs in a comment next to the code so the next person does not have to re-derive it. Diagnostics that break the power budget do not get deployed, which means they diagnose nothing.
KEY INSIGHTHow this connects
- Debugging Embedded Systems Without a Debugger (3.9) — the same instinct, at a different distance. On the bench you have a serial port; in the field you have fourteen bytes a minute and no second chance.
- BLE Advertising and Payload Design (4.6) — where the fourteen-byte element was specified. This lesson spends its last bits.
- Reliability and Field Failure Modes (6.6) — the brown-out counter, the watchdog escalation, and the wear-aware ring buffer are the same code seen from the reliability side.
- Cost, BOM, and Scaling From 25 to 25,000 (6.8) — next: what all of this costs, and why at 25 units the engineer's time dominates every other line on the page.
- Capstone: Beacon Pilot Deployment and Post-Mortem (6.9) — the pilot's incident list in full, each one attributed to a decision from Stages 1 through 5.
Telemetry is the part of the design that assumes you will be wrong. Everything else in this course tried to make the product correct; this lesson accepts that it will not be, and asks what the device should be able to tell you on the day it is not.
GLOSSARYKey terms introduced
- Reset reason
- A hardware-recorded cause for the most recent restart, exposed on the ESP32-C3 by
esp_reset_reason(). It distinguishes power-on, deep-sleep wake, software restart, panic, each watchdog, and brown-out — three bits that diagnose an otherwise invisible class of failure. - Brown-out detector
- On-chip circuitry that resets the processor when the supply falls below a configurable threshold, preventing undefined behaviour. Its value as telemetry is that it converts a mysterious reboot into a named cause.
- RTC SRAM
- Memory kept powered through deep sleep and surviving a reset, but lost when power is removed. It has no wear limit, which makes it the right home for counters and event rings on a device that must not write flash often.
- Pad hold latch
- A per-pin latch that retains a GPIO's output state through deep sleep. Without it, pins that drive external gates float for the whole sleep interval — enabled with
gpio_hold_en()plusgpio_deep_sleep_hold_en(). - Manufacturer-specific advertising data
- A BLE advertising element with type 0xFF carrying a two-byte company identifier and arbitrary payload. Beacon's is twelve bytes of payload in a fourteen-byte element, broadcast non-connectably.
- Fleet liveness
- A gateway-side check for units that have stopped reporting, with a threshold tolerant of normal packet loss. It is the only layer that can detect a device that has gone silent, because no device can report its own absence.
- Fleet analytics
- Comparison of a unit's readings against its peers or its own history, catching plausible-but-wrong values that no error flag will ever set. It found Beacon's contaminated humidity sensor; nothing on the device could have.
- Resolvable private address
- A BLE address that changes periodically and can be resolved only by a peer holding the identity resolving key. The standard mitigation for a broadcast identifier that would otherwise make a tag a tracker.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Three units entered ROM download mode and stopped advertising. Design the flags byte allocation that would have diagnosed this from the gateway. How long should you spend on it?›
None, because it does not exist. A chip in download mode never runs the application, so no field in a payload the application builds is ever transmitted. The failure is an absence, and no device-side telemetry can report an absence — a silent unit is indistinguishable from a healthy sleeping one until you decide how long silence is allowed. The layer that catches it is fleet liveness at the gateway: a last-seen timestamp per unit and a rule such as "alert after five consecutive missed reports." That costs no bits, no energy, and no firmware change, and it would have converted a two-day discovery by a person scrolling a dashboard into an alert within five minutes. The general lesson is the one worth carrying: before spending payload bits on a failure, check whether the device will be alive to transmit them.
2Beacon's battery field is a uint16 in millivolts. Why is that a waste, and what would you do with the reclaimed bits?›
Because the resolution far exceeds the measurement's uncertainty. The voltage is read through a two-megohm divider into the C3's ADC, and after calibration the uncertainty is on the order of ±15 mV, worse across temperature. A field resolving 1 mV is therefore describing about four bits of noise with great precision. The useful range is roughly 3000 to 4200 mV — 1,200 distinct values at 1 mV, but only about 80 meaningfully distinct values at the measurement's real resolution. Encoding the range as an offset in 20 mV steps needs six bits and loses nothing real. That frees ten bits, in a payload where you were arguing over two. What to spend them on: the wake source, a saturating motion count, a watchdog counter, and a spare field for the next thing you did not anticipate. The transferable point is that field sizing is a budget problem, and sizing each field on its own always overspends, because nobody is ever penalised for asking for one more byte.
3A colleague proposes a two-second connectable advertising window every hour so technicians can pull logs. Evaluate it.›
Do the division first. Two seconds of connectable advertising at roughly 12 mA average, once an hour, is (2 / 3600) × 12 mA ≈ 6.7 µA of added average current. Beacon's whole average budget is about 40 µA, so this diagnostic feature consumes roughly seventeen per cent of the product's energy — permanently, on every unit, for something used perhaps twice a year. That is not a good trade. The same window once a day costs about 0.28 µA, under one per cent, and loses very little practical capability, because a technician standing beside a misbehaving unit can press its button to request an immediate window. Better still, gate the window behind a condition: open it only when an error counter is non-zero, so healthy units pay nothing at all. The habit being demonstrated matters more than the answer: on a device that sleeps 99.9 % of the time, every recurring feature must be divided by the reporting period before it is judged, and the arithmetic belongs in a comment beside the code.
4Unit 07 reported humidity 6–9 points high for eighteen days with no error flag set. Was the firmware wrong? What layer catches this, and what would it cost?›
The firmware was correct and so was the sensor. The SHT40 was measuring the air it was in, and coolant mist had contaminated it, producing a genuine offset — there is no error condition to detect, because nothing errored. Neither a flag nor an on-device log can catch this: the device has no reference against which its own reading is wrong. The layer that catches it is fleet analytics — comparing each unit's reading with the median of nearby units, or with its own longer-term history, and flagging persistent divergence. In the pilot that comparison was performed by a person looking at a dashboard, which worked with twenty-five units and will not work with two thousand. The cost is storage of history plus one comparison rule, and the hard part is not the code but choosing the rule: too tight and every genuinely different microclimate raises an alarm, too loose and an eight-point offset hides inside normal variation. That threshold should come from the observed spread across the fleet, which is a thing you only have after you deploy — which is a good argument for storing the history from day one even before you know what to do with it.