Reliability and Field Failure Modes
MTBF is a statistic about a population. Your customer owns one unit, and it either works or it does not.
🎯 By the end you'll understand
What the bathtub curve is actually claiming, why mean time between failures is not a lifetime, and precisely how much a 25-unit, 30-day pilot can support — the answer is a specific number and it is smaller than you would like. What genuinely fails in the field, in the order it fails: connectors, batteries, brown-outs at low charge, flash worn out by your own logging, contamination, electrostatic discharge, and firmware that only misbehaves at four in the morning. Why derating is the reliability tool that costs the least and is skipped the most, with the DC-bias arithmetic that makes a 10 µF capacitor into a 3 µF one. How to design for the failure you already expect — brown-out detection with a defined safe state, wear-aware logging, and a watchdog that escalates instead of looping. And how Beacon's three canon defects behaved once they were in a real workshop.
KEY INSIGHTThe Pareto you measure is not the Pareto you have
A thirty-day pilot shows you the failures whose mechanism completes in thirty days. Every wear-out mechanism in the product — flash cycles, battery cycles, connector insertions, corrosion — is guaranteed to be absent from the data, and equally guaranteed to be present in the product.
Twenty-five Beacons ran for thirty days in a working machine shop. Six incidents needed hands on a unit. If you drew the Pareto chart from that data, it would say: fix the button, fix the boot mode, glue the connector. All three are correct, and all three are cheap. What the chart would not say, because it cannot, is that the same firmware writing a one-kilobyte record to flash every minute wears out its own storage partition in a little over two years — a failure that will arrive on every unit simultaneously, well after the pilot, well after anyone has thought about logging, and looking for all the world like a firmware bug.
This is the structural honesty problem in reliability work. Field data is enormously more valuable than analysis, and it is systematically blind to exactly the failures that arrive slowly. The professional response is to hold two lists: the one the pilot produced, ranked by observed count, and the one the physics produced, ranked by projected count over the product's intended life. Both go in the post-mortem. Acting only on the first is how a company fixes three connectors and then has a recall.
Reliability is not a property you add; it is a set of specific mechanisms you either bounded or did not. For each one, ask three questions: what is the mechanism (cycles, temperature, voltage, vibration, moisture), what is its time constant relative to the product's life, and what does it look like when it happens. A mechanism whose time constant is longer than your test is one you must reason about rather than measure — and reasoning about it costs an afternoon of arithmetic, which is the cheapest reliability work available.
CORE CONCEPTThe bathtub curve, MTBF, and what 25 units actually prove
The bathtub curve plots the hazard rate — the probability that a unit which has survived until now fails in the next interval — against age. It has three regions, and each has a different physical cause.
Mean time between failures is not a lifetime
The most persistent confusion in the field. If a population has an MTBF of 100,000 hours, that does not mean a unit lasts eleven years. It means that during the flat part of the curve, in a large population, roughly one failure occurs per 100,000 unit-hours of operation. A product with an eleven-year MTBF and a two-year wear-out mechanism fails at two years, every time, and the MTBF number was never wrong — it was answering a different question. MTBF describes the plateau; lifetime is set by where the right-hand wall begins. Quote them separately or you will mislead someone, possibly yourself.
What 18,000 unit-hours can support
Beacon's pilot is 25 units × 30 days × 24 hours = 18,000 unit-hours. Suppose it had been flawless, with zero failures. The standard statistical treatment of zero events uses the "rule of three": with no failures observed in n trials, the upper 95 % confidence bound on the rate is approximately 3/n. Here that is 3 / 18,000 ≈ 1.7 × 10−4 failures per hour, which inverts to an MTBF lower bound of about 6,000 hours — roughly eight months. A perfect thirty-day pilot of twenty-five units supports a claim of eight months, at 95 % confidence, and not one hour more. If a customer wants a five-year reliability claim, no pilot of this size can give it to them; you would need either far more units, far more time, or an accelerated-life argument with its own assumptions to defend.
Beacon's pilot was not flawless — six incidents in 18,000 unit-hours is a rate of about 3.3 × 10−4 per hour, an MTBF around 3,000 hours. That number is honest and also nearly meaningless, because five of the six incidents trace to two design defects that Rev B removes, and a rate that mixes systematic defects with random events is not a rate at all. The right way to report it is what the post-mortem does: the raw number, then the attribution, then the projection after the fixes. All three, in that order, so a reader can disagree with your reasoning rather than with your arithmetic.
Burn-in is running units under stress — elevated temperature, power cycling, or simply extended operation — before shipping, so that the infant-mortality failures happen in your building rather than in the customer's. It works because the left wall is steep: a large fraction of latent workmanship defects surface in the first hours to days. It costs calendar time and, if you use a chamber, money. At 25 units the cheap version is a powered soak on a bench for two or three days with the units logging, which is what Beacon's Stage 4 bench run of 72 hours effectively provided. What burn-in cannot do is improve a design; a defect present in every unit is not infant mortality, it is the product.
COMPARISONWhat actually fails, roughly in order
Ask a field-service engineer rather than a component vendor and the list is remarkably consistent across products. Semiconductors, operated within their ratings, are near the bottom. Everything mechanical, everything electrochemical, and everything the firmware only does occasionally is near the top.
| Failure | Mechanism | Time constant | What it looks like | On Beacon |
|---|---|---|---|---|
| Connectors and cables | Fretting, vibration, insertion wear, corrosion at the contact | Days to years, accelerated by vibration | Intermittent, then dead; works when you wiggle it | The JST-PH battery header — a friction fit with no latch, on a bench a CNC machine vibrates all day |
| Battery | Capacity fade with cycles and calendar age; rising internal resistance; swelling | Hundreds of cycles / 2–4 years | Shorter runtime, then brown-outs under load before the voltage looks low | 1000 mAh LiPo, rarely cycled — calendar ageing will dominate |
| Brown-out at low charge | A buck cannot boost: as VBAT falls to the rail voltage the output follows it down, and a transmit peak drops it below the detector's threshold | Appears at end of discharge, every cycle | Spontaneous resets, corrupted writes, a unit that "randomly reboots" | 3.3 V rail from a 1S cell — the margin runs out around 3.4 V |
| Flash wear | NOR flash sectors endure a finite number of erases, typically around 100,000 | Set entirely by your write rate | Writes start failing; NVS returns errors; the device forgets its identity | Depends completely on the logging scheme — see the arithmetic below |
| Contamination and corrosion | Flux residue, condensation, coolant mist, salt; electrochemical migration between adjacent conductors | Months to years; fast in a hostile environment | Leakage, drifting sensors, eventual shorts | The SHT40 is the exposed part by design — it has to be |
| Electrostatic discharge | A person or a tool discharging into an exposed conductor | Random, per handling event | Immediate death, or a weakened junction that dies weeks later | USB port is protected by the TVS; the button and its trace are not |
| Firmware edge cases | States that need a rare coincidence: a counter wrap, a reset during a write, a timer at midnight | Random, but certain at population scale | One unit in the fleet behaving impossibly | Any of them; this is what the telemetry in Lesson 6.7 exists to catch |
| Semiconductors within ratings | Electromigration, gate-oxide wear, thermal cycling of the die attach | Decades at low temperature and low stress | Rare enough that you will blame something else first, correctly | Not a Beacon concern; the parts run cold and lightly loaded |
The flash arithmetic, because it is the one people get wrong
NOR flash endurance is typically specified at around 100,000 erase cycles per sector — check the specific part, since the figure varies and is stated as a minimum under conditions. The ESP32-C3-MINI-1 carries 4 MB of NOR flash, and ESP-IDF's non-volatile storage layer is log-structured: it appends entries into 4 KB pages and, when a page is full, compacts the live entries elsewhere and erases the page. Writes are therefore distributed within the NVS partition, but only within it. A small partition wears proportionally faster, and the general-purpose wear-levelling component in ESP-IDF applies to FAT partitions, not to NVS.
Take a 12 KB NVS partition — three pages. A page holds roughly 126 thirty-two-byte entries once its header and state bitmap are accounted for. Writing one 32-byte record per minute fills a page in about two hours, so with three pages in rotation each page is erased roughly every six hours: about four erases a day, against a 100,000-cycle budget. That is on the order of seventy years. Fine. Now write a one-kilobyte blob instead — a diagnostic dump, a rolling buffer, an "it's only a kilobyte" decision made in an afternoon. A 1 KB blob consumes about 33 entries, so a page holds under four of them and fills in under four minutes; each page is now erased roughly every twelve minutes, about 125 times a day. The budget is gone in 800 days, a little over two years — on every unit, at about the same time, presenting as a device that will not save its configuration. The difference between seventy years and two is not the flash. It is a decision about record size that nobody wrote down.
HOW IT WORKSDerating: the cheapest reliability tool, and the most skipped
Derating means operating a component below its rated limit on purpose, so that variation, ageing, and transients do not push it past that limit. It is the single highest-leverage reliability practice available to a small team because it costs nothing but component selection, and it is skipped constantly because a part operated at its rating works fine on the bench.
| Component | Common derating practice | What happens without it |
|---|---|---|
| Ceramic capacitors (voltage) | Use a part rated at least twice the applied DC voltage | Capacitance collapses under DC bias, and the dielectric ages; the decoupling you designed is not the decoupling you have |
| Ceramic capacitors (dielectric) | X7R or X5R, never Y5V, for anything that matters | Y5V's capacitance can fall by more than 80 % over its temperature range while still being "in specification" |
| Inductors | Peak current below about 80 % of the saturation rating, at temperature | The core saturates on transients, inductance collapses, current spikes, and the regulator's control loop misbehaves in a way that looks like a firmware problem |
| Resistors | Under half the rated power; watch the pulse rating on anything in an inrush path | Drift, then an open circuit, usually months later |
| Semiconductors (voltage) | Below 80 % of the absolute-maximum rating, including transients | A single overshoot outside the absolute maximum is a defect, whether or not the part dies today |
| LiPo cells | Charge below the maximum rate; avoid charging below 0 °C or at high temperature; do not hold at full charge | Accelerated capacity fade, swelling, and in the bad case lithium plating |
A class-II ceramic capacitor's capacitance falls as DC bias is applied, and the effect is dramatic in small packages with high nominal values. A 10 µF X5R in an 0603 package rated at 6.3 V can lose well over half its capacitance at 3.3 V of bias — vendors publish the curves, and it is not unusual to see a part deliver 3 to 4 µF at its intended operating point. This is not a defect and it is not in the tolerance; it is a separate, documented characteristic that is invisible on the schematic. On Beacon it matters at the buck's input and output, where the loop's stability and the transmit-burst response were designed around a capacitance value. The practical rules: choose a higher voltage rating or a larger package for the same value, and check the vendor's bias curve rather than the headline number. If a regulator behaves oddly only on some boards, this is one of the first things to suspect.
The temperature side of derating deserves a sentence of its own, because it is where "indoor, 0–45 °C" from requirement R9 stops being a formality. Every derating factor above is stated at 25 °C. At 45 °C an inductor's saturation current is lower, an X5R's capacitance is further off nominal, a LiPo's calendar ageing is meaningfully faster, and the module's own thermal margin is smaller. A design that is comfortable at 25 °C and marginal at 45 °C will pass every bench test and fail in the one workshop that does not have air conditioning — which, in Jakarta, is most of them.
SEE ITRun the virtual pilot — and see what it hides
The simulator below runs a 25-unit, 30-day pilot with Beacon's design choices, and then projects the same fleet over three years. Two Pareto charts, side by side. The controls are the changes you could actually make. Watch which single change removes the most incidents per rupiah, and watch the one change whose entire benefit is invisible in the pilot column.
Three things to try. Switch the button to require a fresh press: one pilot incident disappears for zero rupiah and three hours of work, and twelve projected failures go with it — the best trade on the page. Move the button off the strapping pin: three more pilot incidents disappear, for one resistor and a layout change. Now switch the logging to one-kilobyte blobs: the observed column does not move at all, and the projected column gains twenty-five failures, one per unit, arriving at around two years. That is the shape of every wear-out mechanism, and it is why the projected column exists.
IN PRACTICEDesigning for the failure you already expect
Three mechanisms on Beacon's list are certain rather than probable: the battery will eventually reach a state of charge where transmit peaks brown the rail out, the flash will accumulate erase cycles at whatever rate you choose, and the firmware will at some point be in a state nobody anticipated. Each has a standard, cheap countermeasure, and each countermeasure has a way of being implemented badly.
Brown-out: detect it, and define what happens next
The ESP32-C3 has a hardware brown-out detector with a configurable threshold; when the supply falls below it, the chip resets and esp_reset_reason() subsequently reports ESP_RST_BROWNOUT. Having the detector is table stakes. What matters is the behaviour around it. A device that brown-outs, reboots, transmits, and brown-outs again is a device that empties the last of its battery in a reset loop and produces a flood of confusing telemetry. The right shape is a latch: on a brown-out reset, read the battery voltage before doing anything expensive, and if it is below a cutoff, record the event and go to deep sleep for a long interval instead of resuming normal operation.
/* Survives deep sleep and resets; lost on power removal — which is exactly the semantics we want for "how bad has this boot cycle been?" */ RTC_DATA_ATTR static uint16_t rtc_brownouts; RTC_DATA_ATTR static uint16_t rtc_wdt_resets; RTC_DATA_ATTR static log_rec_t rtc_ring[48]; // one hour of records, in RTC SRAM RTC_DATA_ATTR static uint8_t rtc_ring_n; #define VBAT_CUTOFF_MV 3450 // below this, a TX peak can pull 3V3 under the BOD #define SAFE_SLEEP_S 3600 void boot_guard(void) { esp_reset_reason_t r = esp_reset_reason(); if (r == ESP_RST_BROWNOUT) rtc_brownouts++; if (r == ESP_RST_TASK_WDT || r == ESP_RST_INT_WDT || r == ESP_RST_WDT) rtc_wdt_resets++; /* A watchdog that reboots into the same hang is a loop, not a recovery. Escalate after three: stop the feature that hung, keep reporting. */ if (rtc_wdt_resets >= 3) { g_cfg.radio_ota_enabled = false; } uint16_t mv = vbat_read_mv(); // settles 300 ms (tau = 50 ms), BEFORE any TX if (r == ESP_RST_BROWNOUT || mv < VBAT_CUTOFF_MV) { log_push(EV_LOWBATT, mv); esp_sleep_enable_timer_wakeup((uint64_t)SAFE_SLEEP_S * 1000000); esp_deep_sleep_start(); // no TX; do not spend what is left } } /* Wear-aware logging: records accumulate in RTC SRAM, which has no wear limit, and reach flash once an hour. 60x fewer erases for the same data. */ void log_push(uint8_t ev, uint16_t val) { rtc_ring[rtc_ring_n % 48] = (log_rec_t){ .ev = ev, .val = val, .seq = g_seq }; if (++rtc_ring_n >= 48) { nvs_handle_t h; if (nvs_open("beacon", NVS_READWRITE, &h) == ESP_OK) { nvs_set_blob(h, "log", rtc_ring, sizeof(rtc_ring)); // 1 erase/hour, not 60 nvs_commit(h); nvs_close(h); } rtc_ring_n = 0; } }
Three design decisions are visible there. The battery is read before the radio is touched, because reading it afterwards tells you the voltage after the load that caused the problem. The watchdog counter escalates rather than merely counting: after three consecutive watchdog resets the firmware disables the feature most likely to be responsible and keeps the device reporting, which is far more useful than a unit that is either perfect or silent. And the log lives in RTC SRAM — retained through deep sleep and through resets, lost on power removal — reaching flash once an hour. That last choice is the difference between the seventy-year and the two-year flash lifetime, and it costs forty-eight records of retained memory.
Enabling the task watchdog and moving on is one of the most common false comforts in embedded work. A watchdog converts a hang into a reset, which is genuinely valuable — a device that reboots is better than a device that is dead. But it also converts a diagnosable, reproducible hang into a periodic reboot that nobody notices, and the underlying bug then lives in the product forever. The discipline that keeps the watchdog honest is counting: every watchdog reset increments a counter that reaches your telemetry, so a fleet dashboard shows you that units are rebooting even though every unit is "working." Lesson 6.7 spends a whole widget on which of those counters is worth its bits.
BEACONThe canon defects, in a workshop, for thirty days
Beacon's canon lists three known defects in Rev A. All three were understood before deployment. What the pilot added was not knowledge of the defects but knowledge of their consequences, and in two of the three cases the consequence was different from the prediction.
| Rev A defect | Predicted consequence | What the pilot showed |
|---|---|---|
| Button on GPIO9, a boot-strapping pin | "Depends on whether anyone holds the button while inserting a battery. We don't know yet." (Lesson 6.1) | Three units failed to start after a battery swap during servicing — the technician held the button while clipping the cell in, the chip entered download mode, and the unit sat there drawing tens of milliamps and never advertised. Recovered by removing and refitting the battery without touching the button. |
| Battery divider leaking 1.8 µA more than planned (OPEN-3) | Covered by the six-fold derating margin on R1; ship it and investigate | Confirmed at 4.5 % of the average current — real, small, and free to fix once the cause was found on day 22. The prediction held. |
| WS2812B-2020 footprint with pin 1 rotated | Fixed in the placement file for the pilot batch; the library is still wrong | The 25 pilot units were correct. One of the five hand-reworked bench boards developed an intermittent LED in week three — the least-characterised joint on the board, as predicted in Lesson 6.2. |
Alongside the canon defects, the pilot produced three incidents that no one had on a list. One unit went flat on day four from the enclosure's button boss holding the switch closed, which Lesson 6.5 dissects. One unit stopped reporting on day eleven when its JST-PH battery connector worked loose on a bench beside a CNC machine — the textbook top-of-the-Pareto failure, arriving exactly where the table above says it will. And one unit, mounted a metre and a half from a coolant nozzle, began reading humidity six to nine points high from day twelve; the SHT40's datasheet includes a reconditioning procedure for contamination-induced offsets, and running it on the bench recovered most of the error.
Two brown-out resets were also recorded, on two different units, in the fourth week, both in the early morning when the workshop was coldest. Neither caused a visible failure — the units rebooted and carried on — and neither would have been noticed at all if the reset reason were not being counted. That is the entire argument for telemetry in one sentence: the two most informative events in thirty days were events that nothing broke.
Thirty days in a dry indoor workshop tested corrosion not at all — humidity ingress and electrochemical migration have time constants measured in seasons, and Jakarta's wet season had not started. It tested electrostatic discharge only incidentally, through however many times a technician happened to touch a button. It tested the battery's calendar ageing for one twelfth of a year and its cycle life for approximately zero cycles, since no unit was recharged. It tested flash wear for 0.4 % of the erase budget under the shipped logging scheme. And because every unit shared the same firmware, the same enclosure batch, and the same assembly run, it tested unit-to-unit variation across a sample that is not a sample of anything but itself. Requirement R1 — six months of battery life — remains a projection. The pilot is evidence that nothing unexpected drains the cell quickly, which is worth having and is not the same claim.
WEAKNESSWhere this bites
"Our MTBF is 200,000 hours" sounds like twenty-three years and means nothing of the sort. MTBF describes the failure rate during the flat region of the bathtub, in a population; lifetime is set by where wear-out begins, which for Beacon is the battery's calendar ageing and, under the wrong logging scheme, the flash. A product can honestly have a 200,000-hour MTBF and a two-year life at the same time, and a customer who is told only the first number has been misled without a single false statement. Quote both, and say which mechanism sets the second one.
Zero corrosion failures in thirty days is not evidence that corrosion is handled; it is evidence that thirty days is shorter than corrosion's time constant. The same applies to every wear-out mechanism, to seasonal effects, and to anything that depends on a rare user action. The correct write-up distinguishes tested and passed from not tested, and the second category is usually larger than the first. A post-mortem that lists only what happened is a report about thirty days; one that also lists what could not have happened is a report about the product.
Beacon's pilot produced one dramatic failure — a unit dead on day four — and two brown-out resets that nobody noticed. The dramatic one absorbed the attention, as dramatic ones do. But the brown-outs are the leading edge of a mechanism that gets worse with every month of battery ageing and every degree the workshop cools, and they were visible only because a counter existed. The habit to build is the one Lesson 6.1 installed with the sleep-current measurement: after you have fixed the obvious thing, go and look at the quiet numbers, because the quiet numbers are describing the failures that have not arrived yet.
KEY INSIGHTHow this connects
- Batteries, Chemistry, and Discharge Curves (4.1) — capacity fade, rising internal resistance, and the flat middle of a LiPo's discharge curve are why the pilot could not measure battery life in thirty days.
- Sleep Modes and the Microamp Budget (4.3) — the 8 µA budget this lesson defends, and the brown-out mechanism that appears when the cell can no longer support a transmit peak.
- Test Jigs, Pogo Pins, and Production Test Firmware (6.3) — burn-in and production test both attack the left wall of the bathtub, from opposite directions: test removes defective units, burn-in removes defective hours.
- Firmware in the Field: Logging, Telemetry, and Remote Diagnosis (6.7) — next: the brown-out counter that made two invisible events visible, and how to spend eight bits of advertising payload on the signals that will actually diagnose something.
- Capstone: Beacon Pilot Deployment and Post-Mortem (6.9) — the full incident list, the attribution of each to a design decision, and Rev B's change list with the evidence attached.
Reliability is the discipline of being specific about things that have not happened yet. It is uncomfortable precisely because the evidence is thin, which is why the arithmetic — erase cycles, unit-hours, confidence bounds — is worth doing rather than gesturing at.
GLOSSARYKey terms introduced
- Hazard rate
- The probability that a unit which has survived to a given age fails in the next interval. The bathtub curve plots hazard rate against age; it is not a plot of failures.
- Infant mortality
- The early, decreasing-hazard region caused by latent workmanship and material defects. A pilot samples this region almost exclusively, which is why it finds blunders well and wear-out not at all.
- MTBF
- Mean time between failures — a population failure rate during the flat region of the curve, not a lifetime. A product can have a long MTBF and a short life if a wear-out mechanism arrives early.
- Rule of three
- With zero failures in n trials, the upper 95 % confidence bound on the failure rate is approximately 3/n. Beacon's 18,000 unit-hours therefore support an MTBF claim of roughly 6,000 hours and no more.
- Derating
- Operating a component below its rating on purpose so that variation, ageing, and transients stay inside the limit. The cheapest reliability practice available and the most frequently skipped.
- DC-bias derating
- The loss of capacitance in a class-II ceramic capacitor as DC voltage is applied. A 10 µF 0603 rated at 6.3 V can deliver a fraction of its nominal value at 3.3 V of bias — documented in the vendor's curves, invisible on the schematic.
- Flash endurance
- The number of erase cycles a NOR flash sector will tolerate, typically around 100,000 as a specified minimum. Combined with your write rate, it fixes a hard lifetime that no other design choice can extend.
- Safe state
- The defined behaviour a device adopts when it detects that it cannot operate correctly — for Beacon, recording the event and sleeping for an hour rather than resuming a transmit cycle that will brown it out again.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1A stakeholder asks what reliability the pilot demonstrated. The pilot ran 25 units for 30 days with six incidents. What is the honest answer, in numbers and in caveats?›
The pilot accumulated 18,000 unit-hours. Six incidents gives an observed rate of about 3.3 × 10−4 per hour, an MTBF near 3,000 hours — roughly four months. That number is honest and nearly useless on its own, because five of the six incidents trace to two specific design defects rather than to random events, and a rate that mixes systematic defects with random failures is not a rate. Even a flawless pilot of this size would only support about 6,000 hours at 95 % confidence, by the rule of three. So the honest answer has three parts: here is the raw rate, here is the attribution showing which incidents Rev B removes, and here is the ceiling on what any pilot of this size could ever have demonstrated. The last part is the one stakeholders most need and least expect, because it says that if a five-year claim is required, no amount of care in this pilot would have produced it — that needs more units, more time, or an accelerated-life argument with its own defensible assumptions.
2Why does writing a 1 KB blob to NVS every minute wear a partition out roughly thirty-five times faster than writing a 32-byte record at the same rate?›
Because NVS is log-structured on 4 KB pages and the erase count is driven by how fast pages fill, not by how often you call the API. A page holds roughly 126 thirty-two-byte entries after its header and state bitmap, so 32-byte records fill a page in about two hours; with three pages in a 12 KB partition each page is erased about every six hours, four times a day, and a 100,000-cycle budget lasts on the order of seventy years. A 1 KB blob consumes about 33 of those entry slots, so fewer than four blobs fit in a page and a page fills in under four minutes; each page is then erased roughly every twelve minutes, about 125 times a day, and the budget is exhausted in about 800 days. The mechanism is entirely about bytes-per-page-per-unit-time. The two mitigations follow directly: write smaller records, or buffer them somewhere without a wear limit — RTC SRAM survives deep sleep and resets — and flush to flash on a much slower schedule.
3Beacon's brown-outs appeared in the fourth week, in the early morning, on units that were not near flat. Explain the mechanism.›
A buck converter cannot boost. As the cell discharges towards 3.4 V, the TPS62840 runs at essentially 100 % duty and the 3.3 V rail simply follows the input minus the switch drop, so the rail is no longer regulated — it is the battery. A BLE transmit burst pulls peaks well above 100 mA, and that current flows through the cell's internal resistance, the protection MOSFETs, and the reverse-battery PMOS, each contributing a drop. If the resulting dip takes the rail below the brown-out detector's threshold, the chip resets in the middle of an advertisement. Two things made the fourth week the moment it appeared: the cells had discharged far enough to be near the transition, and the early morning was the coldest part of the day, which raises a lithium cell's internal resistance and deepens the dip for the same current. The unit's reported battery voltage looked acceptable because it is measured at rest, not under the transmit peak — which is precisely why "the voltage was fine" is not evidence. Rev B's answer is to move the rail to 3.0 V using the regulator's VSET selection, which restores headroom, and to add a firmware cutoff that stops transmitting before the mechanism can engage.
4Your colleague proposes enabling the task watchdog and calling the reliability work done. What is right about that, and what is missing?›
What is right: a watchdog converts a hang into a reset, and a device that reboots is strictly better than one that is dead, especially in a fleet nobody can reach. It is cheap and it should be on. What is missing is everything that makes the watchdog visible and bounded. First, counting: every watchdog reset should increment a counter that survives the reset and reaches your telemetry, or you will have a fleet of units silently rebooting while every one of them reports as working. Second, escalation: a watchdog that reboots into the same hang is a loop, so after a small number of consecutive watchdog resets the firmware should disable the feature most likely responsible and keep reporting in a degraded state rather than cycling forever. Third, scope: a watchdog covers hangs. It does nothing about brown-outs at low charge, flash wear, a connector working loose, or a sensor drifting — which are, on the evidence, where the failures actually are. The watchdog is a recovery mechanism for one failure class, not a reliability strategy.