Safety Architecture: Watchdogs, E-Stops, and Speed Governors
Safety is not a feature you add at the end — a robot that cannot fail safely is not safe, regardless of how well the nominal case works.
🎯 By the end you'll understand
Why hardware E-stops are non-negotiable for mobile robots near people, and why a software-only safety stop is structurally unsafe. How velocity governors work and why speed must be proportional to obstacle proximity. What a layered safety model looks like in practice. How to detect subsystem faults in the ROS 2 node graph before they cause harm. What ISO 13849 demands from your safety architecture and how to think about it without hiring a certification team. And exactly what FollowBot should do when each of its subsystems fails — with concrete ROS 2 code.
KEY INSIGHTSafety is an architecture, not a feature
Every safety system you build will be tested exactly once in the way it matters most — and you will not be there to hit reset.
The most dangerous phrase in mobile robotics is "we'll add safety before deployment." Safety cannot be retrofitted onto an architecture that was not designed for failure. A robot that stops when things go right is trivially achievable. A robot that stops correctly when a node crashes, a sensor drops out, an actuator saturates, or a network packet is delayed — that requires safety to be a load-bearing part of the architecture from day one.
Nominal-case engineering optimizes for capability. Safety engineering optimizes for bounded harm under failure. The two require different mental models. In nominal design you ask "how do I make this work?" In safety design you ask "what is the worst thing that happens when this specific component fails, and is that outcome acceptable?" A system where the answer to that second question is "the robot crashes into the person" is not safe regardless of how well the nominal case works.
CORE CONCEPTHardware E-stops: why software-only is not safe
An emergency stop cuts power to the drive motors — immediately, unconditionally, and without relying on any software. That last part is the load-bearing requirement. Consider the alternative: a software-only safety stop implemented as a ROS 2 node that subscribes to a safety topic and publishes zero velocity when triggered.
This design has a fatal structural flaw: the node that implements the stop can itself crash. If the watchdog node is killed by a segfault, an OOM event, or a DDS communication failure, the safety stop evaporates. The robot continues moving. The failure mode you most need the safety system to handle — a software crash — is precisely the failure mode that disables the software safety system.
The hardware E-stop circuit uses a normally-open or normally-closed relay (the choice is a system requirement: normally-closed is fail-safe — power is cut when the relay loses its hold signal, so a cable break or microcontroller crash is safe). The safety relay is controlled by an output signal device line (OSSD) from either a dedicated safety controller or a microcontroller running a separate, minimal safety firmware that is independent of the main compute board.
The default state of a safety actuator should be safe. A normally-closed relay in the motor power path means: relay deenergized (no signal) = motors off. The robot must actively assert "I am safe to move" at a configured rate to keep the relay closed. If anything stops that assertion — crash, deadlock, cable fault, power glitch — the relay opens and the robot stops. This is the fail-safe design pattern. Contrast with normally-open: the relay must be actively closed to cut power, meaning a crash leaves the motors running. Never use normally-open for a mobile robot near people.
Latency also matters here. The time from the trigger event (person enters stop zone) to the motors actually stopping is the sum of detection latency, processing latency, and relay actuation time. For a hardware E-stop circuit, relay actuation is typically 5–15 ms. For a software-only stop publishing on a ROS 2 topic, total latency including DDS serialization, subscriber scheduling, and controller processing is typically 20–80 ms, with spikes to 200+ ms under load. At 1.2 m/s, 80 ms of additional stop latency is 9.6 cm of extra travel. In a crowded corridor, that gap is the difference between a near-miss and a collision.
HOW IT WORKSSoftware watchdogs and fault detection in the ROS 2 graph
A watchdog is a timer that expects to be reset by a healthy node at a regular interval. If the timer expires — because the node crashed, deadlocked, or fell behind — the watchdog triggers a safety response. In hardware, this is a physical circuit. In software within a ROS 2 system, the pattern is implemented via topic heartbeats and topic deadline QoS.
import rclpy from rclpy.node import Node from rclpy.qos import QoSProfile, QoSDurabilityPolicy, QoSReliabilityPolicy from std_msgs.msg import Bool from geometry_msgs.msg import Twist import time class SafetyWatchdog(Node): HEARTBEAT_TIMEOUT_S = 0.5 # 500 ms → 2 Hz minimum from monitored nodes STOP_CMD = Twist() # all fields default to 0.0 — safe stop def __init__(self): super().__init__('safety_watchdog') # Track last-seen time per subsystem heartbeat self.last_seen = { 'lidar': time.monotonic(), 'tracker': time.monotonic(), 'planner': time.monotonic(), } self.estop_active = False self.vel_pub = self.create_publisher(Twist, '/cmd_vel', 1) self.estop_pub = self.create_publisher(Bool, '/safety/estop', 1) # Subscribe to heartbeats from critical nodes self.create_subscription(Bool, '/hb/lidar', lambda _: self._beat('lidar'), 10) self.create_subscription(Bool, '/hb/tracker', lambda _: self._beat('tracker'), 10) self.create_subscription(Bool, '/hb/planner', lambda _: self._beat('planner'), 10) # 20 Hz check loop — fast enough to catch 500 ms timeout reliably self.create_timer(0.05, self._check) def _beat(self, key): self.last_seen[key] = time.monotonic() def _check(self): now = time.monotonic() faults = [k for k, t in self.last_seen.items() if now - t > self.HEARTBEAT_TIMEOUT_S] if faults and not self.estop_active: self.get_logger().error(f'WATCHDOG: fault in {faults} — issuing E-stop') self.estop_active = True self.estop_pub.publish(Bool(data=True)) # Also assert stop on cmd_vel (belt + suspenders: hardware relay is primary) self.vel_pub.publish(self.STOP_CMD) elif not faults and self.estop_active: self.get_logger().info('WATCHDOG: all nodes healthy — clearing E-stop') self.estop_active = False self.estop_pub.publish(Bool(data=False))
Three things matter in this watchdog design. First, the watchdog runs independently of the nodes it monitors — if the planner crashes, the watchdog's timer still fires and catches the timeout. Second, the watchdog publishes on /safety/estop, which the hardware relay firmware also monitors — so even if the software chain from watchdog to motor driver is broken, the hardware relay sees the deasserted signal and opens. Third, /cmd_vel is published as a belt-and-suspenders measure, not the primary stop mechanism.
The ROS 2 Deadline QoS policy provides a complementary mechanism at the middleware layer: you can declare that a topic must receive a message every N milliseconds, and ROS 2 will invoke a callback if that deadline is missed. This catches silent publisher failures without any application-level heartbeat code, but only if both publisher and subscriber declare matching deadline QoS values — a common misconfiguration source.
HOW IT WORKSVelocity governors: speed proportional to risk
The velocity governor enforces a mapping from obstacle proximity to maximum allowable speed. The principle: the closer an obstacle is, the less time you have to react to a new event, so the slower you must move to maintain a stopping margin. This is not just caution — it is a hard geometric constraint derived from stopping distance physics.
For a mobile robot, the minimum stopping distance at velocity v with deceleration a is:
d_stop = v² / (2a) — where v is current speed (m/s) and a is maximum deceleration (m/s²). FollowBot with a 12 kg total system mass (robot body + compute + battery + arm) and motor braking deceleration of 1.5 m/s² at full speed (1.2 m/s): d_stop = 1.44 / 3.0 = 0.48 m. Add 50 ms of latency (0.06 m of travel during processing) and you need at least 0.54 m of clearance to stop from full speed. This directly sets your outermost safety zone radius.
A practical velocity governor uses concentric speed zones defined by the minimum obstacle distance reported by the LIDAR scan at each control cycle:
| Zone name | Min obstacle distance | Max speed | Rationale |
|---|---|---|---|
| Full speed | > 1.5 m | 1.2 m/s | Stopping margin ≥ 0.54 m with room to spare |
| Half speed | 0.8 – 1.5 m | 0.6 m/s | d_stop = 0.12 m; 0.68 m clearance remains |
| Creep | 0.4 – 0.8 m | 0.2 m/s | d_stop = 0.013 m; near-instant stop possible |
| Stop | < 0.4 m | 0 m/s | Inside collision envelope — stop immediately |
The governor computes the minimum range across all valid LIDAR returns in a scan, maps that to the speed table, and clamps the planner's velocity output. It does not replace the planner's obstacle avoidance — it is an independent safety layer that applies after the planner has made its decision, ensuring that even if the planner produces an unsafe velocity, the commanded speed is physically bounded by proximity.
A naive governor checks minimum distance across the full 360° LIDAR field. This is overly conservative: an obstacle at 0.5 m directly behind a robot moving forward should not trigger a stop. A better governor projects the robot's motion direction and computes minimum distance only within a forward-facing arc proportional to the robot's yaw rate. At high angular velocity, widen the arc; at zero rotation, narrow it to the forward cone. This requires knowing the commanded twist, making the governor slightly more complex but dramatically less prone to false stops in cluttered environments.
SEE ITSafety zone configurator
Configure FollowBot's concentric speed zones, then watch a person approach from different angles. Inject a sensor dropout mid-approach to see the governor's response. Compare hardware vs. software-only E-stop latency for each scenario. The worst choice is having too few zones or too short a stop zone — the widget shows exactly why.
CORE CONCEPTISO 13849 in plain English
ISO 13849 is the international standard for "Safety of machinery — Safety-related parts of control systems." For mobile robots operating near people, it is the relevant framework, but it is written for professional safety engineers, not roboticists. Here is what it actually requires in practice.
The standard defines Performance Levels (PL a–e), where PLe is the highest level of safety assurance. For a person-following robot in an open environment with people, a PL c or PL d assessment is typically required. Each performance level maps to a probability of dangerous failure per hour:
| Performance Level | Max PFH (dangerous failures/hour) | Typical application |
|---|---|---|
| PL a | ≥ 10⁻⁵ | Low risk, infrequent exposure |
| PL b | ≥ 3×10⁻⁶ and < 10⁻⁵ | Low-medium risk (boundary 3×10⁻⁶ belongs to PL b) |
| PL c | ≥ 10⁻⁶ and < 3×10⁻⁶ | Medium risk — indoor mobile robots near people |
| PL d | ≥ 10⁻⁷ and < 10⁻⁶ | High risk, frequent exposure |
| PL e | ≥ 10⁻⁸ and < 10⁻⁷ | Very high risk — industrial presses, etc. |
Achieving PL c or higher requires that your safety function (the E-stop) is implemented in hardware with a defined mean time to dangerous failure (MTTFd), has diagnostic coverage (the safety circuit monitors itself for faults), and has a defined common cause failure (CCF) protection. This is why commodity Arduino-based safety circuits do not achieve PL c — they lack the self-monitoring and fault detection circuitry. Certified safety relays (Pilz PNOZ, Schmersal SRB series, etc.) meet these requirements and provide the necessary documentation for a formal safety assessment.
For a research prototype or academic FollowBot, full ISO 13849 certification is not required. But understanding the PFH concept is still useful: it forces you to ask "how often does this safety function fail to stop the robot when it should?" If your answer is "I don't know," you have a design problem. Even informal failure mode analysis — listing each safety-relevant component, its failure mode, and its failure rate — meaningfully improves safety without requiring a certification budget.
IN PRACTICEGraceful degradation: FollowBot's failure response matrix
A layered safety model assigns different responses to different failure severities. The goal is not to stop immediately for every fault — that makes the robot useless — but to match the response severity to the hazard level of the fault.
| Subsystem that fails | Detection method | Response | Rationale |
|---|---|---|---|
| LIDAR node crash | Watchdog timeout (500 ms) | E-stop — motors off | Obstacle detection lost — cannot move safely |
| LIDAR partial dropout (30% beams invalid) | Validity check: invalid beam fraction | Creep mode — 0.2 m/s, widen stop zone by 50% | Partial sensor still useful; increased conservatism |
| Person tracker lost target | Track age > 2.0 s without update | Stop and rotate — execute recovery behavior | Cannot follow what we cannot see; rotate to search |
| Nav2 planner timeout | Action feedback timeout | Hold position — do not move; retry planner | No safe path known; safer to stop than to guess |
| Behavior tree exception | BT executor crash / exception node | E-stop then restart BT in IDLE state | Mission logic corrupted; restart to known safe state |
| IMU failure | Watchdog timeout; covariance spike | Reduce max speed to 0.3 m/s; alert operator | EKF degrades but does not fail immediately; LIDAR odometry can compensate short-term |
| Power voltage drop below threshold | Battery monitor topic | Return to dock or e-stop if dock unreachable | Prevent controller brownout mid-motion |
The key design principle: the failure response must be at least as safe as the nominal operation for that failure mode. Stopping is not always the safest response — a robot stopping in a doorway may block emergency egress. The response matrix must be designed for the specific deployment environment.
Many teams design the nominal case carefully and leave recovery as an afterthought: "if something goes wrong, stop and wait for the operator." This is unacceptable in environments where the robot cannot wait — a patient being followed into a medical ward, a warehouse where stopping in the aisle creates a hazard. Recovery behaviors (search patterns, return-to-dock, safe zone navigation) must be designed, tested, and treated as first-class safety-relevant components. An untested recovery behavior that fails during an actual fault creates a second failure on top of the first.
FOLLOWBOTWiring safety into FollowBot's architecture
FollowBot's safety architecture integrates with the full Nav2 stack and behavior tree developed across Stage 6. Here is the complete safety layer as it fits into the system:
The behavior tree node that implements person following must be augmented with safety conditions. In Nav2's BT XML format, a safety condition node runs before any navigation action and pre-empts the tree if the E-stop is active:
<root main_tree_to_execute="MainTree"> <BehaviorTree ID="MainTree"> <ReactiveSequence name="root"> <!-- Safety pre-condition: runs every tick before mission actions --> <Inverter> <CheckTopic topic="/safety/estop" field="data" value="true" name="EStopActive" /> </Inverter> <!-- Velocity governor: applied inside the NavThrough node via costmap --> <Fallback name="FollowOrRecover"> <Sequence name="NominalFollow"> <IsPersonTracked max_track_age="2.0" /> <NavToPersonGoal following_distance="1.2" /> </Sequence> <Sequence name="Recovery"> <RotateSearch duration="4.0" angular_vel="0.4" /> <Wait duration="2.0" /> </Sequence> </Fallback> </ReactiveSequence> </BehaviorTree> </root>
The ReactiveSequence re-evaluates all conditions on every tick. If /safety/estop becomes true mid-navigation, the sequence immediately fails on the safety check, which causes Nav2 to cancel the active navigation action and halt. The Inverter wrapping the E-stop check makes it a pre-condition in the standard sense: "proceed only if E-stop is NOT active."
By this point, FollowBot has a complete Nav2 stack, a behavior tree orchestrating the follow mission, a SLAM map with global planner, and now a layered safety architecture. The sim-to-real checklist must include: (1) hardware E-stop wired and tested — manually trigger it and verify motors stop within 15 ms; (2) watchdog timeouts verified — kill the LIDAR node and confirm the robot stops within 600 ms; (3) velocity governor active and verified — approach an obstacle at speed and confirm speed drops at each zone boundary; (4) behavior tree safety conditions tested — assert /safety/estop mid-navigation and confirm BT halts correctly.
WEAKNESSWhere this bites
Safety architecture retrofitted onto a finished system almost always fails. The velocity governor needs to intercept cmd_vel before it reaches the motor driver — if your control architecture has cmd_vel flowing directly from the planner to the driver with no interception point, adding the governor requires architectural surgery. The watchdog needs independent access to the E-stop relay — if the relay is only accessible through the main compute board's GPIO, a main board crash takes out the watchdog's ability to trigger the relay. Safety must be designed in, not bolted on. The cost of redesigning for safety post-implementation typically exceeds the cost of designing it in from the start by a factor of 3–10.
The velocity governor is a software component. It runs as a ROS 2 node (or a callback in the controller node). It can crash, deadlock, or receive garbage proximity data from a failed LIDAR. Relying on the velocity governor alone to prevent collision is placing your safety bet on a component that can fail in exactly the conditions where safety matters most — sensor dropout. The velocity governor is a secondary, belt-and-suspenders measure. The primary mechanism is the hardware E-stop triggered by independent proximity detection or watchdog timeout. A LIDAR dropout must trigger the watchdog, not just cause the governor to read "no obstacles" and remove the speed limit.
A common mistake is choosing zone radii by feel: "0.5 m sounds like enough." The stopping distance equation (d_stop = v²/2a) is not optional. At 1.2 m/s with 1.5 m/s² deceleration, you need 0.48 m of pure braking distance before adding any latency. A stop zone radius of 0.5 m that does not account for the 80 ms worst-case software latency (adding 0.096 m at 1.2 m/s) and the 15 ms hardware relay actuation (adding another 0.018 m) is an 0.5 m zone that geometrically cannot stop the robot before contact at full speed. Run the physics before picking numbers. Then add a safety margin of at least 50% on top of the computed minimum.
KEY INSIGHTHow it connects
- 6.8 · Hardware Bring-Up, Calibration, and Field Diagnostics — the field diagnostics workflow introduced there is the same tooling used to verify safety system function: oscilloscope on the relay output, ROS 2 topic echo on
/safety/estop, watchdog timeout testing. Safety architecture verification is field diagnostics applied to the safety subsystem. - 6.6 · Behavior Trees: Architecture, Ticks, and Reactive Mission Management — the
ReactiveSequencewith safety pre-conditions built in this lesson is a direct application of BT architecture. The fail-and-recover pattern for person tracking loss uses the Fallback node covered there. Safety conditions are BT nodes first-class. - 1.7 · Actuators and the Real-Time Constraint — the stopping distance physics and the latency chain analysis connect directly to actuator response time and the real-time control loop frequency. The 50 Hz minimum control rate and the anti-windup requirement from that lesson are preconditions for the velocity governor to function correctly under deceleration.
- 6.10 · Capstone: FollowBot Complete — Architecture, Gaps, and What Comes Next — the safety architecture documented here becomes part of the capstone's honest system assessment. A robot without documented safety architecture cannot be honestly assessed as deployment-ready.
GLOSSARYKey terms
- Hardware E-stop (emergency stop)
- A physical circuit that cuts power to the drive motors independently of any software. Actuated by a relay controlled by dedicated safety hardware. The only reliable stop mechanism when software has failed.
- OSSD (Output Signal Switching Device)
- A safety-rated output signal used in safety relay systems. The safety circuit actively asserts the OSSD line; if assertion stops (due to fault or timeout), the safety relay deenergizes and the E-stop actuates. Enables fail-safe design.
- Velocity governor
- A software component that clamps the commanded robot velocity as a function of obstacle proximity. Implements the concentric speed zone model: full speed in open space, reduced speed near obstacles, stopped inside the collision envelope.
- Watchdog timer
- A timer that must be reset by a healthy subsystem at a regular interval. If the reset does not arrive within the timeout, the watchdog triggers a safety response. Used to detect crashed or hung software nodes without relying on the failing node to report its own failure.
- Fail-safe
- A design principle in which the default (unpowered, de-energized, failed) state of a safety actuator is the safe state. For a motor power relay: normally-closed means motors are off when the relay is deenergized — a cable break or crash is automatically safe.
- Performance Level (PL)
- ISO 13849 classification of a safety function's reliability, from PL a (lowest) to PL e (highest). Defined by the probability of dangerous failure per hour (PFH). PL c or d is typically targeted for mobile robots operating near people.
- Graceful degradation
- A system behavior in which partial failures produce reduced but safe operation rather than complete failure. Contrasted with catastrophic failure (immediate unsafe stop) and ignored failure (continued unsafe operation). Requires an explicit failure response matrix designed before deployment.
- Defense in depth
- A safety architecture principle in which multiple independent safety layers are stacked so that the failure mode of each layer is caught by the layer below it. No single failure can propagate to harm because the next layer intercepts it.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Why is a software-only E-stop structurally unsafe for a mobile robot near people, even if the software is well-written?›
A software E-stop relies on a ROS 2 node (or equivalent process) running correctly to publish a stop command or deassert motor enable signals. The failure mode that most demands a safety stop — software crash, deadlock, OOM event, DDS failure — is exactly the failure mode that disables the software-based stop. If the safety node crashes, the stop capability evaporates silently. The robot continues moving. The fix is not better software; the fix is a hardware relay controlled by a dedicated microcontroller that asserts a signal to keep the relay closed. When the main compute stops asserting (for any reason, including crash), the relay opens and motors are cut — physically, regardless of software state. The software E-stop then becomes a belt-and-suspenders measure on top of the hardware mechanism, not the primary line of defense.
2FollowBot's LIDAR goes completely silent. Walk through the layered safety response: what happens at each layer, in sequence?›
At Layer 4 (mission): the LIDAR node stops publishing. The person tracker loses LIDAR inputs; after 2.0 s without a track update, the behavior tree's IsPersonTracked condition fails and the BT falls back to the recovery behavior (rotate-search). The planner stops receiving obstacle data and the costmap begins clearing. Before 2.0 s elapses: at Layer 3 (ROS 2 safety), the watchdog detects the LIDAR heartbeat has been absent for more than 500 ms (the configured timeout). It publishes /safety/estop: true, which the BT's safety pre-condition detects on the next tick, immediately pre-empting any navigation action and halting the robot. The watchdog also signals the safety firmware. At Layer 2 (safety firmware): if the main CPU also stops asserting its heartbeat (e.g., in a deeper crash), the firmware deasserts the OSSD after its own 500 ms timeout. At Layer 1 (hardware): the relay opens, physically cutting motor power. The sequence ensures that LIDAR dropout stops the robot within approximately 600 ms regardless of the cause — whether the LIDAR cable came loose or the entire compute stack crashed.
3FollowBot is running at 1.2 m/s and your stop zone is set to 0.4 m. Show the physics of why this is insufficient, and compute the minimum adequate stop zone radius.›
Stopping distance from 1.2 m/s at 1.5 m/s² deceleration: d_stop = v²/(2a) = 1.44/3.0 = 0.48 m. But the stop command is not issued the instant the obstacle enters the zone — there is latency. Worst-case software latency: 80 ms → 0.080 × 1.2 = 0.096 m of travel during processing. Hardware relay actuation: 15 ms → 0.015 × 1.2 = 0.018 m. Total travel from trigger to zero velocity: 0.48 + 0.096 + 0.018 = 0.594 m. A 0.4 m stop zone means the robot cannot physically stop before the obstacle at full speed — it will travel an additional 0.19 m into the stop zone after the stop is triggered. Adding a 50% safety margin: minimum stop zone radius = 0.594 × 1.5 ≈ 0.89 m. The nominal 0.4 m value is appropriate only if full speed is already reduced to creep (0.2 m/s) before reaching that radius, which is exactly what the multi-zone governor enforces — the stop zone is the innermost layer, and the outer zones ensure the robot is already slow enough that 0.4 m is sufficient at creep speed.
4What does ISO 13849 Performance Level c demand from a safety function, and why does a commodity Arduino-based safety circuit typically not meet it?›
PL c requires the safety function to have a probability of dangerous failure per hour (PFH) between 10⁻⁶ and 3×10⁻⁶, which translates to specific demands: (1) a defined mean time to dangerous failure (MTTFd) for each component in the safety circuit, derived from manufacturer datasheets or statistical data; (2) diagnostic coverage (DC) — the ability of the safety system to detect its own faults; (3) common cause failure (CCF) protection — measures ensuring that a single event cannot disable all redundant safety channels simultaneously. A commodity Arduino lacks all three: its MTTFd is not characterized for safety use; its firmware is not self-monitoring (if the watchdog routine freezes, it cannot report its own failure); and it has no redundant channel architecture. Certified safety relays (Pilz, Schmersal, etc.) document all three properties and include internal self-testing circuits, which is why they cost significantly more and why they are required for any robot near people in a commercial deployment context.