Capstone: FollowBot Complete — Architecture, Gaps, and What Comes Next
The gap between a robot that works in the lab and one that can be trusted in the field is not an algorithm — it is an honest accounting of every assumption the system makes.
🎯 By the end you'll understand
The complete node graph, topic graph, and transform tree FollowBot uses — every subsystem named, every algorithm identified, every assumption exposed. Which of FollowBot's subsystems are research-grade prototypes vs. production-ready. The known failure modes at every layer, and why honest failure-mode accounting is more valuable than any individual algorithm improvement. Where the open research questions are — the things no lesson in this course has answered. And the most dangerous belief a roboticist can hold: that their system is done.
KEY INSIGHTA robot is never finished — only honestly characterized
The most dangerous belief in field robotics is not "my algorithm is suboptimal." It is "my system is done." Done implies you have accounted for every failure mode. Nobody has.
Six stages and fifty-nine lessons ago, FollowBot was a URDF file and a requirements document. Now it is a complete system: a differential-drive robot with 2D LIDAR, RGB-D camera, wheel encoders, and IMU, running ROS 2 Humble, with a full Nav2 stack, a SLAM-built map, an EKF fusing four sensor streams, a behavior tree orchestrating person-following, and a safety architecture with hardware and software E-stops. That is real. What is also real is that every algorithm in that stack carries assumptions, and most of those assumptions have never been tested at 3 a.m. in a crowded hospital corridor by someone who has never read the source code.
The right question at the end of a robotics project is not "does it work?" It is: under what conditions does it work, which of those conditions are guaranteed in the deployment environment, and what is the failure mode when each condition is violated? This lesson is an honest attempt to answer that question for FollowBot. Where the answer is "we don't know," we say so.
HOW IT WORKSFull system architecture: every node, topic, and transform
FollowBot's ROS 2 graph at the end of Stage 6 comprises seventeen named nodes, organized across six logical subsystems. The TF tree has seven frames (map → odom → base_link → laser_frame / camera_frame / imu_frame / arm_base; arm_base is only active when the arm driver is running). The topic graph has twenty-three active topics in nominal following operation. Here is the complete picture before we assess each piece.
There are fifty-six named concepts across this course — from homogeneous transforms to behavior tree ticks to sim-to-real taxonomy. Every one of them lives inside one of these seventeen nodes. The dashboard below maps them.
SEE ITFollowBot system dashboard
Click any subsystem button to see: which course lessons built it, which algorithms run in it, its sim-vs-real performance grade, and its known failure modes. Then use the tab row to switch between views. Every grade reflects honest field experience — there is no "looks good in Gazebo" grade here.
Notice the pattern in the Grades tab: every subsystem that works in simulation depends on assumptions about the physical world (sensor calibration, environment structure, human behavior) that Gazebo satisfies automatically and the real world does not. The sim-to-real gap is not one problem — it is the sum of every violated assumption across every subsystem simultaneously.
COMPARISONHonest subsystem grades
Here is the unsparing summary of where FollowBot stands after six stages. These grades reflect what is realistic for a well-built research prototype running the algorithms in this course, deployed in a structured indoor environment with a cooperative human. They are not grades for "any environment, any human, any day."
| Subsystem | Simulation | Structured lab | Unstructured real-world | Known ceiling |
|---|---|---|---|---|
| LIDAR sensing | Excellent | Good | Moderate | Absorptive/reflective materials; rain; glass; crowded scenes |
| RGB-D sensing | Excellent | Moderate | Poor | Sunlight; dark clothing; IR interference between units; range >5 m |
| EKF localization | Excellent | Good | Moderate | Long corridors (degenerate scan matching); wheel slip on uneven floors |
| Person tracking | Excellent | Moderate | Research-grade | Occlusion; crowds; similar-looking people; non-frontal gaits |
| SLAM (SLAM Toolbox) | Excellent | Good | Moderate | Dynamic environments; large open spaces; long-term map drift |
| A* global planning | Excellent | Good | Good | Known-map assumption fails when map is stale; narrow-gap inflation |
| DWA local planning | Excellent | Moderate | Moderate | Velocity model mismatch; high crowd density; non-convex obstacles |
| Behavior tree | Excellent | Moderate | Research-grade | Incomplete recovery coverage; no learned recovery; tree growth complexity |
| Safety architecture | Excellent | Good | Moderate | Watchdog covers single-node death; does not cover correlated multi-node failure |
The pattern is stark: sensing and planning are weakest in the real world, and the weaknesses are correlated — a crowd that confuses the person tracker also degrades the costmap used by the planner. When multiple subsystems degrade simultaneously, recovery behaviors designed for single-failure scenarios fail silently or oscillate.
CORE CONCEPTOpen research questions by layer
This course has taught correct, well-established algorithms at every layer. It has not taught the open problems at every layer, because those problems do not have textbook answers. Here they are, honestly stated. These are not exotic research curiosities — they are failure modes that regularly appear in real deployments.
Sensing: reliable person identity across occlusion
FollowBot tracks a cluster of LIDAR points associated with a person. When the person walks behind an obstacle and reappears, the tracker must decide: is this the same person? No algorithm in this course solves person re-identification reliably in LIDAR-only operation. RGB-D adds appearance cues, but appearance changes with lighting, pose, and clothing. The problem is open in the sense that no published algorithm achieves production-grade re-identification across arbitrary occlusion durations in unconstrained indoor environments. The safest current approach: time-limit the occlusion window and stop following rather than guess.
Estimation: long-term localization drift in dynamic environments
SLAM Toolbox builds a map from laser scans. In a hospital corridor, people walk through the field of view, chairs move, doors open and close. The map slowly diverges from the physical environment. AMCL localizes against a static saved map — if the map was built when the corridor was empty and the robot runs when it is crowded, the particle filter will localize incorrectly. Dynamic object removal from SLAM maps is an active research area. The practically honest answer: rebuild the map periodically, not once at commissioning.
Planning: social navigation
DWA treats every moving object as an obstacle to avoid. A person walking toward the robot in a narrow corridor is not just an obstacle — they have a social expectation about where the robot will yield. Social force models and learned trajectory prediction exist in the research literature but are not part of Nav2's default stack. FollowBot will sometimes block a corridor rather than yield correctly. This is rude and occasionally unsafe, and it is not fixable by tuning DWA's parameters.
Behavior: recovery from compound failures
The behavior tree in Lesson 6.6 handles single-failure recoveries: person lost → search; path blocked → replan; low battery → return to dock. It does not handle: person lost AND path blocked AND map stale simultaneously. In this compound failure mode, the recovery subtrees interfere. The robot may oscillate between recovery behaviors, never converging. Behavior trees compose well for independent failure modes and compose poorly for correlated ones. Handling compound failures robustly is an open system design problem, not an algorithm problem.
Safety: certifiable safety in an open-world environment
The safety architecture in Lesson 6.9 provides watchdogs, E-stops, and speed governors. It does not provide formal safety certification. ISO 13849 (machinery safety) and ISO/PAS 21448 (SOTIF — Safety of the Intended Functionality) require formal hazard analysis, systematic coverage arguments, and in some interpretations verified software. None of the algorithms in this course have been formally verified. For a research prototype, this is fine. For a robot operating in a public space, it is a deployment blocker that no algorithm can fix — it requires an engineering and legal process.
FOLLOWBOTWhat we actually built: requirements to deployment traceability
Lesson 1.8 defined FollowBot's requirements as a block diagram and a requirements document. Six stages later, here is the traceability from each requirement to what was implemented and what its current status is.
| Requirement (from Lesson 1.8) | Implemented in | Algorithm/approach | Status in sim | Status on hardware |
|---|---|---|---|---|
| Follow person at 1.0–1.5 m | Lessons 5.5, 6.4, 6.6 | LIDAR cluster + DWA following goal | Working | Working (cooperative person, clear corridor) |
| Stop within 0.5 m of person | Lessons 6.4, 6.9 | DWA cost term + safety stop | Working | Working (tested on flat surfaces) |
| Navigate around static obstacles | Lessons 6.2, 6.4 | A* global + DWA local on costmap | Working | Working |
| Navigate around dynamic obstacles | Lesson 6.4 | DWA local planner only (no prediction) | Partial | Reactive only — no trajectory prediction |
| Build and use an indoor map | Lessons 5.9, 5.10 | SLAM Toolbox (lifelong SLAM) | Working | Working (static env) |
| Handle person occlusion (recover) | Lesson 6.6 | BT: SearchForPerson recovery subtree | Partial | Short occlusions only (<3 s) |
| Hardware E-stop compliance | Lesson 6.9 | Dedicated estop_handler node | N/A | Working |
| Kinematic constraint awareness | Lessons 2.1, 2.9, 6.4 | DWA dynamic window from URDF limits | Working | Working (wheel slip not modeled) |
| 6-DOF arm for handoff gestures | Lessons 2.9, 3.9 | MPC joint trajectory (arm only) | Basic | Not field-tested (scope reduction) |
The Nav2 stack is fully configured: A* global planner on a SLAM Toolbox map, DWA local planner with FollowBot's dynamic constraints, costmap with LIDAR obstacle layer and inflation. The behavior tree orchestrates detect → navigate → maintain following distance → handle occlusion → recover. Sim-to-real checklist has been completed for the core following behavior. The milestone is met. The open items in the "Status on hardware" column above are the honest frontier.
This is what honest engineering documentation looks like. Every "working" in the hardware column is contingent on a set of environmental assumptions. Every "partial" or "not field-tested" is a scope decision that made sense given the course constraints and should be explicit, not buried.
# FollowBot Stage 6 capstone — sim-to-real checklist (partial) # Each item: verified (V), partial (P), or open (O) sensor_calibration: lidar_mounting_tf: V # laser_frame in URDF matches physical mount camera_intrinsics: V # calibrated with checkerboard imu_orientation: V # EKF Q/R tuned on hardware wheel_encoder_tpr: V # 1000 tick/rev confirmed lidar_to_camera_extrinsics: P # estimated, not measured ekf_configuration: process_noise_Q: V # tuned; NEES < 3.0 on straight runs measurement_noise_R: V # tuned per sensor long_corridor_drift: O # degenerate scan match not mitigated nav2_stack: global_planner_inflation: V # 0.20 m inflation (doorways clear) dwa_velocity_model: V # max_vel_x=1.2, max_rot_vel=1.5 rad/s costmap_update_rate: V # 5 Hz LIDAR layer confirmed narrow_gap_recovery: P # reduces inflation, not guaranteed safety: hardware_estop: V # tested: <50 ms stop on button watchdog_timeouts: V # 500 ms per node, confirmed compound_failure: O # multi-node correlated failure not covered behavior_tree: follow_nominal: V # tested 20 min, cooperative person occlusion_recover: P # <3 s; beyond that: person lost social_navigation: O # DWA treats people as obstacles
IN PRACTICEPaths forward: what six more stages would cover
FollowBot is a capable research prototype. There are four directions that would transform it into a production system, each requiring a separate body of knowledge beyond what this course covers.
1. Manipulation integration. The 6-DOF arm was introduced in Lesson 2.9 and given an MPC joint trajectory controller in Lesson 3.9. What was not covered: whole-body control (coordinating base motion and arm simultaneously), grasp planning (computing a pre-grasp pose from a point cloud), contact detection and force control (knowing when the arm has made contact with an object and regulating force). These require courses in manipulation, grasp planning, and force control that parallel this one in depth.
2. Multi-robot coordination. Two FollowBots operating in the same environment will interfere: their local costmaps are independent, their person trackers may track each other's target, their path plans may deadlock in narrow corridors. Multi-robot SLAM, centralized or decentralized fleet management, and multi-agent planning are active research areas. The ROS 2 architecture supports multi-robot namespacing; the algorithms to exploit it are substantially more complex than anything in this course.
3. Learned perception. The person tracker in this course uses LIDAR clustering and RGB-D bounding boxes. A learned detector (YOLOv8 on the RGB-D stream) provides substantially better recall on non-frontal poses, partial occlusion, and crowded scenes. Integrating a learned detector requires calibrating confidence thresholds for the EKF measurement update, handling inference latency (50–150 ms on embedded hardware), and maintaining detection robustness across the distribution shift between the detector's training data and the deployment environment. This is the sim-to-real problem applied to neural networks.
4. Reinforcement learning for local navigation. DWA's velocity model is kinematic: it rolls out constant-velocity arcs and picks the best one. A learned local planner trained with RL (e.g., in a physics simulator with randomized obstacles) can generalize better to crowded, unstructured environments and can implicitly learn social navigation norms from demonstrations. This is an active research direction — algorithms like BARN (Benchmark for Autonomous Robot Navigation) and DRL-VO have demonstrated improvements over DWA in dense crowds. The tradeoff: interpretability and formal safety guarantees are harder to establish for learned policies than for classical planners.
Reinforcement learning for local navigation is promising but not a drop-in replacement for DWA in production. Trained policies fail on out-of-distribution environments (a robot trained in a simulated office may perform poorly in a hospital). The sim-to-real gap for RL is the sim-to-real gap for physics simulation, dynamics randomization, and human behavior models simultaneously. Current research hedges this with domain randomization and conservative velocity limits. The result is often comparable to DWA in normal environments and better in specific high-density scenarios — a genuine improvement, not a revolution.
WEAKNESSWhere this bites: the failure modes of ending a project
Every demo environment is a selection bias. The person cooperates. The lighting is controlled. The corridor is empty. The floor is flat. The battery is fresh. The ROS 2 network is unloaded. Every one of those conditions is a hidden assumption. A robot that "works in the lab" has demonstrated that it works when all of those assumptions hold simultaneously. Field deployment is the experiment that violates them one at a time — and then all at once. The gap between demo and deployment is not a matter of polishing the code. It is a matter of identifying and testing every assumption your system makes and deciding which ones are acceptable to leave unmitigated. That decision is an engineering choice, not a technical one.
It is tempting, at the end of a robotics project, to improve the algorithm with the best-documented failure mode. If the DWA planner sometimes takes suboptimal arcs around obstacles, swap it for TEB. If SLAM Toolbox drifts in corridors, switch to Cartographer. This is frequently the wrong move. Most field failures in person-following robots arise from sensor failures (the LIDAR doesn't see the person) or behavioral edge cases (the person enters a revolving door) — not from suboptimal trajectory curvature. Profile where your actual failures occur before investing in algorithm upgrades. The algorithm that costs you the most field time is almost never the one with the most published papers comparing it to alternatives.
This lesson's tables and checklists are not documentation for its own sake. They are the outputs of a process — systematically asking "under what condition does each subsystem fail, how does that failure propagate, and who is responsible for handling it?" Skipping this process produces systems that fail in the field and leave the debugging team unable to reproduce the failure because nobody wrote down what conditions had been tested. The failure-mode inventory is the most valuable engineering artifact a robotics project can produce, and it is consistently the last thing teams write and the first thing they skip when schedules compress.
KEY INSIGHTHow it connects
- 6.9 · Safety Architecture: Watchdogs, E-Stops, and Speed Governors — the safety layer that sits above the entire Nav2 stack; the watchdog and estop_handler nodes that appear in the architecture diagram above.
- 6.8 · Hardware Bring-Up, Calibration, and Field Diagnostics — the calibration checklist items in the YAML above come from this lesson's systematic bring-up procedure.
- 6.7 · The Sim-to-Real Gap: Taxonomy, Root Causes, and Honest Mitigations — the honest grades table above applies that taxonomy to FollowBot's specific subsystems.
- 6.6 · Behavior Trees: Architecture, Ticks, and Reactive Mission Management — the bt_navigator node and recovery subtrees that appear in the architecture.
- 6.5 · The Nav2 Stack: Architecture, Lifecycle Nodes, and Failure Modes — the lifecycle_manager and the full Nav2 lifecycle state machine.
- 6.4 · Local Planners: DWA, TEB, and Real-Time Obstacle Avoidance — the nav2_controller with DWA; velocity_smoother; the dynamic window parameters in the YAML.
- 6.3 · Sampling-Based Planning: RRT and RRT* — context for why A* (not RRT) is the right global planner for FollowBot's structured indoor environment.
- 6.2 · Graph Search: A*, Heuristics, and Costmap Planning — the nav2_planner node running A* on the SLAM Toolbox map.
- 6.1 · Configuration Space: The Correct Abstraction for Planning — the conceptual foundation that the DWA planner, costmap, and A* all depend on.
- 5.10 · Production SLAM Systems: Cartographer, SLAM Toolbox, and Choosing Wisely — the slam_toolbox node producing the /map topic; the honest comparison that led to choosing SLAM Toolbox over Cartographer.
- 5.9 · SLAM: Simultaneous Localization and Mapping — the theoretical foundation for why FollowBot can build and localize against a map simultaneously.
- 4.9 · Multi-Sensor Fusion on FollowBot: Architecture and Covariance Design — the ekf_filter_node configuration; the Q and R matrix choices that determine estimation quality in the field.
- 3.9 · Model Predictive Control: Constraints as the Killer Feature — the arm trajectory controller; context for why MPC (not PID) is used for constrained joint control.
- 2.9 · Mobile Manipulator Kinematics: When FollowBot Reaches Out — the kinematic chain that connects base_link to arm_base to the arm's end-effector.
- 1.8 · FollowBot System Architecture: Requirements to Block Diagram — the requirements document this lesson traces back to; the TF tree first drawn in that lesson is the TF tree running in production today.
This course has built one robot, end to end, with honest accounting at every step. The algorithms are real, the failure modes are real, and the open questions are real. The most important thing you can carry forward is not any specific algorithm — it is the habit of asking: under what conditions does this work, what happens when those conditions are violated, and have I actually tested it?
GLOSSARYKey terms introduced
- Requirements-to-deployment traceability
- The explicit mapping from each system requirement (written at design time) to the algorithm or subsystem that implements it and the evidence that the implementation satisfies the requirement. Traceability is the engineering discipline that makes "working in the lab" claims auditable.
- Sim-to-real gap (system-level)
- The aggregate deviation between a robot's behavior in simulation and its behavior on physical hardware, arising from the sum of violated assumptions across all subsystems simultaneously. System-level sim-to-real failures are generally not predictable from single-subsystem testing.
- Compound failure mode
- A failure scenario in which two or more subsystems fail simultaneously or sequentially, where each failure's recovery behavior interferes with the others. Behavior trees designed for independent failures often oscillate or fail to converge in compound failure scenarios.
- Social navigation
- Navigation behavior that respects implicit social norms around human movement — yielding appropriately, not blocking pathways, maintaining comfortable distances. Classical planners (DWA, TEB) treat humans as obstacles and do not model social norms; social navigation requires either learned models or hand-coded social force models.
- Failure-mode inventory
- A structured document listing, for each subsystem: the conditions under which it fails, the failure mode's observability (silent vs. loud), the downstream cascade, and the mitigation or known limitation. The most valuable and most commonly skipped engineering artifact in robotics projects.
- SOTIF (Safety of the Intended Functionality)
- ISO/PAS 21448 — a standard addressing functional insufficiencies and reasonably foreseeable misuse scenarios that can cause harm even when the system functions as specified. Distinct from functional safety (ISO 13849/61508), which addresses hardware failures.
- Domain randomization
- A sim-to-real technique for learned policies: training in simulation with randomized physical parameters (friction, mass, sensor noise, lighting) to produce policies robust to the distribution shift between simulator and physical hardware.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1The honest grades table shows that person tracking is "research-grade" in unstructured real-world environments but "excellent" in simulation. What specific assumptions does Gazebo satisfy automatically that the real world does not?›
Gazebo guarantees: a single person in the scene with a distinctive bounding-box shape; no IR interference (Gazebo's depth simulation does not model IR crosstalk); consistent lighting (no sunlight causing overexposure); person always moving in a gait the LIDAR cluster matches (no wheelchairs, luggage carts, or side-by-side walking pairs that merge into one cluster). Critically, Gazebo also guarantees that the person never looks like someone else — re-identification is trivially solved in simulation because there is only one person. The real world violates all of these simultaneously.
2Why does FollowBot's behavior tree perform poorly in compound failure scenarios (person lost AND path blocked simultaneously)?›
Behavior trees compose recovery subtrees assuming independence: the SearchForPerson subtree assumes the robot can move freely to search; the Replan subtree assumes the person is visible to validate replanning. When both failures occur, SearchForPerson tries to move and fails (path blocked), so it returns FAILURE, which triggers Replan. Replan tries to navigate to the last known person position, but the person is lost so there is no valid goal, returning FAILURE. The tree's fallback chain exhausts without finding a valid state, and the root fallback (typically: stop and signal operator) triggers — which is the correct answer but requires the tree to have been designed with compound failure as an explicit case. Trees designed for single failures oscillate rather than converge in compound scenarios.
3A colleague proposes improving FollowBot by replacing DWA with a trained RL policy. What specific questions should you ask before accepting this proposal, and what is the key tradeoff?›
Questions to ask: (1) What simulator was the policy trained in, and how were the dynamics randomized? (2) What is the distribution over environment types and crowd densities in the training distribution, and how does the deployment environment compare? (3) What is the policy's inference latency on the onboard compute, and does it fit within the control loop deadline? (4) What is the failure mode when the policy encounters an out-of-distribution scenario — does it fail loudly (stop) or silently (make bad decisions)? (5) Does the policy satisfy the dynamic constraints (velocity limits, minimum turning radius) that were explicitly encoded in DWA's dynamic window? The key tradeoff: RL local planners generalize better to dense crowds and can implicitly learn social norms, but they are harder to formally verify and their failure modes are less predictable than classical planners. For a research prototype, this is an acceptable tradeoff. For a certified medical device, it is likely a deployment blocker.
4What does the requirements-to-deployment traceability table reveal that the architecture diagram alone does not?›
The architecture diagram shows what is connected and how. The traceability table shows what was promised, what was delivered, and where the gap is. Specifically, it surfaces: (1) requirements that are only partially implemented ("dynamic obstacle avoidance" is reactive, not predictive); (2) requirements that were descoped ("6-DOF arm handoff not field-tested"); (3) the conditions under which each "working" claim holds (flat surfaces; cooperative person; static environment). The architecture diagram is always optimistic — it shows every node connected and functional. The traceability table is honest — it shows which connections deliver what was required and which deliver something less. Without traceability, a future engineer inheriting the system has no way to distinguish "implemented and tested" from "implemented and assumed to work."