ProjekinAja. Robotics Stage 5 · Perception SLAM Lesson 5.5
● Stage 5 · Lesson 5.5

Person Detection and 3D Localization: LIDAR + Camera Fusion

A bounding box tells you what — LIDAR tells you where — and getting the coordinate transform wrong between them means the robot follows a ghost.

⏱ ~22 min read 🔬 Sensor fusion 🧍 Person tracking

🎯 By the end you'll understand

How YOLO-family detectors produce bounding boxes and what confidence threshold and IoU mean in practice. Why projecting a 3D LIDAR cluster onto the 2D image plane requires the extrinsic calibration — and what happens when that calibration is stale. How the Hungarian algorithm assigns LIDAR clusters to camera detections, and why it fails at occlusion boundaries. How to combine the 2D box centroid and 3D cluster centroid into a confidence-weighted 3D person position. And the critical failure mode: a bumped extrinsic makes the robot follow a position that was never where the person stood.

KEY INSIGHTThe camera knows who, the LIDAR knows where — neither alone is enough

A bounding box gives you a rectangle in pixel coordinates. Pixels have no distance. A LIDAR cluster gives you a 3D centroid in sensor space. Clusters have no semantics. The fusion of the two — and only the fusion — gives FollowBot what it needs: a labeled 3D position for the person it is following.

Person following sounds like a computer vision problem. You run a detector, find the bounding box, follow it. But bounding boxes live in 2D pixel space. To move a robot in the physical world, you need 3D coordinates: how far ahead, how far to the left, how high. A monocular camera cannot give you distance without depth estimation, which is a whole inference problem with its own failure modes. A depth camera (RGB-D) can give you depth, but only reliably in the 0.3–3.5 m range and only for non-reflective surfaces. A 2D LIDAR, by contrast, gives you sub-centimeter range accuracy out to 12 m — but it produces unlabeled range arrays, with no way to know which cluster is a person and which is a chair.

🔑 The fusion argument in one sentence

Use the camera's classifier to label the clusters; use the LIDAR's range measurement to localize the label. The only data that flows between them is a 2D projection: take the 3D LIDAR cluster, project it onto the image plane using the calibrated extrinsic, check if it lands inside a detector bounding box. If it does, you have a matched person with a 3D position. If the extrinsic is wrong by even 5°, the projection lands outside the box, the association fails, and the tracker either loses the person or associates with the wrong cluster.

CORE CONCEPTYOLO inference: what a detector actually gives you

YOLOv8 (and its family) runs as a single-pass convolutional network: one forward pass over the image produces a set of bounding-box proposals, each with a class-probability vector and a confidence score. You do not retrain it — you run inference only, with the pretrained COCO weights. On FollowBot's onboard compute (Jetson Orin or NUC), YOLOv8n (nano) runs at 15–25 Hz on the 640×480 camera stream; YOLOv8s runs at 8–15 Hz. These are your realistic operating points.

Two thresholds gate the output:

1

Confidence threshold (conf_thresh)

Each detection proposal carries an objectness × class-probability score in [0, 1]. Proposals below conf_thresh are discarded before you ever see them. A typical value is 0.45–0.60 for person detection indoors. Set it too low and you get false positives — the tracker will associate LIDAR clusters with ghost bounding boxes. Set it too high and you drop real detections during brief occlusions or motion blur. Neither extreme is safe; the right value is determined experimentally on your test environment.

2

IoU threshold (iou_thresh) for non-maximum suppression

When two overlapping boxes both survive the confidence gate, non-maximum suppression (NMS) keeps only the highest-confidence one, suppressing any box whose Intersection-over-Union with it exceeds iou_thresh (typically 0.45). This prevents the same person from generating two bounding boxes. For person following, the consequence of a missed suppression is an extra "ghost" detection that the LIDAR association step will try — and fail — to match, creating spurious tracks.

In ROS 2, you wrap the inference in a node that subscribes to /camera/color/image_raw and publishes to a custom Detection2DArray topic (from vision_msgs). Each detection carries: bbox (center x, center y, width, height in pixels), score, and class_id (0 = person in COCO). Critically, it also carries a header.stamp. You will need this timestamp when synchronising with the LIDAR scan in the fusion step.

yolo_detector_node.pyrclpy · YOLOv8 inference node
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from vision_msgs.msg import Detection2DArray, Detection2D, BoundingBox2D
from cv_bridge import CvBridge
from ultralytics import YOLO
import numpy as np

class YoloDetectorNode(Node):
    def __init__(self):
        super().__init__('yolo_detector')
        self.model = YOLO('yolov8n.pt')          # inference-only; no grad
        self.bridge = CvBridge()
        self.conf  = 0.50
        self.iou   = 0.45
        self.create_subscription(Image, '/camera/color/image_raw',
                                  self.img_cb, 10)
        self.pub = self.create_publisher(Detection2DArray,
                                          '/perception/detections_2d', 10)

    def img_cb(self, msg):
        cv_img = self.bridge.imgmsg_to_cv2(msg, 'bgr8')
        results = self.model.predict(cv_img, conf=self.conf,
                                      iou=self.iou, classes=[0],   # person only
                                      verbose=False)
        arr = Detection2DArray()
        arr.header = msg.header                 # preserve camera timestamp
        for box in results[0].boxes:
            d = Detection2D()
            d.bbox = BoundingBox2D()
            cx, cy, w, h = box.xywh[0].tolist()
            d.bbox.center.position.x, d.bbox.center.position.y = cx, cy
            d.bbox.size_x, d.bbox.size_y = w, h
            d.results.append(...) # ObjectHypothesisWithPose, score=float(box.conf)
            arr.detections.append(d)
        self.pub.publish(arr)

HOW IT WORKSProjecting LIDAR onto the image plane

This is the step where everything can go silently wrong. The LIDAR lives in its own coordinate frame (laser); the camera lives in its own frame (camera_color_optical_frame). To project a LIDAR point onto the image, you need two things: the extrinsic calibration (the rigid-body transform T that maps laser frame → camera frame, encoded as a 4×4 homogeneous matrix or an SE(3) quaternion+translation), and the camera intrinsic matrix K (focal lengths and optical center, calibrated in Lesson 5.1).

The projection in three steps:

1

Transform to camera frame

For each 3D LIDAR point PL = [x, y, z, 1]T (z from the 2D scan is 0; you use the cluster centroid), compute PC = T · PL. T is the extrinsic: a 4×4 matrix encoding rotation R and translation t. If you use TF2, this is a tf2_ros.Buffer.lookup_transform('camera_color_optical_frame', 'laser', rclpy.time.Time()).

2

Apply the intrinsic projection

Project PC = [X, Y, Z]T through K: u = fx · X/Z + cx, v = fy · Y/Z + cy. The result is the pixel coordinate (u, v) where the LIDAR point appears in the image. Check that Z > 0 (point is in front of the camera) and that (u, v) is within image bounds before continuing.

3

Test containment in bounding box

For each LIDAR cluster centroid's projected pixel (u, v), check whether it falls inside any detector bounding box. If yes, this cluster is a candidate for that detection. If no cluster projects into a box, the detection is "unmatched" — camera saw a person, LIDAR did not cluster there (person may be at the edge of LIDAR range, or wearing an IR-absorptive material).

LIDAR cluster centroid P_L [x, y, 0] laser frame Extrinsic T R (rotation) + t (translation) ← stale = failure Camera frame P_C = T · P_L intrinsic K → (u, v) pixel Image plane YOLO bbox hit → match stale T → miss 3D point transform project Green dot: projection hits bbox → cluster-detection match. Red dot: stale extrinsic causes drift → miss → tracker loses person.
The LIDAR-to-camera projection pipeline. The extrinsic T is the single most fragile component: it is calibrated once, stored, and assumed static. A robot bump of 5° in roll can shift the projection by 30+ pixels at 2 m range, enough to miss every bounding box.
📖 Why the 2D LIDAR z-coordinate is 0, not measured

FollowBot's RPLIDAR A2 is a planar scanner. It emits one ring of laser pulses in the horizontal plane and returns [range, angle] pairs — no elevation angle. Each point has x and y in the laser frame, but z is 0 by convention. When we say "project a LIDAR cluster onto the image," we are projecting a horizontal-plane centroid. The camera, which is mounted slightly above the LIDAR plane, sees the person as a full 3D object — but the fused 3D position carries only x and y from the LIDAR (accurate) plus an estimated z (height) that we either fix to a nominal torso height or leave as a prior. For person following, this is acceptable: ground-plane navigation only needs the 2D horizontal position, and the LIDAR provides that exactly.

HOW IT WORKSHungarian algorithm: assigning clusters to detections

At each timestep you have N LIDAR clusters (unlabeled 3D centroids) and M camera detections (labeled 2D bounding boxes). You need to find the optimal assignment — which cluster belongs to which detection — minimizing total cost. This is the linear assignment problem, solved optimally in O(n³) time by the Hungarian algorithm.

The cost matrix entry C[i][j] is the pixel distance between the projected centroid of cluster i and the center of detection j's bounding box. Pairs where the projected centroid falls outside the detection box entirely are given a large penalty cost (e.g., 9999) — this makes them effectively unavailable for assignment. The Hungarian solver finds the minimum-cost assignment. Unmatched clusters (no detection within cost threshold) are labeled "unknown obstacle." Unmatched detections (no cluster projection fell in the box) are labeled "unlocalized person" — you know someone is there but not exactly where in 3D.

hungarian_association.pyscipy · cluster-detection assignment
import numpy as np
from scipy.optimize import linear_sum_assignment

def build_cost_matrix(proj_uv, bboxes, img_w, img_h, penalty=9999.0):
    """
    proj_uv : (N,2) projected pixel coords of N LIDAR cluster centroids
    bboxes  : (M,4) detector boxes [cx, cy, w, h] in pixels
    returns  : (N,M) cost matrix
    """
    N, M = len(proj_uv), len(bboxes)
    C = np.full((N, M), penalty)
    for i, (u, v) in enumerate(proj_uv):
        if u < 0 or u >= img_w or v < 0 or v >= img_h:
            continue                        # point outside image — all penalty
        for j, (cx, cy, bw, bh) in enumerate(bboxes):
            x1, y1 = cx - bw/2, cy - bh/2
            x2, y2 = cx + bw/2, cy + bh/2
            if x1 <= u <= x2 and y1 <= v <= y2:
                C[i, j] = np.hypot(u - cx, v - cy)   # pixel distance to center
    row_ind, col_ind = linear_sum_assignment(C)
    assignments = [(r, c) for r, c in zip(row_ind, col_ind)
                   if C[r, c] < penalty]             # drop penalty pairs
    return assignments

The critical failure happens at occlusions. When the person walks behind a pillar and reappears, they may have a different cluster shape, a different projected centroid, and a new bounding box position. If the ID assignment from the previous frame does not survive the occlusion — because the cost matrix now has a better assignment to a different cluster — the tracker switches IDs. This is the ID-switch problem, treated in depth in Lesson 5.6. For now, understand that the Hungarian algorithm by itself does not track identity over time: it is a per-frame greedy optimal assignment. Identity tracking requires a Kalman filter on top of the assignment (Lesson 5.6).

SEE ITCamera-LIDAR fusion sandbox

Adjust the extrinsic error sliders to introduce rotation and translation drift. Watch the LIDAR cluster's projected pixel drift relative to the bounding box. Observe the 3D position estimate degrade. Then trigger the occlusion test to see the Hungarian ID-switch failure. The widget is deterministic — same slider values always produce the same result.

⚙ Interactive · LIDAR-camera fusion sandbox

Set extrinsic errors → observe projection drift → trigger occlusion → see Hungarian reassignment failure.

0.0°
0 cm
2.0 m
0.75
Projection status
Projected pixel: —
Bbox center: —
Pixel offset: —
In bbox: —
3D position estimate
LIDAR position: —
Fused position: —
Position error: —
Assignment: —

Nominal: no extrinsic error. Projection lands inside bounding box — person matched.

CORE CONCEPTConfidence-weighted fusion of 2D and 3D centroids

Once a cluster is matched to a detection, you have two location signals for the person:

The confidence-weighted fusion combines them:

📖 Confidence-weighted 3D position

Let PL be the 3D LIDAR centroid (high confidence in range) and PC be the camera-derived 3D point (moderate confidence, degrades with distance). Let wL = detector confidence score (0–1) and wC = 1 − wL. Then: Pfused = wL · PL + wC · PC. In practice, LIDAR range is so much more accurate than camera-derived depth that wL is typically 0.75–0.90 for all distances. The camera contributes primarily as a bearing confirmation — the weighted average ensures that if the LIDAR cluster centroid is slightly off (edge of cluster), the camera bearing pulls it back toward the true person center.

This fusion runs at 10 Hz — the rate of the LIDAR scan — not at the camera's 25 Hz. The camera detection is stamped and held until the next LIDAR scan arrives; if no camera detection arrives within ±50 ms of the LIDAR scan, the fusion falls back to LIDAR-only with reduced confidence. The output is a geometry_msgs/PoseWithCovarianceStamped published to /perception/person_position_3d, which the Kalman tracker in Lesson 5.6 will consume.

fusion_node.pyrclpy · confidence-weighted 3D fusion
def fuse_position(self, lidar_3d, camera_3d, det_conf):
    # w_L in [0.75, 0.90] — LIDAR range is the authoritative source
    w_L = 0.75 + 0.15 * det_conf   # higher detector conf → trust LIDAR more
    w_C = 1.0 - w_L
    fused = np.array([
        w_L * lidar_3d[0] + w_C * camera_3d[0],
        w_L * lidar_3d[1] + w_C * camera_3d[1],
        0.0                  # ground-plane navigation; z not used
    ])
    # Covariance: dominated by LIDAR (~0.03 m std) scaled by confidence
    sigma = 0.03 / det_conf       # uncertainty grows as confidence falls
    return fused, sigma

One subtlety: when you call lookup_transform in TF2, you request the transform at the LIDAR scan's timestamp, not at rclpy.time.Time() (which is "now"). This matters because TF2 interpolates between published transforms. If you use "now" while processing a scan that arrived 40 ms ago, you get the current camera position, not the camera position when the scan was taken. On a moving robot, this introduces a parallax error proportional to the robot's velocity. Always use the scan's header stamp.

FOLLOWBOTIntegrating person detection and 3D localization

At this point in FollowBot's development, the camera intrinsics are calibrated (reprojection error < 0.5 px, from Lesson 5.1) and the LIDAR-camera extrinsic is calibrated using a checkerboard target visible to both sensors. The extrinsic calibration residual should be verified explicitly — not just accepted from the calibration tool output — by projecting a known 3D target onto the image and measuring the pixel error. A residual below 3 pixels at 2 m is achievable and necessary; above 8 pixels, the projection will miss bounding boxes for a person at 3 m range.

🔑 Stage 5 milestone check: extrinsic validation

After calibrating the LIDAR-camera extrinsic, publish the LIDAR scan as a colored point cloud (sensor_msgs/PointCloud2) and visualize it in RViz with the camera image overlay. The LIDAR returns for walls, floor, and known objects should align with the image edges at those surfaces. If the wall corners in the point cloud are visibly offset from the wall corners in the image by more than one brick width at 2 m range, your extrinsic needs to be recalibrated or your TF tree has a frame mismatch.

The full FollowBot perception stack at this milestone produces:

1

LIDAR scan → cluster centroid (10 Hz)

DBSCAN or Euclidean clustering on the 2D scan (from Lesson 5.3) produces labeled clusters with 2D centroids in the laser frame. Person-sized clusters (width 0.3–0.8 m, centroid distance 0.5–5.0 m) are forwarded; others are filtered as obstacles.

2

Camera image → bounding boxes (15 Hz)

YOLOv8n inference on the 640×480 color stream produces person bounding boxes with confidence scores. Detections are timestamped and buffered in a 200 ms window for synchronization with the LIDAR scan.

3

Projection + Hungarian assignment (10 Hz, LIDAR-paced)

At each LIDAR scan arrival, the fusion node: (a) retrieves the closest camera detection within ±50 ms, (b) projects each cluster centroid onto the image plane using the TF2-looked-up extrinsic at the scan timestamp, (c) builds the cost matrix, (d) runs scipy.optimize.linear_sum_assignment, (e) produces matched (cluster, detection) pairs.

4

Confidence-weighted 3D fusion → PoseWithCovarianceStamped (10 Hz)

For each matched pair, compute the fused 3D position and publish to /perception/person_position_3d. The Kalman tracker in Lesson 5.6 subscribes here. If no match is found for N consecutive frames (configurable, default N=5), publish a "person lost" event on /perception/person_status.

The result: a stable 3D person position at 10 Hz, with uncertainty covariance. On FollowBot's hardware, this pipeline has an end-to-end latency of approximately 45–70 ms from scan acquisition to published position — well within the tracking Kalman filter's prediction window.

💡 Why LIDAR-paced, not camera-paced

The camera runs at 15–25 Hz; the LIDAR runs at 10 Hz. You could fuse at camera rate by interpolating LIDAR data between scans — but interpolating LIDAR is dangerous (a person can move 5 cm between scans). Instead, fuse at LIDAR rate: the camera detection is a near-synchronous "label" that does not need sub-LIDAR-period accuracy. The LIDAR is the authoritative range source; the camera is the semantic gate. Pace at the slower, more accurate sensor.

WEAKNESSWhere this bites

✗ The stale extrinsic: the silent calibration drift failure

The extrinsic calibration T is computed once and stored in a YAML file loaded at launch. If the robot is bumped — another robot collides with it, a human kicks the LIDAR mount, a screw works loose over 50 km of use — the physical transform between camera and LIDAR changes, but the stored T does not. The projection now drifts. At 3 m range, a 5° rotation error in yaw shifts the projected LIDAR point by approximately 26 cm in the image — at 640×480 with a typical 60° horizontal FOV, that is roughly 50 pixels. A person's bounding box at 3 m is about 120 pixels wide; a 50-pixel shift moves the projected centroid outside the box. The Hungarian algorithm finds no match; the tracker gradually loses the person with no error logged. The robot continues following the last known position — a ghost. This failure is indistinguishable from "the person walked out of LIDAR range" from the log file alone. The fix: a post-collision checklist that re-validates the extrinsic residual, and a runtime sanity check that monitors the average match rate and alerts when it drops below 70% over a 5-second window.

✗ Confidence threshold miscalibration for your environment

YOLOv8 was trained on the COCO dataset — mostly outdoor, well-lit, full-body person images. In a dim corridor, from behind (posterior view), at 4 m range, confidence scores for real persons routinely fall below 0.5. If you set conf_thresh = 0.60 (a reasonable indoor default) without testing your specific environment, you will drop real detections and the fusion node will fall back to LIDAR-only mode — which cannot distinguish a person from a chair. The correct workflow: record a bag file of your test environment, run the detector offline with a sweep of thresholds, plot true-positive vs. false-positive rate, and pick the threshold that maximizes recall for your lighting and viewing angles at the cost of some false positives (which the tracker in Lesson 5.6 will filter). Never use a default threshold in production.

✗ Using "now" instead of the scan timestamp in TF2 lookups

This is a subtle but real bug that appears only when the robot is moving. When you call tf_buffer.lookup_transform('camera_color_optical_frame', 'laser', rclpy.time.Time()), you get the transform at the current wall-clock time. If the LIDAR scan was acquired 45 ms ago (while the robot was turning at 0.5 rad/s), the camera and LIDAR frames have rotated approximately 1.3° since the scan was taken. That rotation translates to ~12 px of shift in the projected point at 2 m range — enough to degrade the match rate. The correct call uses the scan's header stamp: tf_buffer.lookup_transform(..., msg.header.stamp, timeout=...). This is not a theoretical concern; it shows up as degraded tracking performance during FollowBot's turning maneuvers and is one of the first things to check when match rates drop.

KEY INSIGHTHow it connects

The geometry established in this lesson — extrinsic projection, Hungarian assignment, confidence-weighted fusion — is the foundation of every person-position estimate FollowBot produces. Anything that follows it (the tracker in 5.6, the planner in Stage 6) runs on its output. Getting the extrinsic right and monitoring the match rate are not optional polish steps; they are the primary correctness criterion for the entire perception stack.

GLOSSARYKey terms introduced

Extrinsic calibration
The rigid-body transform (rotation R + translation t) that maps 3D coordinates from one sensor's frame to another's — here, from the LIDAR laser frame to the camera optical frame. Stored as a 4×4 homogeneous matrix or SE(3) quaternion+translation. The primary failure point in camera-LIDAR fusion systems.
Confidence threshold (conf_thresh)
The minimum objectness × class-probability score a YOLO detection proposal must exceed to be included in the output. Controls the tradeoff between false positives (low threshold) and missed detections (high threshold). Must be calibrated per environment, not taken from documentation defaults.
Intersection over Union (IoU)
The ratio of the area of overlap between two bounding boxes to the area of their union. Used in non-maximum suppression to decide which overlapping detections to keep (keep the highest-confidence one if IoU > iou_thresh) and can also be used as an association cost metric.
Hungarian algorithm
An O(n³) algorithm that solves the linear assignment problem optimally: given an N×M cost matrix, find the minimum-cost assignment of N rows to M columns such that each row and column is used at most once. Used here to assign LIDAR clusters to camera detections at each timestep.
ID-switch
A tracking failure in which a persistent identity (track ID) is incorrectly reassigned to a different physical object — most commonly at occlusion boundaries, where the tracked person disappears and reappears with a different cluster or bounding box position. The per-frame Hungarian algorithm alone cannot prevent ID-switches; a temporal Kalman tracker is needed.
Confidence-weighted fusion
A simple sensor fusion approach in which the final estimate is a weighted average of two sensor outputs, with weights proportional to their respective confidence or accuracy. Here: P_fused = w_L · P_LIDAR + w_C · P_camera, where w_L dominates because LIDAR range is more accurate than camera-derived depth.
Non-maximum suppression (NMS)
A post-processing step in object detectors that removes redundant bounding box proposals. All proposals are sorted by confidence; for each proposal, any other proposal whose IoU with it exceeds iou_thresh is suppressed. Prevents the same physical object from generating multiple detections.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Why does projecting a LIDAR cluster onto the image plane require the extrinsic calibration, not just the camera's intrinsic matrix K?›

The intrinsic matrix K converts 3D coordinates in the camera's own optical frame into pixel coordinates. It says nothing about where the LIDAR is relative to the camera. The LIDAR cluster exists in the laser frame — a different coordinate system with a different origin and orientation. To project it onto the image, you first need to express the LIDAR point in the camera frame, which requires the extrinsic transform T (a rotation R and translation t). Only after T maps the point into the camera frame can K project it to pixels. Skipping the extrinsic step — using K directly on LIDAR points in the laser frame — produces garbage pixel coordinates unrelated to the actual image geometry.

2FollowBot's match rate between LIDAR clusters and camera detections drops from 85% to 40% over the course of a single run. The detector is still producing confident bounding boxes. What is the most likely failure and how do you diagnose it?›

The most likely failure is a stale extrinsic calibration — the physical transform between the LIDAR and camera has changed (the robot was bumped, a mount screw loosened) but the stored T was not updated. As the projection drifts, fewer LIDAR centroids land inside bounding boxes, so the Hungarian cost matrix fills with penalty values and produces fewer valid assignments. To diagnose: visualize the LIDAR scan as a colored point cloud overlaid on the camera image in RViz. If the point cloud edges are visibly misaligned with the image edges (e.g., a wall corner in the cloud is offset from the corner in the image), the extrinsic is stale. Recalibrate. A secondary check: log the mean pixel distance between matched projected centroids and bbox centers — a rising trend over the run confirms growing extrinsic drift.

3The Hungarian algorithm is "optimal" — why does it still produce ID-switches when the same person walks behind an obstacle and reappears?›

The Hungarian algorithm is optimal for the current frame's cost matrix — it finds the minimum-cost assignment of clusters to detections in a single timestep. It has no memory: it does not know that cluster 3 in frame t was the same person as cluster 7 in frame t+12 (after occlusion). When the person reappears, the new cluster may be closest (in cost) to a different unmatched detection than the original track, and the algorithm assigns it there — correctly minimizing per-frame cost but incorrectly assigning identity. The fix is to maintain persistent track IDs with a Kalman filter that predicts where each tracked person should be, and then match current observations to predicted track positions rather than to the raw current-frame clusters. The Kalman prediction provides temporal continuity that the per-frame Hungarian cannot.

4Why does the fusion node use the LIDAR scan's header timestamp when looking up the TF2 transform, rather than using rclpy.time.Time() (which returns the current time)?›

The LIDAR scan was acquired at a specific moment in the past — typically 45–70 ms ago after traveling through the DDS middleware and driver queue. At that moment, the robot was at a different pose than it is "now." If you look up the transform at the current time, you get the current camera-to-laser transform — which reflects the robot's current pose, not its pose when the scan was taken. On a robot turning at 0.5 rad/s, 50 ms of pose difference corresponds to approximately 1.4° of rotation, which at 2 m range projects to about 10 pixels of error in the image. By using the scan's header.stamp, TF2 interpolates its transform buffer to return the camera-laser transform exactly as it was when the scan was acquired, eliminating this pose-lag parallax error.

Next: object tracking — keeping identity across frames →