ProjekinAja. Robotics Stage 5 · Perception SLAM Lesson 5.4
● Stage 5 · Lesson 5.4

Feature Detection and Visual Odometry

Visual odometry gives you metric ego-motion from a camera — but scale ambiguity in monocular VO means you are always estimating "movement," never "distance," without additional information.

⏱ ~22 min read 🎯 Feature tracking ⚠ Scale ambiguity

🎯 By the end you'll understand

How Harris, FAST, and ORB detect repeatable image features, and why corner response measures what they do. How ORB descriptors enable fast binary matching and what the Hamming distance threshold actually controls. Why KLT optical flow tracks without descriptor extraction and when it fails first. How RANSAC estimates the essential matrix robustly by iterating over minimal samples — and why an inlier fraction below ~40% breaks the statistical guarantee. How R and t are recovered from E, and why that t is only a direction vector in monocular VO. How IMU integration or known camera height restores the missing scale factor.

KEY INSIGHTA camera is a ruler with no units

A camera projects the 3D world onto a 2D image plane, and that projection destroys one degree of freedom — the depth of every point. You can measure angles precisely. You cannot measure distances without help.

Visual odometry estimates the camera's motion between consecutive frames by tracking where image features move. Get enough correspondences between frame N and frame N+1, filter the outliers with RANSAC, decompose the essential matrix, and you recover the rotation R and a translation direction t. That direction is all monocular VO ever gives you — a unit vector pointing the way the camera moved. The actual distance requires an external anchor: a stereo baseline, an IMU integration, or a known object of fixed physical size. Every monocular VO system that reports "moved 1.2 meters" is borrowing that number from somewhere other than the camera alone.

🔑 Hold onto this

Scale ambiguity is not a bug, a failure mode, or a calibration error. It is a fundamental consequence of projective geometry: multiplying the scene by any scalar k while dividing the camera translation by k produces identical images. RANSAC and essential matrix decomposition are exact — they just can't distinguish these cases. The fix always involves information that is not in the image: a second camera at a known separation (stereo), an IMU whose accelerometer integration provides metric scale, or explicit knowledge of something in the scene whose physical size is known. FollowBot has all three potential anchors. Most single-camera drones have only the IMU.

CORE CONCEPTFeature detectors: what makes a good keypoint

Visual odometry works by finding specific image locations that are reliably detectable across frames despite small viewpoint changes, lighting variation, and image noise. These are keypoints, and the algorithm that finds them is the detector. Three dominate practical robotics use.

1

Harris corners — the original corner response measure

Harris (1988) asks: for a small image patch centered at pixel (x, y), how much does the intensity change when you shift the patch in any direction? Compute the structure tensor M — a 2×2 matrix of squared intensity gradients averaged over the patch:

harris.py
# Ix, Iy: image gradient x and y (e.g. Sobel)
M = [[sum(Ix*Ix), sum(Ix*Iy)],
     [sum(Ix*Iy), sum(Iy*Iy)]]

# Corner response R: large when both eigenvalues are large
det_M  = M[0][0]*M[1][1] - M[0][1]*M[1][0]
trace_M = M[0][0] + M[1][1]
R = det_M - k * trace_M**2   # k ≈ 0.04–0.06

When R is large and positive: corner. When negative: edge (one large eigenvalue). When near zero: flat region. Corners have large intensity gradients in two directions — the structure tensor's eigenvalues are both large — so the patch shifts away from the canonical position in every direction. That makes them unique and repeatable. Harris is rotationally invariant but not scale-invariant.

2

FAST — Features from Accelerated Segment Test

Harris computes gradients over a full patch — expensive at high resolution. FAST (Rosten & Drummond 2006) takes a more direct route: a candidate corner pixel p passes the test if at least N contiguous pixels on a circle of radius 3 pixels around p are all brighter than p + threshold, or all darker than p − threshold (N is typically 9 out of 16 circle pixels). This can be evaluated in a few comparisons using a learned decision tree, making FAST roughly 10–100× faster than Harris. FAST has no scale invariance or rotation invariance by itself — that's why it is combined with orientation assignment and scale pyramids in ORB.

3

ORB — Oriented FAST + Rotated BRIEF

ORB (Rublee et al., ICCV 2011) combines FAST keypoint detection with a rotation-aware binary descriptor. The orientation is computed from the image moment of the patch — the intensity-weighted centroid relative to the center gives a canonical orientation angle. BRIEF (the base descriptor) takes N pairs of pixel intensity comparisons within the patch and encodes each comparison as a 0 or 1 bit. ORB steers the BRIEF sampling pattern by the computed orientation, making the descriptor rotation-invariant. The result is a 256-bit binary string per keypoint — matching two descriptors requires only a Hamming distance computation (XOR + popcount), fast enough to run on a CPU at 30 Hz without a GPU. ORB is the default feature detector in ORB-SLAM2 and ORB-SLAM3.

Flat region ∇I ≈ 0 everywhere low intensity gradient in all directions R ≈ 0 (skip) Edge R < 0 (edge) Corner ✓ R > 0 (keep)
Harris corner response R: flat regions (R ≈ 0) and edges (R < 0) are discarded. Only corners — where intensity gradients are large in two orthogonal directions — produce large positive R. ORB uses FAST for speed but adds orientation assignment to achieve rotation invariance.

CORE CONCEPTFeature matching: descriptor distance and ratio test

Detected keypoints in frame N are paired with keypoints in frame N+1 by comparing their descriptors. For ORB, this means computing the Hamming distance between 256-bit strings — the number of bits that differ (XOR + popcount). Lower Hamming distance = better match.

Two critical decisions control match quality:

💡 Lowe's ratio test — the most important filter in the pipeline

For each keypoint in frame N, find its two nearest descriptor neighbors in frame N+1 (by Hamming distance). Accept the match only if d1 / d2 < 0.75 (Lowe's original threshold). The intuition: if the best and second-best match have similar distances, the feature is ambiguous — a wall of identical bricks, a repeating pattern, a window grid. When the best match is clearly closer (ratio < 0.75), the match is distinctive enough to trust. This single filter typically removes 50–80% of incorrect matches before RANSAC even starts.

After the ratio test, you still have outliers — incorrect matches that pass the distance threshold because two different scene points happen to have similar descriptors. That is exactly what RANSAC is for: robustly estimating a geometric model (the essential matrix) despite a fraction of corrupted correspondences.

Detector / DescriptorMatch metricSpeedScale invariantUsed in
ORBHamming (XOR)Very fast (CPU)Approximate (pyramid)ORB-SLAM2/3, ROS image_pipeline
SIFTL2 (Euclidean)Slow (float)YesOffline reconstruction
SURFL2MediumYesLegacy; patent expired 2020
AKAZEHammingFastYes (nonlinear scale space)Embedded VO systems

HOW IT WORKSKLT optical flow: tracking without descriptor matching

The descriptor approach — detect, describe, match — is powerful but expensive per frame. Lucas-Kanade tracking (1981), extended by Tomasi and Kanade and integrated with pyramidal search by Bouguet (2001), takes a different approach: assume that a patch of pixels around a keypoint moves with a velocity (u, v) that can be estimated by solving a linear system derived from the brightness constancy assumption.

The brightness constancy assumption: the intensity of a physical patch in the world does not change between frames, only its position in the image. If I(x, y, t) is the intensity at pixel (x,y) at time t:

📖 Optical flow constraint equation

dI/dt + (∂I/∂x)·u + (∂I/∂y)·v = 0, or in matrix form: [Ix, Iy] · [u, v]ᵀ = −It. One equation, two unknowns — the aperture problem. KLT solves this over a window of pixels (overconstrained system, least-squares) and propagates the solution across a Gaussian image pyramid to handle large displacements between frames.

KLT does not compute descriptors. Instead it initializes a set of "good features to track" (typically Harris corners) in the first frame and then propagates their positions forward frame-by-frame. This is fast — OpenCV's cv2.calcOpticalFlowPyrLK tracks 300 points at 30 Hz on a Raspberry Pi 4 — but accumulates drift because errors compound each frame. Descriptor-based matching (ORB, SIFT) restarts every frame and is resistant to drift, but cannot run as fast.

⚠ KLT assumes small inter-frame motion

The linear brightness constancy equation is only valid when the true displacement is small enough that the linearization holds — typically under 2–3 pixels between frames. The pyramid extension (typically 3–4 levels) relaxes this to ~30 pixels by tracking on a downsampled image first, but fast camera motion (sudden pan, dropped frame, wheel slip) or large rotation can still cause patches to drift to a wrong local minimum. When this happens, KLT reports a confidence flag (the residual), but learners often miss it. Always check status output from calcOpticalFlowPyrLK and discard tracks where status is 0 or the residual exceeds a threshold.

HOW IT WORKSRANSAC, the essential matrix, and recovering R and t

After matching or tracking features, you have a set of 2D-to-2D correspondences: pixel (u₁, v₁) in frame 1 matches pixel (u₂, v₂) in frame 2, in normalized image coordinates (divide by the focal length and subtract the principal point). The geometric constraint these correspondences must satisfy is the epipolar constraint:

x₂ᵀ · E · x₁ = 0

where x₁ and x₂ are the normalized homogeneous image coordinates of the matching points, and E is the 3×3 essential matrix encoding the relative rotation R and translation t between the two camera poses: E = [t]×R. The essential matrix has 5 degrees of freedom (R has 3, t has 2 because only direction, not magnitude, is recoverable). With 5 or more point correspondences you can solve for E.

RANSAC for robust estimation:

1

Minimal sample

Randomly draw 5 point pairs (the minimum needed to uniquely determine E using the 5-point algorithm, Nister 2004). Solve for up to 10 candidate E matrices (the 5-point polynomial system has up to 10 solutions).

2

Count inliers

For each candidate E, compute the Sampson distance (a symmetric algebraic reprojection error) for every correspondence. A point pair is an inlier if its Sampson distance is below the RANSAC threshold τ (typically 1–3 pixels). Count the inliers.

3

Store best model

If this candidate has more inliers than any previous candidate, store it as the current best model.

4

Adaptive iterations

Repeat until the probability of having missed a good model is below a target (typically 99%). The required number of iterations N = log(1-p) / log(1 - ε⁵), where ε is the estimated inlier fraction. At ε = 0.5, N ≈ 145 iterations to achieve p = 0.99.

5

Re-estimate with all inliers

Use all inliers of the best model to fit a final E by SVD decomposition, improving accuracy. Then decompose E to recover R and t.

Decomposing E into R and t: SVD of E = U Σ Vᵀ yields four candidate (R, t) solutions. Three are geometric phantoms (cameras behind the scene); the correct one is selected by verifying that the reconstructed 3D points are in front of both cameras (positive depth test). The recovered t is a unit vector — it tells you which direction the camera moved but not how far.

📖 Sampson distance vs. symmetric reprojection error

Symmetric reprojection error requires triangulating each point to 3D and reprojecting into both images — expensive inside a RANSAC loop with thousands of iterations. Sampson distance is a first-order approximation that avoids the triangulation and costs only a few scalar multiplications per point pair. It is slightly less accurate than the true reprojection error but accurate enough for inlier classification. The final refit after RANSAC uses the full SVD on all inliers, recovering accuracy.

SEE ITMonocular VO step-runner

Step through the complete monocular VO pipeline on a pair of synthetic frames. Adjust the RANSAC threshold and scene texture, then deliberately create a low-texture scene and watch what happens to inlier count. Compare monocular vs. stereo to see scale ambiguity directly.

⚙ Interactive · Monocular VO pipeline step-runner

Choose scene texture, RANSAC threshold, and sensor mode. Then step through: Detect → Match → RANSAC → Inlier split → Essential matrix → Recover R, t.

Step 1 of 6
Frame N (t) Frame N+1 (t+Δt) → ego-motion
Keypoints detected
—
per frame
Matches after ratio test
—
Lowe ratio < 0.75
RANSAC inliers
—
— of matches
Scale
—
—

CORE CONCEPTScale ambiguity: why monocular VO never gives you meters

The projective geometry argument is clean. If every 3D point X is scaled by factor k (the scene becomes k times larger) and the camera translation T is divided by k (the camera moves k times less), the images are identical. The essential matrix E is the same. RANSAC selects the same inliers. R and the direction of t are the same. Only the magnitude of t differs — and the camera has no way to distinguish these scenarios.

Config A: scale = 1 cam 1 cam 2 t = 1.0 m depth ≈ 5 m ≡ same images Config B: scale = 2 cam 1 cam 2 t = 0.5 m depth ≈ 10 m Resolution Stereo baseline b |t| = b (metric) IMU integration ΔX = ∫∫ a dt² (metric) Known height h ground plane anchors scale
Scale ambiguity in monocular VO. Config A (scale = 1) and Config B (scale = 2) produce identical image sequences — E, R, and the direction of t are identical. The only way to distinguish them is an external metric anchor: stereo baseline, IMU, or known scene geometry.
Monocular VO

Recovers R (full rotation) and t̂ (unit direction vector). Inertial-monocular fusion (VIO) restores scale by tightly coupling IMU pre-integration. Scale drift accumulates over long trajectories without loop closure.

Stereo VO

Recovers R and metric t directly — the stereo baseline b is physically calibrated. More robust to low texture because depth can be estimated from disparity even without motion. Heavier to calibrate (6 DOF extrinsic + two intrinsics).

RGB-D VO

Depth sensor provides absolute per-pixel depth — no scale ambiguity. ICP or photometric alignment on the depth channel. Fails in outdoor environments (IR structured light / ToF saturation in sunlight). FollowBot's RealSense D435 is RGB-D.

💡 Using known camera height on FollowBot

FollowBot mounts its camera at a fixed height h above the floor. With known camera intrinsics and calibrated roll/pitch from the IMU, you can estimate the horizon line in the image and use the ground plane homography to recover metric scale frame-to-frame — a lightweight alternative to full IMU integration. This is exactly what many autonomous vehicles do to initialize VO before the GPS signal improves. The key requirement: the floor must be visible in a significant portion of the image, which rules out looking straight ahead at a person's torso without a downward component in the camera FOV.

FOLLOWBOTIntegrating VO into FollowBot's state estimation stack

FollowBot already uses wheel encoders for odometry and an IMU for heading. Visual odometry adds a third, complementary measurement: drift-resistant pose change estimation that does not depend on wheel-ground contact. Each sensor has a different failure mode, and the EKF fuses them to exploit complementarity.

🔑 Stage 5 milestone state

At this point in FollowBot's build: camera intrinsics are calibrated (reprojection error < 0.5 px); LIDAR-camera extrinsics are calibrated; YOLOv8 person detection is integrated; LIDAR cluster + camera bbox fusion is producing a stable 3D person position at 10 Hz; SLAM Toolbox is building an occupancy map. VO now adds continuous ego-motion estimation between SLAM keyframes, reducing drift during the intervals between loop closures.

visual_odometry_node.pyrclpy · ORB + RANSAC VO node skeleton
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from nav_msgs.msg import Odometry
from cv_bridge import CvBridge
import cv2
import numpy as np

class VisualOdometryNode(Node):
    def __init__(self):
        super().__init__('visual_odometry')
        self.bridge = CvBridge()
        self.orb = cv2.ORB_create(500)          # 500 keypoints per frame
        self.bf  = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=False)
        # camera intrinsics — set from calibration, not defaults
        self.K = np.array([[615.0, 0, 320.0],
                             [0, 615.0, 240.0],
                             [0, 0, 1.0]])
        self.prev_gray = None
        self.prev_kp   = None
        self.prev_des  = None
        self.create_subscription(Image, '/camera/color/image_raw',
                                self.image_cb, 10)
        self.pub = self.create_publisher(Odometry, '/vo/odom', 10)

    def image_cb(self, msg):
        gray = self.bridge.imgmsg_to_cv2(msg, desired_encoding='mono8')
        kp, des = self.orb.detectAndCompute(gray, None)
        if self.prev_des is None or des is None or len(kp) < 10:
            self.prev_gray, self.prev_kp, self.prev_des = gray, kp, des
            return
        # kNN match + Lowe ratio test
        matches_knn = self.bf.knnMatch(self.prev_des, des, k=2)
        good = [m for m, n in matches_knn if m.distance < 0.75 * n.distance]
        if len(good) < 10:
            self.get_logger().warn(ff'VO: only {len(good)} matches — skipping')
            return
        pts1 = np.float32([self.prev_kp[m.queryIdx].pt for m in good])
        pts2 = np.float32([kp[m.trainIdx].pt for m in good])
        # Essential matrix with RANSAC (5-point, threshold=2 px)
        E, mask = cv2.findEssentialMat(pts1, pts2, self.K,
                                        method=cv2.RANSAC, prob=0.999,
                                        threshold=2.0)
        if E is None:
            return
        inliers = int(mask.sum()) if mask is not None else 0
        inlier_frac = inliers / len(good)
        if inlier_frac < 0.4:               # ← CRITICAL: reject bad E estimates
            self.get_logger().warn(
                ff'RANSAC inlier fraction {inlier_frac:.2f} < 0.4 — pose unreliable')
            return
        _, R, t, _ = cv2.recoverPose(E, pts1, pts2, self.K, mask=mask)
        # NOTE: t is a unit vector — scale is unknown in monocular mode
        self.prev_gray, self.prev_kp, self.prev_des = gray, kp, des
        # publish odom with scale-ambiguous t (covariance reflects uncertainty)
        self.publish_odom(R, t, inlier_frac)

    def publish_odom(self, R, t, inlier_frac):
        pass  # fill quaternion from R, set covariance ∝ 1/inlier_frac
⚠ Always check inlier_frac before trusting the pose

The highlighted line above is load-bearing. cv2.findEssentialMat will return an E matrix even when the inlier fraction is 10% — it just returns the best model it found, which at low inlier fractions is statistically likely to be wrong. A returned E does not mean a reliable E. Check inlier count and inlier fraction. Below ~40% inliers, discard the frame pair and rely on the EKF's motion model prediction. Log the warning — repeated warnings indicate a scene that is too texturally poor for ORB VO, and the architecture should switch to LiDAR-based odometry or IMU dead-reckoning.

In FollowBot's full EKF stack, the VO pose change feeds as a measurement update at approximately 15 Hz (limited by ORB processing time on the Jetson). The wheel odometry feeds at 100 Hz. The IMU feeds at 200 Hz. The EKF fuses all three using a process noise model that is weighted by the inverse of each sensor's covariance — VO with low inlier fraction gets high covariance and is effectively down-weighted automatically. The scale ambiguity is handled by using only the rotation component from monocular VO and obtaining translation magnitude from the IMU pre-integration, following the VIO architecture of VINS-Mono.

WEAKNESSWhere this bites in the field

⚠ RANSAC returns a wrong model when inlier fraction drops below ~40%

The RANSAC statistical guarantee depends on the inlier fraction ε. At ε = 0.5 with N = 145 iterations, p(finding correct model) ≥ 0.99. At ε = 0.3, you need N ≈ 1893 iterations for the same guarantee (N = log(1−0.99)/log(1−0.3⁵) ≈ 1893) — many VO implementations cap iterations at 200 for speed, silently running well under the required count. At ε = 0.2 (8 inliers from 40 matches — a low-texture corridor with a repeating tiling pattern), the correct model may never be sampled, and RANSAC returns the least-wrong wrong model. This is an especially nasty failure: findEssentialMat returns a status code of success, E looks numerically plausible, and the recovered R and t are plausible but wrong. The EKF will confidently integrate a biased measurement until loop closure corrects it. Always monitor inlier count, not just the function return status.

⚠ Degeneracy: pure rotation produces no unique essential matrix

The essential matrix encodes both rotation and translation. When the camera rotates in place with zero translation — FollowBot spinning to face the person — the epipolar geometry degenerates. The essential matrix is no longer uniquely determined from 5 point correspondences. Practical symptom: RANSAC returns an E but recoverPose produces an implausible t (large sideways translation where only rotation occurred). The fix: detect near-zero translation frames by checking the inlier count after removing pure-rotation hypothesis, or by fusing with IMU linear acceleration magnitude. ORB-SLAM3 explicitly handles this degenerate case; a naive VO implementation does not.

⚠ ORB fails on blur, uniform surfaces, and specular reflections

ORB's FAST detector requires local intensity variation: corners, edges, texture boundaries. A white wall, a glass door reflecting the camera, a motion-blurred frame from a fast pan — all produce few or no stable keypoints. Fewer than 10 good matches after the ratio test means RANSAC has no meaningful data. The VO node should detect this and publish a high-covariance "skip" measurement rather than interpolating the last pose or silently dropping the update. In FollowBot's operating environment (indoor corridors), the most common failure trigger is long stretches of painted drywall — exactly the surfaces that dominate hospital and office building interiors. Maintain a feature count health monitor and trigger a sensor-fusion reconfiguration (higher EKF process noise; rely more on wheel odom) when VO goes dark.

KEY INSIGHTHow this connects

Feature detection and visual odometry sit at the intersection of geometry, statistics, and real-time systems engineering. The algorithms are well understood — what separates a working VO system from a textbook VO system is disciplined handling of the degenerate cases: low texture, pure rotation, motion blur, RANSAC failure. Monocular VO without scale resolution is a compass with no ruler. Build the ruler before depending on the distance.

GLOSSARYKey terms introduced

Visual odometry (VO)
Estimating ego-motion (rotation and translation) from consecutive camera frames by tracking or matching image features. Monocular VO recovers rotation and translation direction but not scale. Stereo and RGB-D VO recover metric translation.
Essential matrix (E)
A 3×3 matrix with rank 2 that encodes the relative rotation R and translation direction t between two camera poses, acting on normalized image coordinates. x₂ᵀ E x₁ = 0 for all true correspondences (epipolar constraint). Has 5 DOF.
RANSAC (Random Sample Consensus)
A robust estimation framework that iteratively samples minimal subsets of data, fits a model to each, and counts inliers within a threshold distance. Tolerates an outlier fraction up to ~50%; reliability degrades sharply below ~40% inliers.
ORB descriptor
A 256-bit binary feature descriptor computed by steering BRIEF's intensity comparisons by the keypoint's orientation (computed from intensity moments). Matched by Hamming distance; fast enough for real-time CPU use.
KLT optical flow
Lucas-Kanade tracking: estimates patch displacement between frames by assuming brightness constancy and solving a least-squares system over a window of pixels. Runs at very high frame rates but accumulates drift; best used with periodic re-initialization.
Scale ambiguity
The fundamental indeterminacy in monocular VO: multiplying scene depths by k and dividing camera translation magnitude by k produces identical images. The essential matrix and decomposed R, t̂ are unchanged. Resolved by stereo, IMU, or known geometry.
Sampson distance
A first-order algebraic approximation to the symmetric reprojection error, used as the RANSAC inlier metric for the essential matrix. Avoids expensive triangulation inside the RANSAC loop while closely approximating the true geometric error.
Inlier fraction (ε)
The proportion of correspondences that are consistent with the true geometric model (within the RANSAC threshold). Determines the required number of RANSAC iterations for a given confidence level. Below ~40%, practical RANSAC implementations often fail silently.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Why does the Harris detector compute the eigenvalues of the structure tensor M rather than just the gradient magnitude?›

Gradient magnitude alone (|∇I|) is large at both edges and corners. The structure tensor M captures the distribution of gradient directions within a patch. Its eigenvalues reveal whether the patch has large gradients in one direction (edge: one large eigenvalue, one near-zero) or in two orthogonal directions (corner: both eigenvalues large). A flat region has both eigenvalues near zero. The corner response R = det(M) − k·trace(M)² is an efficient proxy that avoids explicitly computing eigenvalues. Only corners — where the intensity pattern is distinctive in all directions — make reliable keypoints for VO: you can localize them sub-pixel in both x and y, whereas edges are poorly localized perpendicular to the edge direction.

2A VO node reports E successfully but the recovered trajectory curves 20° to the right of the true path. You know the scene has 200 detected features, 80 matches after the ratio test, and 22 RANSAC inliers. What is the most likely root cause?›

22 inliers from 80 matches is an inlier fraction of 27.5% — well below the 40% threshold where RANSAC's statistical guarantee begins to break down. At this fraction, RANSAC likely returned a locally optimal but globally incorrect E, one that happens to fit 22 outlier correspondences that share a systematic bias (e.g., a repeating texture pattern where many feature pairs have similar descriptors, creating false correspondences with a consistent geometric offset). The fix is to reject this frame pair entirely when inlier fraction is below 0.4 and either interpolate from the IMU motion model or wait for the next frame. Checking the scene: 80 matches from 200 detected features is reasonable, but the low inlier fraction suggests many of those matches are wrong — typical of a corridor with a repeating tile or wallpaper pattern where the ratio test passes but the correspondences are to geometrically incorrect image positions.

3FollowBot is spinning in place to face a person who stepped sideways. The VO node calls cv2.findEssentialMat and gets an E back with a high inlier count. Why should you still be skeptical of the recovered t?›

This is the pure-rotation degeneracy. When the camera rotates with zero (or near-zero) translation, all feature correspondences satisfy the epipolar constraint for any translation direction, because the epipolar lines degenerate — they all pass through the same point (the epipole goes to infinity). The essential matrix is under-constrained: many E matrices explain the data equally well, including ones with large spurious translations. The 5-point algorithm may still return a numerically stable E (the RANSAC inlier count looks good because any consistent rotation-only E fits all points), but recoverPose will produce a t that is essentially noise amplified. To detect this: check that the IMU linear acceleration magnitude is above a threshold before trusting the t vector. ORB-SLAM3 explicitly detects the degenerate motion class and handles it separately. A naive implementation will inject a bad translation measurement into the EKF.

4You add a stereo camera to FollowBot (baseline b = 0.12 m). Explain precisely why scale ambiguity is resolved, and name one failure mode that stereo VO has that monocular VO does not.›

In stereo VO, the inter-camera translation is physically fixed and known from extrinsic calibration: |t| = b = 0.12 m. When you recover R and t from the stereo pair's essential matrix, you can set the translation magnitude to b rather than leaving it as a unit vector. Triangulated 3D points from the stereo pair then have absolute metric depth (disparity × b / f). The ego-motion scale is anchored by the physical baseline. A failure mode specific to stereo VO: if both cameras observe the same texturally uniform surface (a white wall filling both frames), neither camera produces enough good keypoints, and stereo matching fails. Monocular VO degrades gracefully in this case (few matches, high covariance) but still produces a rotation estimate if any features remain. Stereo also requires more careful extrinsic calibration — a thermal expansion or mechanical shock that shifts one camera relative to the other by 2 mm changes b by ~1.7%, introducing a systematic scale error that monocular VO (which adapts to each frame) would not have.

Next: person detection and 3D localization — LIDAR cluster + camera bbox fusion →