Stereo Vision and RGB-D Depth Sensing
Depth uncertainty grows quadratically with distance for stereo — knowing this quantitatively tells you when to trust the depth camera and when to trust the LIDAR.
🎯 By the end you'll understand
How epipolar geometry turns a left-right image pair into a disparity map. The exact derivation of Z = fB/d and why depth uncertainty grows as the square of distance. What baseline and focal length trade-offs actually mean for a real camera like the RealSense D435. The three physical principles behind depth sensing — passive stereo, structured light, and time-of-flight — and the specific conditions that break each one. Why outdoor daylight kills active IR sensors and what that means for FollowBot leaving an indoor environment.
KEY INSIGHTDepth is not free — its uncertainty is a precise, unavoidable function of geometry
A depth camera does not "see distance." It infers distance from disparity, and the uncertainty in that inference compounds as the square of the distance — no firmware update will fix this.
When you hold up two fingers in front of your face and close each eye alternately, a nearby finger appears to jump more than a distant one. That shift is disparity — the angular difference in apparent position between the two views. Your brain exploits this to perceive depth. Stereo cameras do the same thing, and the governing math is simple enough to derive on a napkin. The consequential part is what happens to the error: a 1-pixel uncertainty in disparity measurement propagates into a depth error that grows as Z² / (fB). At 1 m from a typical robot camera, that might be 3 cm. At 4 m, it is 50 cm. At 6 m, the "depth" reading is largely noise. Understanding that quadratic law — not the algorithm behind it, not the OpenCV call — is the real engineering insight of this lesson.
Depth cameras give you dense, rich, low-latency depth data at close range and near-useless data beyond their reliable limit. LIDAR gives you sparse but accurate ranging to 12–30 m. These are not competing sensors — they are complementary sensors with complementary failure modes. FollowBot uses both, and your job as a systems engineer is to know where each sensor's uncertainty budget is acceptable for the downstream task.
HOW IT WORKSEpipolar geometry: why two cameras are enough for depth
Two cameras separated by a known baseline B and looking in the same direction form a stereo pair. After lens distortion is removed (calibration — covered in Lesson 5.1) and the images are rectified so matching points lie on the same horizontal scanline, the core geometry is two similar triangles.
The derivation is a two-triangle argument. For the left camera, the projection of P onto the image plane obeys:
The left camera projects P at image coordinate x_L = f · X_L / Z, where X_L is the lateral offset of P from the left optical axis and f is the focal length in pixels. The right camera sees P at x_R = f · (X_L − B) / Z (P has shifted left by the baseline B in the right camera's frame). Their difference is the disparity:
d = x_L − x_R = f · B / Z
Rearranging for depth: Z = f · B / d. This is the stereo depth equation — exact under the pinhole model, real under calibration.
The equation tells you everything important without algebra. Depth Z is inversely proportional to disparity d. When the target is close, d is large and easy to measure. When the target is far, d shrinks toward zero and measurement noise dominates. There is also a minimum depth: when the target is so close that its disparity exceeds the image width, the right camera cannot see it at all. For the RealSense D435 (baseline ≈ 55 mm, f ≈ 1.93 mm effective, image 848 × 480): minimum depth ≈ 0.17 m, reliable range ≈ 0.3–5 m.
Stereo rectification: making the geometry work in practice
In theory, epipolar lines are perfectly horizontal after rectification. In practice, two cameras are never perfectly aligned, their image sensors have different intrinsics, and lenses introduce distortion. Stereo rectification is the pre-processing step that transforms both images so they conform to the ideal parallel-camera geometry. It requires both cameras' intrinsic parameters and their relative pose (the extrinsic calibration). In ROS 2, the image_proc package provides /image_rect_color and /image_rect topics that publish rectified imagery from raw sensor feeds.
A miscalibrated stereo rig is the dominant source of systematic depth error. Epipolar lines that are slightly non-horizontal cause the stereo matching algorithm (block matching, semi-global matching) to search in the wrong scanlines, producing wrong disparities — and unlike sensor noise, these errors are systematic: points at specific depths will always be wrong, in the same direction.
CORE CONCEPTDepth uncertainty: the Z² law
The precision of a stereo depth measurement is bounded by how precisely you can measure disparity. Even with perfect calibration, disparity can only be measured to some minimum uncertainty σ_d (typically 0.1–1.0 pixels for block-matching algorithms on good texture). To find the resulting depth uncertainty, differentiate Z = fB/d with respect to d:
Differentiating Z = fB/d: dZ/dd = −fB/d². By the chain rule of error propagation, depth uncertainty σ_Z relates to disparity uncertainty σ_d by:
σ_Z = (Z² / (f · B)) · σ_d
Depth uncertainty grows as the square of depth, divided by the product of focal length and baseline. The only engineering levers are f and B — and both appear linearly in the denominator, so doubling either halves σ_Z at all ranges.
This is the law that determines where a stereo camera is useful and where it is not. A wider baseline improves accuracy but requires more physical space and creates a larger "stereo blind zone" where objects are too close for both cameras to see simultaneously. A longer focal length (zoom) improves accuracy but narrows the field of view. There is no free lunch — these are fundamental physics tradeoffs, not software options.
| Range Z | σ_Z (D435 default: B=55 mm, f≈639 px) | Practical interpretation for FollowBot |
|---|---|---|
| 0.5 m | ~4 mm | Sub-centimeter — excellent for arm grasp planning |
| 1.0 m | ~14 mm | 1.4 cm — very good for person tracking at contact range |
| 2.0 m | ~57 mm | 5.7 cm — marginal for person following; acceptable fused with LIDAR |
| 4.0 m | ~228 mm | 23 cm — depth is largely unreliable; use LIDAR only |
| 6.0 m | ~512 mm | 51 cm — noise floor; treat depth map as invalid |
These numbers use σ_d ≈ 0.5 px (a reasonable estimate for the D435's on-device stereo processor). The table explains why FollowBot's person-following algorithm switches to LIDAR as the primary range sensor beyond 3 m and uses the D435 depth primarily for close-in navigation and arm workspace sensing.
Edge dropout: where stereo matching silently fails
Stereo matching correlates patches of pixels between left and right images. Edges and depth discontinuities break this assumption: a pixel on the edge of an object projects to different backgrounds in the two images, making block matching ambiguous. The result is a characteristic "halos" or "flying pixels" artifact at depth edges — points that appear at intermediate depths that do not correspond to any real surface. These invalid depth values are not flagged as errors; they appear as plausible-looking numbers. Any downstream algorithm (grasping, obstacle avoidance) that does not explicitly check for edge-adjacent pixels in the depth map will act on phantom obstacles or surfaces.
SEE ITStereo depth uncertainty calculator
You are configuring the stereo camera for FollowBot. Set the baseline (physical separation between left and right cameras) and focal length (in pixels, determined by lens and resolution), then sweep the target distance. Watch how depth uncertainty σ_Z evolves — and find the distance at which the D435 becomes unreliable for your task.
The key insight from the widget: for FollowBot person-following at the default D435 baseline, the reliable depth range is roughly 0.3–3 m. Beyond 3 m the depth error exceeds 12 cm, which is large enough that the sensor fusion algorithm must weight depth heavily against LIDAR — and beyond 5 m, the D435 depth should be given near-zero weight. This is not a firmware bug or a calibration problem; it is the fundamental physics of triangulation.
COMPARISONThree physical principles — passive stereo, structured light, time-of-flight
The RealSense D435 is not a passive stereo camera, even though its depth algorithm uses stereo geometry. It is a structured light / active stereo hybrid: it projects an IR dot pattern onto the scene to provide artificial texture in featureless areas, then uses stereo disparity between the two IR cameras to recover depth. Understanding the three distinct physical principles explains their tradeoffs at a fundamental level.
| Principle | How it works | Range | Outdoor/sunlight | Textureless scenes | Representative sensors |
|---|---|---|---|---|---|
| Passive stereo | Block-match patches in rectified left/right images. No light source needed. | Limited by baseline (Z² law) | Works outdoors | Fails on white walls, sky, floors | ZED cameras (pure stereo mode), most research stereo rigs |
| Structured light | Project a coded IR pattern; camera sees the deformation of the pattern to infer depth | 0.2–8 m typical | Sunlight washes out IR pattern | Fills in texture-poor areas well | Kinect v1, Intel RealSense L515 |
| Active IR stereo | Project IR dot/flood; stereo match between two IR cameras (hybrid approach) | 0.3–10 m | IR projector competes with solar IR | Good on textureless surfaces | RealSense D435, D455 |
| Time-of-flight (iToF) | Emit amplitude-modulated IR; measure phase shift of reflected signal | 0.1–6 m typical | Solar IR saturates detector | Works on any surface | Microsoft Azure Kinect, OAK-D ToF, many smartphones |
| dToF / LIDAR | Emit laser pulses; measure time-of-flight of individual photons | 0.1–100+ m | High peak power overwhelms solar background | Works on most surfaces | RPLIDAR A2, Velodyne, Livox |
This is worth stating precisely because it surprises many engineers. The RealSense D435's IR projector outputs roughly 60 mW at 860 nm. Direct sunlight delivers approximately 1,000 W/m² to the ground, with roughly 30–50 mW/cm² in the 800–900 nm band — easily overwhelming the D435's IR projector in direct sun at any range. The result is not slightly noisier depth data; it is the complete absence of an IR pattern on the scene, causing the active stereo algorithm to fall back to passive stereo on whatever ambient texture exists. In practice: point the D435 at a sunlit concrete floor outdoors and you get a depth map with 90%+ invalid pixels. This is a categorical failure mode, not a parameter to tune.
Minimum and maximum range limits
Every depth sensor has hard physical range limits, not just noise-floor limits. For the D435:
- Minimum range (≈ 0.17 m): below this, disparity exceeds the image width and the right camera cannot see the target. The D435's practical minimum usable range is ~0.3 m. Getting closer returns
NaNor 0 in the depth image. - Maximum range (≈ 10 m absolute, ~5 m reliable): at long range, pixel-level disparity rounds toward integer values and the depth becomes quantized in large steps. The sensor will still produce numbers — they just cluster at discrete depths (the staircase effect).
For time-of-flight sensors, there is an additional phase wrapping limit: if the target is so far away that the returning signal has gone through more than one full phase cycle, the measured phase corresponds to a shorter apparent distance. The iToF sensor confidently reports a target at 1.5 m when the real target is at 7.5 m (for a 25 MHz modulation frequency, where the unambiguous range = c/(2×25 MHz) = 6.0 m and 7.5 mod 6.0 = 1.5 m). This is not a marginal error; it is a measurement that is wrong by a factor of 5.
FOLLOWBOTDepth sensing for FollowBot: when to trust what
FollowBot carries two depth-sensing modalities: the RPLIDAR A2 (2D scanning LIDAR at 10 Hz, reliable to 12 m, works in any lighting) and the Intel RealSense D435 (active IR stereo, 30 Hz depth at 640×480, reliable to ~5 m indoors). Understanding their respective reliable ranges is the key to building a sensor fusion strategy that does not break in the field.
The person tracking pipeline (LIDAR cluster + D435 depth + YOLO bounding box) uses a range-dependent weighting scheme. Inside 2 m, D435 depth is the primary ranging sensor (high frame rate, dense, works for arm workspace). From 2–5 m, LIDAR clusters are weighted more heavily and D435 provides confirmation. Beyond 5 m, D435 depth input is gated out entirely — only LIDAR range is fed to the person tracker. This policy is implemented by checking the target depth against thresholds before fusing, not by post-hoc outlier rejection.
Here is the ROS 2 node fragment that implements range-dependent depth gating for FollowBot's person tracker:
import rclpy from rclpy.node import Node from sensor_msgs.msg import Image import numpy as np # Stereo uncertainty constants for RealSense D435 FOCAL_PX = 639.0 # focal length in pixels (848x480 mode) BASELINE_M = 0.055 # 55 mm baseline in metres SIGMA_D_PX = 0.5 # disparity precision estimate (pixels) def sigma_z(z_m: float) -> float: """σ_Z = Z² · σ_d / (f · B) — in metres.""" return (z_m ** 2) * SIGMA_D_PX / (FOCAL_PX * BASELINE_M) # Thresholds (task: person following, acceptable σ_Z = 50 mm) MAX_RELIABLE_Z = 3.0 # metres — beyond this, gate D435 depth SIGMA_THRESH_M = 0.05 # 50 mm def depth_weight(z_m: float) -> float: """Return fusion weight [0, 1] for D435 depth at range z_m.""" sig = sigma_z(z_m) if sig >= SIGMA_THRESH_M: return 0.0 # unreliable — gate out # linear ramp: weight = 1 at z=0.3 m, 0 at σ_Z = threshold return max(0.0, 1.0 - sig / SIGMA_THRESH_M)
The depth_weight function is called for each person candidate before fusing the D435 depth with the LIDAR range estimate. When it returns 0.0, the depth measurement is dropped and the Kalman filter update uses LIDAR alone. The cutoff is not a magic number — it is derived directly from the σ_Z formula and the acceptable error budget for the task.
If FollowBot is taken outside into direct sunlight, the D435's active IR projection is overwhelmed by solar IR, as described above. The depth image becomes largely invalid. This is not a gradual degradation — it happens abruptly as the robot crosses a threshold of solar irradiance. The person-following pipeline must detect this condition (e.g., by monitoring the fraction of valid pixels in the depth image or using ambient light measurements) and switch to a LIDAR-only person tracking mode. Failing to handle this transition means the robot silently loses its depth input while continuing to attempt depth-based fusion with garbage data.
WEAKNESSWhere this bites
The most common mistake: engineers calibrate the D435 at 1 m in good lighting, verify it produces accurate depth, and then deploy it at 4 m without revisiting the error model. At 4 m the depth error is 16× larger than at 1 m (Z² dependency). Code that computes an obstacle distance from D435 depth and passes it directly to a planner's costmap without distance-dependent weighting is making systematic errors at long range. The correct approach is either to gate depth data by range (as shown above) or to propagate σ_Z into the sensor's noise covariance matrix and let the filter weight it accordingly. Ignoring range-dependent uncertainty is not a minor approximation; at 5 m it produces 80 cm errors feeding into a planner that expects 5 cm accuracy.
Depth discontinuities produce the "flying pixels" artifact: stereo matching assigns invalid intermediate depths to edge pixels. These are not flagged — they appear as plausible depth values. An arm planning to grasp an object at 0.6 m will encounter these artifacts at the object's silhouette. If the grasping planner does not explicitly mask out depth-edge regions (computable from the color image or from a depth edge filter), it will attempt to grasp at phantom depths that do not correspond to any surface. This category of failure is particularly common with thin objects (rods, cables, fingers) where the entire cross-section is near an edge in stereo images.
A wider baseline improves σ_Z (it appears in the denominator), but comes with hard constraints: a wider baseline creates a larger stereo blind zone close to the camera (anything closer than Z_min = fB / d_max cannot be seen by both cameras simultaneously). For FollowBot's arm, this matters: if the baseline is widened to 120 mm for better long-range depth, the minimum detectable range increases from 17 cm to ~35 cm — objects inside arm reach may fall in the blind zone. There is also the practical constraint of camera enclosure size and the need for rigid baseline maintenance (any flex or vibration-induced baseline change directly corrupts calibration). Baseline is a tradeoff, not a free parameter to maximize.
KEY INSIGHTHow this connects
- 5.1 · Camera Geometry: The Pinhole Model, Distortion, and Why Calibration Is Not Optional — epipolar geometry builds directly on the pinhole model and the intrinsic matrix. Stereo rectification uses calibrated intrinsics and the inter-camera extrinsic to construct the rectified image geometry. Without calibration, the disparity map is corrupted by systematic bias that no amount of post-processing can remove.
- 1.6 · Sensors: Physics, Noise Models, and Failure Modes — the sunlight failure of active IR sensors is a specific instance of the sensor saturation failure class introduced in 1.6. The σ_Z formula is the stereo camera's formal noise model — the general framework for thinking about sensor uncertainty in robotics.
- 5.3 · LIDAR Point Cloud Processing: From Scan to Structure — the complementary sensor to the D435 in FollowBot's stack. LIDAR provides range accuracy that is distance-independent and works outdoors; this lesson's fusion policy determines when the robot uses LIDAR vs. depth camera inputs for person tracking.
The quadratic uncertainty law — Z² / (fB) — is this lesson's organizing principle. It is a design constraint, a calibration target, and a fusion policy all in one. Whenever you encounter a system that uses a depth camera, ask: what is the reliable range for this application, and what sensor takes over when the depth camera's noise floor is exceeded? That question has a precise numerical answer, and this lesson gives you the tools to compute it.
GLOSSARYKey terms introduced
- Disparity
- The horizontal pixel offset between the projections of a 3D point in rectified left and right stereo images. d = x_L − x_R. Inversely proportional to depth: d = fB/Z. Smaller disparity means greater depth and more measurement uncertainty.
- Stereo rectification
- The image transformation that warps both images so that matching points for any world point lie on the same horizontal scanline. Requires calibrated intrinsics and inter-camera extrinsics. Prerequisite for stereo matching.
- Epipolar line
- The locus of image points in one camera that could correspond to a given point in the other camera. After rectification, all epipolar lines are horizontal, enabling 1D stereo matching instead of 2D search.
- Depth uncertainty (σ_Z)
- The standard deviation of the depth estimate at a given range. For stereo: σ_Z = Z² · σ_d / (f · B). Grows quadratically with distance. The fundamental limit on how accurately a stereo camera can measure depth.
- Active IR stereo
- A stereo depth sensing modality that projects an infrared pattern onto the scene to provide artificial texture, then uses stereo matching between two IR cameras to recover depth. Used in the RealSense D435. Fails in direct sunlight because solar IR overwhelms the projected pattern.
- Flying pixels / edge artifact
- Invalid depth values produced at depth discontinuities in stereo matching, where patches spanning both foreground and background are matched ambiguously. Appear as valid depth values at intermediate depths along object silhouettes. Must be explicitly filtered.
- Phase wrapping (iToF)
- An ambiguity artifact in indirect time-of-flight sensors: if a target is farther than the unambiguous range (c / (2f_mod)), the measured phase corresponds to a shorter apparent depth. Sensor confidently reports the wrong distance; the error is a multiple of the unambiguous range, not a small noise term.
- Stereo baseline (B)
- The physical separation between left and right camera optical centers. Appears in the denominator of σ_Z = Z² · σ_d / (f · B) — a wider baseline reduces depth uncertainty. Trade-off: wider baselines increase the minimum depth and require larger enclosures.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Derive the stereo depth equation Z = fB/d from first principles using similar triangles. What does each variable represent physically?›
After rectification, both cameras view a 3D point P. The left camera center is at the origin; the right camera center is at (B, 0, 0). P is at (X, Y, Z) in world coordinates. The left camera projects P at image coordinate x_L = f · X / Z (pinhole projection). The right camera sees the same point displaced by the baseline: x_R = f · (X − B) / Z. Their difference is d = x_L − x_R = f · B / Z. Rearranging: Z = fB/d. Here: Z = depth (what we want), f = focal length in pixels (camera geometry, known from calibration), B = baseline in the same length units as Z (physical camera separation, measured), d = disparity in pixels (measured by stereo matching). Every variable has a direct physical meaning — the formula is a geometric identity, not an approximation.
2Why does depth uncertainty grow as Z² rather than linearly with distance? What does this imply for sensor fusion at long range?›
Differentiate Z = fB/d with respect to d: dZ/dd = −fB/d². Substituting d = fB/Z gives dZ/dd = −Z²/(fB). By error propagation, σ_Z = |dZ/dd| · σ_d = Z² · σ_d / (fB). The Z² arises because depth is an inverse function of disparity: as Z increases, d decreases, and the same absolute uncertainty in d corresponds to an increasingly large uncertainty in Z. For sensor fusion: at long range, the Kalman filter should assign near-zero weight to depth camera measurements (their covariance is large) and high weight to LIDAR measurements (whose covariance is approximately distance-independent). Fusion that ignores distance-dependent uncertainty treats all depth measurements equally and will produce a biased fused estimate that trusts noisy long-range depth data.
3FollowBot is deployed outdoors on a sunny day and its depth-based person tracker starts failing. An engineer suggests recalibrating the RealSense. Why will recalibration not fix this problem?›
The D435 is an active IR stereo sensor: it projects an IR dot pattern at ~860 nm to provide texture for stereo matching. In direct sunlight, solar irradiance at 860 nm (~30–50 mW/cm²) overwhelms the projector's ~60 mW output. The stereo algorithm receives no projected pattern to match — not a distorted pattern, but no pattern. Recalibration corrects geometric parameters (intrinsics, extrinsics, lens distortion) but does not change the power level of the IR projector or filter out solar IR. The failure is physical, not geometric. The fix is either to use a sensor with higher-powered IR (some outdoor-rated depth sensors) or to switch to a sensor that is not saturated by sunlight — specifically, the RPLIDAR A2, whose high peak laser power overcomes solar background in the 905 nm band.
4A roboticist proposes increasing FollowBot's D435 baseline from 55 mm to 150 mm to improve depth accuracy. What concrete tradeoff must they evaluate before doing this?›
A wider baseline B reduces σ_Z by the ratio 55/150 ≈ 0.37 — about a 63% improvement in depth precision at all ranges. However: (1) Minimum range increases: Z_min = fB/d_max. For B=55 mm and d_max≈500 px, Z_min ≈ 0.17 m. For B=150 mm, Z_min ≈ 0.46 m. Objects inside 46 cm cannot be seen by both cameras simultaneously — the arm workspace (0.2–0.5 m from the body) may fall entirely in the stereo blind zone. (2) Calibration rigidity is harder to maintain: a wider baseline requires a stiffer mounting bracket; thermal expansion or vibration-induced flex will shift the extrinsic calibration more than for a compact sensor. (3) Occlusion increases: objects that occlude the left camera may not occlude the right camera (or vice versa) more often with a wide baseline, creating larger regions of missing depth. The decision must be made by analyzing the actual use case: if depth accuracy beyond 3 m matters more than close-range arm workspace, widen the baseline; if arm grasping is primary, do not.