Camera Geometry: The Pinhole Model, Distortion, and Why Calibration Is Not Optional
A camera that has never been calibrated is a measurement device that has never been zeroed — every pixel coordinate it produces is wrong by an unknown amount.
🎯 By the end you'll understand
How the pinhole camera model projects 3D world points to 2D pixel coordinates, and what each entry of the intrinsic matrix K physically means. How radial and tangential lens distortion corrupt those pixel coordinates and how they are modeled mathematically. How Zhang's planar checkerboard method recovers K and the distortion coefficients from a handful of images. Why reprojection error is the right validation metric — and critically, why a calibration with excellent reprojection error on the calibration images can still fail badly in a different lighting environment.
KEY INSIGHTEvery pixel is a direction, not a measurement
A ruler that has never been calibrated can still look like a ruler. A camera that has never been calibrated looks exactly like a calibrated one — but every bearing it reports is wrong by an unknown amount.
When FollowBot's camera reports that a person's face is at pixel (384, 210), what does that mean geometrically? Nothing at all, without calibration. The pixel coordinate encodes a ray direction from the camera optical center out into the world — but you only know which ray if you know the camera's focal length and principal point. You only know the ray's true direction in 3D if you have also removed the lens distortion that bent it. Calibration is the step that turns pixel coordinates from arbitrary integers into geometric measurements you can reason about.
Every algorithm in Stage 5 that uses a camera — YOLO person detection, stereo depth, visual odometry, SLAM loop-closure — takes pixel coordinates as input and assumes those coordinates already represent calibrated, undistorted image measurements. If they do not, the algorithm fails silently: the math is applied to the wrong geometry and produces wrong answers without raising an error. Calibration is the mandatory prerequisite for all of it, not an optional polish step.
HOW IT WORKSThe pinhole camera model
The pinhole model is a deliberate simplification: a camera with an infinitesimally small aperture that admits exactly one ray of light for each scene point. Real lenses are not pinholes — they have finite aperture, they focus by bending light through curved glass, and they introduce the distortions we will address shortly. But the pinhole model captures the fundamental geometry of central projection, and every calibration pipeline in OpenCV and ROS 2 is built on top of it.
The geometry gives us the pinhole projection equations. If a point in camera coordinates is P = (X, Y, Z), then its projected coordinates on the image plane before converting to pixels are:
x' = X / Z y' = Y / Z
These are normalized image coordinates — the point's angular position scaled to unit depth. They have a clean physical meaning: x' and y' are the tangents of the horizontal and vertical angle from the optical axis. Every distortion model is applied here, to x' and y', before the pixel conversion.
The ratio X/Z makes intuitive sense: an object twice as far away (Z doubled) maps to the same x' only if X is also doubled — distant objects appear smaller. This is the projective nature of camera geometry that distinguishes it from orthographic projection. It is also why a camera alone cannot measure depth: many different 3D points project to the same 2D pixel, all lying on the same ray from C.
CORE CONCEPTThe intrinsic matrix K
Normalized image coordinates (x', y') must be converted to pixel coordinates (u, v) measured in the sensor's pixel grid. This requires knowing three things: the sensor's horizontal and vertical pixel densities (which encode the focal length in pixel units), and where the optical axis intersects the sensor (the principal point). These four numbers — fₓ, fᵧ, cₓ, cᵧ — are the camera intrinsics, and they form the 3×3 upper-triangular matrix K:
K = [ fₓ 0 cₓ ]
[ 0 fᵧ cᵧ ]
[ 0 0 1 ]
The pixel coordinate is: [u, v, 1]ᵀ = K · [x', y', 1]ᵀ, which expands to u = fₓ·x' + cₓ and v = fᵧ·y' + cᵧ.
Let's be concrete about what each entry means physically:
| Symbol | Name | Physical meaning | Typical value (1080p) |
|---|---|---|---|
| fₓ | Horizontal focal length | Lens focal length × horizontal pixels per meter of sensor. Scales how much horizontal displacement in the scene shifts a pixel column. | 900–1800 px |
| fᵧ | Vertical focal length | Same as fₓ in the vertical direction. Equal to fₓ for square pixels (almost always). | ≈ fₓ |
| cₓ | Principal point x | Column of the pixel where the optical axis intersects the sensor. Often close to width/2 but rarely exact. | W/2 ± 20 px |
| cᵧ | Principal point y | Row of the pixel where the optical axis intersects the sensor. Often close to height/2 but rarely exact. | H/2 ± 20 px |
Notice that K has no off-diagonal terms except the skew slot (row 0, col 1), which is zero for virtually all modern sensors — pixel rows and columns are perfectly orthogonal. This leaves exactly four parameters to calibrate. If you assumed K = diag(f, f, 1) with principal point at the image center, you would be off by tens of pixels in most real cameras, corrupting any geometric computation downstream.
In ROS 2, K is published in the sensor_msgs/CameraInfo message as a flat 9-element array — the standard contract between the camera driver and every algorithm that needs it:
import rclpy from rclpy.node import Node from sensor_msgs.msg import CameraInfo import numpy as np class IntrinsicsReader(Node): def __init__(self): super().__init__('intrinsics_reader') self.create_subscription( CameraInfo, '/camera/color/camera_info', self.cb, 1 ) def cb(self, msg): # K is a row-major 3x3 flat array in CameraInfo K = np.array(msg.k).reshape(3, 3) fx, fy = K[0, 0], K[1, 1] cx, cy = K[0, 2], K[1, 2] D = list(msg.d) # distortion coefficients [k1,k2,p1,p2,k3] self.get_logger().info( f'fx={fx:.1f} fy={fy:.1f} cx={cx:.1f} cy={cy:.1f} D={D}' )
One important subtlety: msg.d is the distortion coefficient vector. Its length and meaning depend on the distortion model in use. The most common for narrow-FOV cameras is the 5-parameter "plumb bob" model (k1, k2, p1, p2, k3). For wide-angle or fisheye lenses, OpenCV's fisheye model with 4 parameters is used instead, and mixing the two produces silently wrong undistortion.
CORE CONCEPTLens distortion: radial and tangential
Real lenses bend light through curved glass. The bending is not perfectly uniform — it varies with the distance from the optical axis. This produces radial distortion, the dominant term. The lens mounting is also not perfectly centered and perpendicular — this produces tangential distortion, the secondary term. Both act on the normalized image coordinates (x', y') before the pixel conversion.
The mathematical model for the "plumb bob" distortion (the OpenCV default) is applied to normalized image coordinates (x', y'). Let r² = x'² + y'².
Radial correction: x'' = x'·(1 + k1·r² + k2·r⁴ + k3·r⁶)
Tangential correction applied simultaneously:
x'' += 2·p1·x'·y' + p2·(r² + 2·x'²)
y'' += p1·(r² + 2·y'²) + 2·p2·x'·y'
Then pixel coordinate: u = fₓ·x'' + cₓ, v = fᵧ·y'' + cᵧ
Five distortion coefficients: [k1, k2, p1, p2, k3]. In practice, k3 is often close to zero and dropped (yielding 4-coefficient calibrations), but OpenCV uses 5 by default. For most cameras, |k1| dominates and the tangential terms (p1, p2) are small but non-negligible — especially for cheap lenses where the sensor is slightly tilted relative to the optical axis.
What makes distortion insidious is not its magnitude but its spatial non-uniformity. A 5-pixel error at the image center is bad; a 20-pixel error at the corners is worse — and the error varies continuously across the frame. Any algorithm that assumes a linear relationship between pixel coordinates and scene geometry (such as a homography-based pose estimator) breaks catastrophically in the corners of an uncalibrated image.
There are two ways to handle distortion in practice. You can undistort the entire image — remap every pixel to its corrected position — then work with calibrated images downstream. Or you can undistort individual points as needed, applying the inverse of the distortion model. Undistorting the whole image is convenient but introduces a black border at the frame edges and requires resampling (introducing some blur). For real-time ROS 2 pipelines, undistorting only the feature points you actually use (sparse undistortion) is faster and avoids the resampling artifact. OpenCV's cv2.undistortPoints() does this directly.
IN PRACTICEZhang's method: checkerboard calibration
How do you find K and the distortion coefficients without access to a laser distance meter and a precisely manufactured calibration rig? Zhengyou Zhang's 2000 method answers this elegantly: use a flat printed checkerboard, photograph it from many orientations, and solve for the intrinsics that are consistent across all views. The method requires a minimum of 3 views (but 15–25 is standard for sub-pixel accuracy) and needs no metric knowledge of the checkerboard — only the square size in consistent units.
Detect corners
For each image, detect the inner corners of the checkerboard grid to sub-pixel accuracy using cv2.findChessboardCorners() followed by cv2.cornerSubPix(). A 9×6 inner corner grid on a 10×7 square board is standard. Each detected corner gives you a 2D pixel coordinate paired with a known 3D coordinate in the board's local frame.
Estimate homographies
For each view, solve the 2D–2D correspondence between board coordinates and pixel coordinates for the homography H. This 3×3 matrix encodes how the flat board maps to the image. Each H constrains K; with at least 3 views, K is fully determined.
Closed-form initialization
Stack the constraints from all H matrices into a linear system and solve for a closed-form initial estimate of K. This is Zhang's key insight: the intrinsics appear in a structured bilinear form across the homographies, making them recoverable by eigendecomposition of a 6×6 matrix — no nonlinear optimization needed at this stage.
Nonlinear refinement
Jointly refine K, the distortion coefficients [k1, k2, p1, p2, k3], and the per-image extrinsic poses using Levenberg-Marquardt optimization. The objective is to minimize the sum of squared reprojection errors across all calibration images. This is where accuracy is won or lost — the closed-form solution is a starting point, not the answer.
Validate with reprojection error
For each calibration image, reproject the 3D board corners using the calibrated K and distortion coefficients, and compute the RMS pixel distance between reprojected corners and detected corners. A reprojection error below 0.5 pixels indicates a good calibration for a typical 1080p camera. Above 1 pixel warrants re-calibration. If the error is very low on some images and high on others, those high-error images likely had motion blur or partial occlusion — remove them and recalibrate.
In ROS 2, the standard tool is the camera_calibration package from image_pipeline. It provides a GUI that guides you through collecting sufficient checkerboard coverage (the X, Y, size, and skew progress bars must all fill before it recommends calibrating), then writes the result to a YAML file that the camera driver reads and republishes as CameraInfo:
# Install (once) sudo apt install ros-humble-camera-calibration # Run calibrator — 8x6 inner corners, 25mm squares ros2 run camera_calibration cameracalibrator \ --size 8x6 \ --square 0.025 \ image:=/camera/color/image_raw \ camera:=/camera/color # After calibrating, the YAML is saved to: # ~/.ros/camera_info/<camera_name>.yaml # Load it in your camera launch file via camera_info_url parameter
Coverage matters as much as quantity. Calibration images where the board only occupies the center of the frame poorly constrain the principal point and are almost useless for distortion estimation (the distortion model is most observable at the edges). The camera_calibration GUI's progress bars directly encode which coverage regions are still missing.
SEE ITPinhole projection explorer
This widget lets you see exactly how each intrinsic parameter and distortion coefficient affects where a 3D point lands in the image — and shows you the reprojection error that results when you perturb the intrinsics away from the ground-truth values. The worst choice you can make: assume the principal point is at the image center and set distortion to zero. The widget will show you how wrong that is.
The reprojection error display is exactly what cv2.calibrateCamera() reports at the end of calibration — the RMS pixel distance between where calibration points were detected and where the calibrated model says they should be. Notice that perturbing the principal point by even 20 pixels pushes several calibration points' errors above 2 pixels — well above the 0.5 px threshold for a good calibration. This is why assuming the principal point is at the image center is not a reasonable approximation for any quantitative geometry.
FOLLOWBOTCalibrating FollowBot's RealSense D435
FollowBot's Intel RealSense D435 ships with a factory calibration stored on the device. For many applications this is sufficient. For person following with pixel-accurate detections fused with LIDAR geometry — which is exactly what we need for Stage 5's milestone — it is not. The factory calibration is performed under controlled lighting conditions; your deployment environment may differ significantly. You should calibrate the specific camera you deploy, not rely on a factory default.
For FollowBot's Stage 5 milestone, the camera calibration must achieve reprojection error < 0.5 px on a validation set of checkerboard images not used during calibration. This is the gate that unlocks the LIDAR-camera extrinsic calibration in Lesson 5.3 — you cannot meaningfully fuse two sensors whose individual intrinsic calibrations are uncertain. A calibration that achieves 0.3 px reprojection error on calibration images but 1.2 px on held-out images is a calibration that has overfit to its calibration set, not generalized.
The D435 also publishes a depth_to_color extrinsic transform via its SDK, describing the rigid offset between the RGB sensor and the depth sensor (they are physically separate imagers on the same housing). This transform is used in Lesson 5.2 when we align depth frames to color frames. For now, focus on the RGB camera intrinsics — those are what YOLO detections and camera-geometry algorithms consume.
Print and mount the checkerboard
Print a 9×6 inner corner checkerboard on a rigid flat surface — mounting on foam board or aluminum plate prevents warping. Warped calibration targets introduce systematic error. Square size: 25 mm is standard for a D435 at 0.5–2 m distance. Verify the printed size with a caliper before calibrating.
Collect calibration images
Capture 20–30 images with the checkerboard covering the full FOV: corners, edges, tilted ±45° in both axes, close and far. Collect under consistent ambient lighting. Avoid motion blur — use a tripod or hold the board still. Reject images where fewer than 90% of corners are detected with the sub-pixel accuracy check passing.
Run calibration and validate
Run camera_calibration, check that RMS reprojection error is below 0.5 px. Reserve 5 images as a held-out validation set: reprojection error on validation images should not exceed 2× the calibration error. If it does, the calibration has not generalized — collect more coverage images, especially at oblique angles and edges.
Deploy the calibration file
Copy the YAML to your workspace's config/ directory and load it via the camera_info_url parameter in your camera launch file. Confirm that /camera/color/camera_info is publishing the calibrated K and D coefficients — not zeros (the default when no calibration is loaded).
After loading the calibration, draw horizontal lines on a raw image captured of a real-world scene with straight edges (a door frame, a shelf). Apply cv2.undistort() and check that straight edges in the scene are straight in the undistorted image. Any remaining barrel or pincushion effect in a well-defined straight edge indicates residual calibration error. This test takes 30 seconds and catches gross calibration failures before they contaminate your detection pipeline.
WEAKNESSWhere calibration bites back
This is the most consequential calibration failure mode. Zhang's method minimizes reprojection error on the calibration images. If those images all share the same lighting condition (e.g., all captured under fluorescent lab lights), the calibration captures both the true intrinsics and any systematic detection bias that correlates with that lighting (slight halo effects in corner detection, for instance). In a darker deployment environment, corner detection runs differently, the systematic bias changes, and the effective reprojection error in deployment is significantly higher than it was in calibration. The mitigation is straightforward but often skipped: collect calibration images under representative deployment lighting conditions, and always evaluate on a held-out validation set.
Camera calibration encodes the physical relationship between the lens and the sensor. If the camera is dropped, bumped hard, or the lens is refocused (zoom cameras), the intrinsics change. A calibration from before the bump is silently wrong. In a mobile robot that operates in the real world and is occasionally bumped during field deployments, this is not a theoretical concern. Production robotics teams build calibration verification steps into startup routines — typically, capturing a reference checkerboard image and checking that reprojection error stays below threshold before the robot is cleared to operate.
The radial + tangential model (k1, k2, p1, p2, k3) is only valid for cameras with FOV below roughly 150°. Above that, the polynomial radial model diverges — the distortion grows faster than the polynomial can track. Wide-angle and fisheye lenses require OpenCV's fisheye calibration model (cv2.fisheye.calibrate()) with its own 4-coefficient model. If you apply the standard plumb bob calibration to a fisheye lens, the corner reprojection errors will appear acceptable in the center of the frame and catastrophic at the edges — and the undistorted image will look severely wrong. FollowBot's D435 (horizontal FOV ≈ 69°) is safely within the plumb bob model's range, but be aware of this when selecting lenses for other sensors.
KEY INSIGHTHow it connects
- 1.6 · Sensors: Physics, Noise Models, and Failure Modes — camera intrinsics are not the only source of pixel error. Sensor noise, rolling-shutter distortion, and motion blur are additive errors on top of calibration residuals. That lesson's noise model framework applies to camera pixels exactly as it applies to LIDAR ranges.
- 1.2 · Coordinate Frames and Homogeneous Transforms — the intrinsic matrix K is a projection from camera 3D coordinates to image 2D pixel coordinates. It is a non-square projection matrix, not a rigid-body transform, but the homogeneous coordinates machinery from that lesson is what makes it a clean matrix multiplication. The camera-to-world extrinsic transform (used in Lesson 5.3) is a standard rigid-body transform in SE(3).
- 5.2 · Stereo Vision and RGB-D Depth Sensing — the next lesson. The D435's depth image is generated by its infrared stereo pair, with the same pinhole model applied to each camera. The disparity-to-depth conversion requires the stereo baseline and the calibrated focal length — both of which are precisely what we built in this lesson.
Camera geometry is the geometric foundation for everything that follows in Stage 5. The person detection bounding boxes (Lesson 5.4) are pixel-coordinate outputs from YOLO — they only become useful 3D measurements after being unprojected through K. The SLAM loop closures (Lesson 5.7) compare image features using geometric consistency checks that assume calibrated cameras. Get the calibration right once, at the beginning of Stage 5, and every subsequent algorithm inherits a clean geometric foundation.
GLOSSARYKey terms introduced
- Pinhole camera model
- A geometric model of central projection in which a 3D world point is mapped to a 2D image plane by a straight ray through the optical center. Ignores lens effects; describes the projective geometry that the intrinsic matrix K encodes.
- Intrinsic matrix K
- The 3×3 upper-triangular matrix encoding the camera's internal geometry: focal lengths fₓ and fᵧ in pixels, and principal point (cₓ, cᵧ). Maps normalized image coordinates to pixel coordinates. Camera-specific; does not change with the camera's position or orientation in the world.
- Principal point
- The pixel coordinate (cₓ, cᵧ) where the optical axis intersects the image sensor. Often close to but not exactly at the sensor center. Calibration is required to find it; assuming it is at the image center introduces systematic pixel errors.
- Radial distortion
- Lens distortion that pushes pixels outward (barrel, k1 < 0) or inward (pincushion, k1 > 0) from the optical axis, increasing nonlinearly with distance r from the principal point. The dominant distortion term for most lenses.
- Tangential distortion
- Lens distortion arising from the sensor plane not being perfectly perpendicular to the optical axis. Described by coefficients p1 and p2; typically smaller in magnitude than radial distortion but non-negligible for precise geometric computation.
- Reprojection error
- The RMS pixel distance between detected feature points (e.g., checkerboard corners) in calibration images and the positions those points would project to under the calibrated camera model. The primary validation metric for intrinsic calibration; values below 0.5 px are considered good for 1080p cameras.
- Zhang's calibration method
- A planar calibration technique by Zhengyou Zhang (2000) that recovers camera intrinsics and distortion coefficients from multiple images of a flat checkerboard pattern. The dominant method in practice; implemented in OpenCV's
cv2.calibrateCamera()and the ROS 2 camera_calibration package. - Normalized image coordinates
- The 2D coordinates (x', y') = (X/Z, Y/Z) of a projected point before the pixel-scale conversion by K. Distortion models are applied in normalized coordinates, making them independent of the specific sensor resolution.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1What does it mean physically for the principal point (cₓ, cᵧ) to be offset from the image center, and why does assuming it is at the center cause problems?›
The principal point is where the optical axis of the lens intersects the physical sensor. Due to manufacturing tolerances — the lens mount not being perfectly centered, the sensor not being perfectly aligned within the housing — this intersection rarely falls exactly at the sensor's geometric center. Assuming cₓ = W/2, cᵧ = H/2 introduces a systematic pixel offset in all projected coordinates. For a camera where the true principal point is 15 pixels from the assumed center, every pixel-to-ray conversion is wrong by a constant angular offset — which means any geometric algorithm (pose estimation, stereo triangulation, camera-LIDAR fusion) will produce a consistent but wrong answer. The error does not average out over many measurements; it biases every single one. This is why calibration is non-negotiable even when the principal point offset looks "small."
2You run Zhang's calibration on 25 images captured in your lab and get RMS reprojection error 0.28 px. You then take the calibrated camera to a dimly lit corridor and find that corner detection runs poorly and your fusion pipeline produces wrong 3D positions. What went wrong and how do you fix it?›
The calibration overfit to the lighting conditions of the lab. Zhang's method minimizes reprojection error on the calibration images; if those images all share the same lighting, the calibration absorbs any systematic bias in corner detection that is specific to that lighting. In the dim corridor, the corner detection algorithm runs differently — different sub-pixel interpolation convergence, different edge gradient strengths — and the systematic detection bias that the calibration implicitly compensated for is gone. The reprojection error in deployment is significantly higher than 0.28 px even though the calibration looks excellent. Fix: recollect calibration images under representative deployment lighting (or at multiple lighting levels), and always evaluate reprojection error on a held-out validation set captured under deployment conditions — not just the training set.
3Explain the difference between undistorting the full image vs. undistorting only the feature points you use, and when you would prefer each approach.›
Full image undistortion (cv2.undistort()) remaps every pixel to its corrected position via a precomputed remap table, producing a geometrically clean image that any algorithm can use without knowing about distortion. The downsides: it requires resampling (introducing a small amount of interpolation blur), leaves black borders at frame edges (valid image area is slightly cropped), and consumes compute proportional to the full image resolution. Sparse undistortion (cv2.undistortPoints()) applies the inverse distortion model to individual 2D points only — zero image resampling, no border, and compute proportional to the number of points. Prefer full undistortion when downstream algorithms assume globally undistorted images (e.g., homography estimation, optical flow over the full frame). Prefer sparse undistortion when you only use a small set of feature points (e.g., YOLO bounding box centers, tracked keypoints for visual odometry) — it is faster and avoids the resampling artifact.
4Why can a camera alone not measure the depth of a scene point, and what does the pinhole model reveal about this fundamental limitation?›
The pinhole projection maps a 3D point P = (X, Y, Z) to 2D pixel coordinates u = fₓ·X/Z + cₓ, v = fᵧ·Y/Z + cᵧ. The division by Z is irreversible from a single image: if you observe pixel (u, v), you know the ray direction (the ratio X/Z and Y/Z), but you cannot recover X, Y, or Z individually — any point on the ray through the optical center maps to the same pixel. Infinitely many 3D points (a, b, λ) for any scalar λ > 0 project to the same (u, v). This is the fundamental ambiguity of monocular vision. The pinhole model makes it explicit: depth information is discarded by the projection, not corrupted — it was never in the image to begin with. Recovering depth requires either a second camera (stereo), an active depth sensor (structured light or time-of-flight, as in the D435), or motion of the camera with corresponding point tracking (structure from motion).