LIDAR Point Cloud Processing: From Scan to Structure
A raw LIDAR scan is not a map — voxelization, outlier removal, and ground segmentation must run before anything downstream works reliably.
🎯 By the end you'll understand
How a raw LIDAR scan becomes a Cartesian point cloud, and why the coordinate conversion is non-trivial when the robot is moving. Why motion distortion is a silent corruption source, not a rare edge case. How voxel downsampling controls the density-versus-detail tradeoff — and why picking the leaf size naively breaks your person detector. How statistical outlier removal distinguishes real geometry from sensor noise. Why RANSAC plane fitting is the right tool for ground removal. Why ground segmentation must happen before clustering — and what breaks catastrophically when you skip it. The full pre-processing pipeline from raw scan to labeled clusters, implemented in ROS 2 with rclpy and Open3D.
KEY INSIGHTA raw scan is geometry soup
Feeding a raw LIDAR scan directly to a person detector is like feeding a restaurant bill to a nutritionist. The data is real, but the representation is wrong for what you want to ask of it.
A LIDAR scan returns N range values, each paired with an angle. That is it. No semantics, no structure, no indication of which returns are floor and which are furniture and which are the person you want to follow. Converting that geometry soup into something useful — a set of labeled, size-appropriate clusters — requires a pipeline of four operations: downsampling, outlier removal, ground segmentation, and Euclidean clustering. Each step has a parameter with a tradeoff. Each step has a failure mode that poisons everything downstream. This lesson is about those tradeoffs and failures, not just the nominal operation.
The operations must run in the right order, and the order is non-obvious. You might expect "find clusters, then remove the ground cluster." That is exactly wrong. The floor is the largest connected region in an unprocessed scan. Euclidean clustering will always find it first — and the resulting "floor cluster" will be enormous, consuming the person you were looking for. Ground segmentation must precede clustering, not follow it. This is the single most common pipeline bug in newcomer LIDAR code.
HOW IT WORKSFrom scan to point cloud: the coordinate conversion
A 2D LIDAR such as the RPLIDAR A2 returns 360 (or more) range-angle pairs, published on ROS 2's sensor_msgs/LaserScan topic. A 3D LIDAR such as the Velodyne VLP-16 or Ouster OS1 returns a structured array of range-intensity values for multiple vertical beams, published as sensor_msgs/PointCloud2. The coordinate conversion is the same in both cases — just repeated for each beam.
In ROS 2, the RPLIDAR driver publishes LaserScan messages. The laser_geometry package converts these to PointCloud2 in a single call. For 3D sensors, the driver publishes PointCloud2 directly. Either way, the first pre-processing step is validity filtering: drop inf, nan, and returns outside the sensor's reliable range bracket (e.g., below 0.15 m where the return window is not clean, or above 12 m where noise dominates for an A2).
import rclpy from rclpy.node import Node from sensor_msgs.msg import LaserScan, PointCloud2 from laser_geometry import LaserProjection import tf2_ros, tf2_sensor_msgs # for frame transform class ScanToCloud(Node): def __init__(self): super().__init__('scan_to_cloud') self.proj = LaserProjection() self.tf_buf = tf2_ros.Buffer() self.tf_lis = tf2_ros.TransformListener(self.tf_buf, self) self.create_subscription(LaserScan, '/scan', self.cb, 10) self.pub = self.create_publisher(PointCloud2, '/cloud_raw', 5) def cb(self, msg): # project into laser frame (no transform yet) cloud = self.proj.projectLaser( msg, range_cutoff=12.0, # drop > 12 m channel_options=0x07 # x, y, intensity ) self.pub.publish(cloud)
CORE CONCEPTMotion distortion: the silent corruption
Here is a subtlety that textbooks tend to bury: a spinning LIDAR does not capture all N points simultaneously. The RPLIDAR A2 spins at 10 Hz — one full rotation every 100 ms. The first return is captured at time t₀; the return 180° away is captured at t₀ + 50 ms; the last return closes at t₀ + 100 ms. If the robot is moving during that 100 ms window, every return is captured from a slightly different robot pose. Treating the scan as if all points were captured simultaneously produces a distorted point cloud.
At 0.6 m/s forward velocity, the robot moves 6 cm during one scan revolution. That 6 cm displacement maps directly into point cloud distortion — more than enough to degrade SLAM scan matching and person detection. The fix is motion compensation: use the robot's odometry or IMU to interpolate the pose at each beam's timestamp and de-rotate each point into a common reference frame (typically the pose at scan end). In ROS 2, the laser_geometry package accepts a TF2 buffer and performs this compensation automatically when the /odom → /base_link transform is available and publishing at sufficient rate.
For a slow following robot (<0.3 m/s) on a stationary target, motion distortion is often negligible. For a robot turning at 60°/s (entirely possible during sharp person-following maneuvers), distortion becomes significant even in a 2D scan. The practical test: look at SLAM scan overlaps in RViz during aggressive turns. If scan edges misalign while walls are clearly linear, motion distortion is the culprit, not scan matching failure.
CORE CONCEPTVoxel downsampling: controlled thinning
A raw 3D LIDAR point cloud from a VLP-16 contains roughly 300,000 points per second (16 beams × ~18,000 points/beam/second). Processing that density at every pipeline stage is computationally wasteful, and in many regions (a flat wall) the density adds no information. Voxel downsampling divides space into a regular 3D grid of cubes (voxels), replaces all points inside each voxel with their centroid, and discards the rest.
The parameter is the leaf size L (meters) — the voxel side length. Choosing L is a consequential tradeoff:
| Leaf size L | Points retained | Good for | Risks |
|---|---|---|---|
| 0.02 m (2 cm) | ~85% of raw | Fine-grained object modeling, arm manipulation | High compute cost; outliers survive; person occupies 1,000+ points |
| 0.05 m (5 cm) | ~40–50% | Person detection, SLAM with features | Thin wires/cables may lose intermediate points |
| 0.10 m (10 cm) | ~15–25% | Occupancy mapping, large-environment SLAM | Small objects (fire hydrant, chair leg) may collapse to 1–2 points |
| 0.20 m (20 cm) | ~5–10% | Very fast mapping of large open spaces | Person may collapse to 3–5 points; person detection breaks |
For FollowBot's person-detection task, 0.05 m is a reasonable default: it reduces compute by ~55% while keeping a person-shaped cluster (~30–60 points for a 1.7 m person at 2 m range) that is still recognizable. Going to 0.20 m collapses the person to 4–6 points — below the minimum cluster size for reliable detection.
import open3d as o3d import numpy as np def voxel_downsample(points_xyz: np.ndarray, leaf: float) -> np.ndarray: """points_xyz: (N,3) float32; leaf: voxel side in metres.""" pcd = o3d.geometry.PointCloud() pcd.points = o3d.utility.Vector3dVector(points_xyz) pcd = pcd.voxel_down_sample(voxel_size=leaf) return np.asarray(pcd.points)
CORE CONCEPTStatistical outlier removal
After downsampling, the cloud still contains noise: spurious returns from dust, glass reflections, laser multi-path, and the LIDAR's own receiver noise. These appear as isolated points far from any real surface. Statistical outlier removal (SOR) removes them using a neighbor-density criterion.
For each point p, SOR computes the mean distance μ to its k nearest neighbors (the nb_neighbors parameter). It then computes the mean μ̄ and standard deviation σ of this value across all points. Any point whose mean neighbor distance exceeds μ̄ + std_ratio·σ is labeled an outlier and removed.
nb_neighbors (k): how many neighbors to consult. Larger k = smoother distance estimate, but more expensive. Typical range: 10–50. For a sparse cloud (post-voxel), k=20 is reasonable. std_ratio: how many standard deviations above the mean triggers removal. Lower ratio = more aggressive removal (also removes real edge-of-surface points). Higher ratio = gentler (allows noisy outliers to survive). Typical range: 1.0–3.0; 2.0 is a common default. The failure mode: setting std_ratio too low removes legitimate sparse points — the trailing edge of a thin person cluster, for example, disappears because it has fewer close neighbors than a dense wall region.
def remove_outliers(points_xyz: np.ndarray, nb_neighbors: int = 20, std_ratio: float = 2.0) -> np.ndarray: pcd = o3d.geometry.PointCloud() pcd.points = o3d.utility.Vector3dVector(points_xyz) cl, ind = pcd.remove_statistical_outlier( nb_neighbors=nb_neighbors, std_ratio=std_ratio ) return np.asarray(cl.points)
CORE CONCEPTRANSAC ground removal: finding the dominant plane
After filtering, the largest coherent structure in an indoor point cloud is usually the floor. We want to remove it before clustering — leaving only above-floor objects. The standard approach is RANSAC plane fitting.
RANSAC (Random Sample Consensus) for plane fitting works as follows:
Sample 3 points at random
Three non-collinear points define a unique plane in 3D. Pick 3 points uniformly at random from the cloud. This is the candidate plane hypothesis.
Count inliers
For every other point in the cloud, compute its perpendicular distance to the candidate plane. If the distance is below the distance threshold d (a parameter, typically 0.02–0.10 m), the point is an inlier. Count the inliers.
Iterate and keep the best
Repeat steps 1–2 for N iterations (typically 100–500). The hypothesis with the most inliers wins. Refit the plane to all winning inliers using least squares to get the final plane parameters [a, b, c, d] for ax + by + cz + d = 0.
Separate inliers from outliers
Points within distance threshold d of the best-fit plane are labeled ground. All others are "non-ground" — objects. Pass only the non-ground set to the clustering step.
The critical parameter is the distance threshold d. Too small (0.01 m) and floor roughness causes legitimate floor returns to be classified as objects. Too large (0.20 m) and low objects (coffee table, dog, small child) get absorbed into the ground set and disappear. For FollowBot on typical indoor floors, 0.03–0.06 m works well. On uneven outdoor terrain, you may need to switch to progressive morphological filtering or height-based segmentation instead of global RANSAC.
def remove_ground(points_xyz: np.ndarray, distance_threshold: float = 0.05, ransac_n: int = 3, num_iterations: int = 200): """Returns (above_ground, ground_pts) both as (M,3) arrays.""" pcd = o3d.geometry.PointCloud() pcd.points = o3d.utility.Vector3dVector(points_xyz) plane_model, inliers = pcd.segment_plane( distance_threshold=distance_threshold, ransac_n=ransac_n, num_iterations=num_iterations ) ground = pcd.select_by_index(inliers) objects = pcd.select_by_index(inliers, invert=True) return np.asarray(objects.points), np.asarray(ground.points)
CORE CONCEPTEuclidean clustering: separating objects
After ground removal, the non-ground cloud contains separated spatial clusters corresponding to real objects: walls, furniture, people. Euclidean clustering groups points that are within a spatial distance tolerance of each other.
The algorithm is essentially DBSCAN with ε as the cluster tolerance and a minimum cluster size constraint. Two points belong to the same cluster if they are within distance ε. The parameters:
| Parameter | Typical value | Effect of too small | Effect of too large |
|---|---|---|---|
| cluster_tolerance (ε) | 0.04–0.10 m | One person split into torso + legs clusters | Adjacent persons merged into one cluster |
| min_cluster_size | 30–100 pts | Noise spikes reported as objects | Small objects (bottles, pets) missed |
| max_cluster_size | 1,000–5,000 pts | Wall fragments or large furniture truncated | N/A (upper bound only) |
For person detection, a good starting point is ε = 0.08 m, min_size = 50 points, max_size = 2,000 points. A person at 2 m range in a 0.05 m downsampled cloud yields ~40–80 points — just above the min_size floor. Getting closer than 1.5 m increases density fast; getting farther than 4 m drops it below the min_size threshold — a legitimate failure mode for long-range person detection.
If you run Euclidean clustering before ground removal, the floor is always the largest cluster — consuming thousands of points. Your min/max constraints will reject it only if the floor cluster exceeds max_cluster_size, which is unlikely to be set that low. The surviving floor-containing cluster will confound everything: the bounding box will span the room, the centroid will be at floor level, and your person detector will hallucinate a "person" the size of a classroom. Always: voxel → outlier remove → ground remove → cluster.
SEE ITPoint cloud processing pipeline
Below is a synthetic 3D scene: a ground plane plus a "person" cluster (a vertical cylinder of points) and some scattered noise. Step through the pipeline and adjust parameters. The critical question: what happens to the person cluster if you skip ground removal and cluster directly?
The widget above makes the ordering constraint visceral. When ground removal is skipped, the clustering step finds a massive ground cluster that swallows the person. The cluster count increases but the meaningful cluster — the person — is not isolated. This is why production LIDAR pipelines always include ground segmentation as a mandatory stage, not an optional optimization.
FOLLOWBOTIntegrating the pipeline for person following
FollowBot's milestone for Stage 5 includes producing a stable 3D person position at 10 Hz from LIDAR cluster and camera bounding box fusion. The point cloud processing pipeline described here is the LIDAR half of that fusion — it runs on every PointCloud2 message from the sensor, publishing cluster centroids and bounding boxes that the fusion node consumes.
The processing chain maps cleanly to ROS 2 nodes: scan_to_cloud subscribes to /scan and publishes /cloud_raw. cloud_processor subscribes to /cloud_raw, runs voxel → SOR → ground removal → clustering, and publishes /clusters (as a custom message or as a visualization_msgs/MarkerArray for RViz). person_tracker subscribes to /clusters plus /detections from YOLO and fuses them into a tracked 3D person position.
import rclpy, numpy as np, open3d as o3d from rclpy.node import Node from sensor_msgs.msg import PointCloud2 from sensor_msgs_py import point_cloud2 as pc2 from visualization_msgs.msg import MarkerArray, Marker LEAF = 0.05 # voxel leaf size (m) K_SOR = 20 # SOR k-neighbours STD_R = 2.0 # SOR std ratio DTHR = 0.05 # RANSAC distance threshold (m) EPS = 0.08 # cluster tolerance (m) MINPTS = 50 # min cluster size MAXPTS = 2000 # max cluster size class CloudProcessor(Node): def __init__(self): super().__init__('cloud_processor') self.create_subscription(PointCloud2, '/cloud_raw', self.cb, 5) self.pub = self.create_publisher(MarkerArray, '/clusters', 5) def cb(self, msg): pts = np.array( list(pc2.read_points(msg, field_names=('x','y','z'), skip_nans=True)), dtype=np.float32 )[:, :3] pts = voxel_downsample(pts, LEAF) # step 1 pts = remove_outliers(pts, K_SOR, STD_R) # step 2 obj, gnd = remove_ground(pts, DTHR) # step 3 — MUST precede clustering clusters = euclidean_cluster( obj, eps=EPS, min_pts=MINPTS, max_pts=MAXPTS ) # step 4 self.pub.publish(clusters_to_markers(clusters, msg.header))
A few implementation notes that bite people in practice. First, Open3D's cluster_dbscan is not real-time at full cloud density — run it after voxel downsampling. Second, the frame of the published clusters matters: they should be in /base_link or /map, not in /laser, so the fusion node can compare them to camera detections expressed in /base_link. Third, the RANSAC ground model should not be re-fitted from scratch every scan — the ground plane is stable, so fitting once every 2–3 seconds and caching the model is both faster and more robust.
After clustering, filter clusters by bounding box dimensions. A standing adult at 1–5 m range produces a cluster roughly 0.4–0.8 m wide, 0.3–0.6 m deep, and 0.8–1.8 m tall. Reject clusters outside [0.2–1.2 m] width and [0.5–2.0 m] height as non-person. This naive size filter alone eliminates chairs, tables, walls, and ceiling artifacts before any YOLO fusion — reducing the fusion node's workload by 80–90% in a typical indoor scene.
Stage 5 milestone status after this lesson: the LIDAR pre-processing pipeline is running in ROS 2, producing stable clusters at 10 Hz. The next step (Lesson 5.4 introduces visual odometry; Lesson 5.5 handles object detection) brings in the YOLO bounding boxes that fuse with these LIDAR clusters to produce the final 3D person position at the milestone's required 10 Hz rate.
WEAKNESSWhere this bites
This cannot be stated firmly enough: if you run Euclidean clustering on the full non-outlier-removed cloud — floor included — the floor is the dominant cluster in every scan. A 10 m × 8 m room floor at 0.05 m voxel resolution produces ~32,000 ground-level points. The clustering algorithm groups them (correctly, given the distance tolerance) into a single enormous cluster. Your person-cluster size filter rejects it as too large; but the floor takes ~80% of the CPU budget and the person cluster, which might have attached floor points around the person's feet, is now merged into a "person+floor" supercluster whose centroid is at ankle height. The fix is trivially cheap (one RANSAC call) but the failure is invisible unless you visualize the clusters in RViz before trusting them.
Engineers often increase the leaf size to reduce latency when the compute budget is tight. At 0.15 m leaf size, a person at 3 m range collapses to 8–12 points — below any sensible min_cluster_size. The person literally disappears from the output. The symptom is intermittent person detection: the tracker sees the person at close range (where density is still adequate) and loses them as they back up beyond ~2 m. The correct fix is to use a spatially adaptive voxel grid — finer near the sensor (high density), coarser at range (already sparse) — rather than a single global leaf size. Open3D's voxel_down_sample uses a uniform grid; adaptive downsampling requires manual implementation or PCL's ApproximateVoxelGrid.
RANSAC fits a single global plane. On a concrete ramp, a tilted floor, or a terrain with a step, the "dominant plane" found by RANSAC may be an average of two surfaces — fitting neither correctly. Points on the true floor then have large residuals and survive into the object cloud, appearing as enormous ground artifacts that trigger false cluster detections. On outdoor terrain or multi-level indoor environments, replace global RANSAC with a height-threshold filter (all points below z = robot_height + threshold in a stabilized frame) or with progressive morphological ground filtering. RANSAC is the right tool only when the ground is approximately planar in each scan window.
KEY INSIGHTHow this connects
- Sensors: Physics, Noise Models, and Failure Modes (1.6) — the fundamental noise model for LIDAR (range noise, angular uncertainty, multipath) that motivates statistical outlier removal. Understanding why returns are noisy is required to tune SOR parameters intelligently rather than by trial and error.
- Coordinate Frames and Homogeneous Transforms (1.2) — the coordinate conversion from sensor frame to base_link frame that makes multi-sensor fusion possible. Cluster centroids published in the wrong frame are geometrically valid but behaviorally wrong; the TF2 transform tree governs which frame each processing step operates in.
- Stereo Vision and RGB-D Depth Sensing (5.2) — the camera-side depth data that fuses with LIDAR clusters to produce stable 3D person positions. LIDAR clusters give geometric structure; camera detections give semantic labels. The fusion node depends on both.
- Feature Detection and Visual Odometry (5.4) → — next: camera-side features and how visual odometry complements LIDAR scan matching for motion estimation, particularly in featureless environments where LIDAR degrades.
The pipeline structure introduced here — convert, filter, segment, cluster — is not unique to LIDAR. RGB-D depth maps go through the same stages. The mathematical tools differ (RANSAC is agnostic to sensor type) but the ordering and the failure modes are the same. Internalize the ordering rule now and it will serve you across every 3D perception task.
GLOSSARYKey terms introduced
- Point cloud
- A set of 3D Cartesian coordinates (x, y, z) representing the surfaces visible to a depth sensor. May optionally carry additional channels (intensity, RGB, ring index). The primary output of 3D LIDAR sensors and the input format for most 3D perception pipelines.
- Voxel downsampling
- Spatial subsampling that replaces all points within each cell of a 3D grid (voxel) with their centroid. Controlled by the leaf size L. Reduces compute cost at the expense of geometric detail below scale L.
- Statistical outlier removal (SOR)
- A noise filter that computes, for each point, the mean distance to its k nearest neighbors. Points whose mean neighbor distance exceeds mean + std_ratio × σ across the cloud are removed as outliers. Effective against isolated noise returns without requiring range thresholds.
- RANSAC (Random Sample Consensus)
- An iterative robust estimator that fits a model (here: a plane) by repeatedly sampling the minimum number of points, counting inliers (points consistent with the model within a distance threshold), and keeping the best hypothesis. Robust to outliers by design; sensitive to the distance threshold and iteration count.
- Ground segmentation
- The process of separating floor or terrain returns from object returns in a 3D point cloud. Must precede Euclidean clustering; typically implemented via RANSAC plane fitting (indoor) or morphological filtering (outdoor terrain).
- Euclidean clustering
- A spatial grouping algorithm that assigns points to the same cluster if they are within a distance tolerance ε of at least one other cluster member. Equivalent to DBSCAN with min_samples = 1. Parameters: cluster tolerance ε, minimum and maximum cluster size.
- Motion distortion
- Geometric corruption of a point cloud caused by robot motion during the sensor acquisition window. Points at different scan angles were acquired from different robot poses; treating them as simultaneous produces a sheared or blurred cloud. Corrected by interpolating the robot pose at each beam's timestamp (motion compensation).
- PointCloud2
- The ROS 2 standard message type for 3D point cloud data. Stores points as a byte buffer with a field descriptor schema (field names, types, offsets). Parsed with
sensor_msgs_py.point_cloud2.read_points()in Python.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1You run Euclidean clustering on a raw (unprocessed) indoor LIDAR scan. What do you expect the largest cluster to contain, and why does this break person detection?›
The largest cluster will contain the floor. In an indoor environment, the floor is the largest continuous planar surface — thousands of LIDAR returns all within the cluster tolerance ε of each other. Euclidean clustering correctly groups them into one massive cluster. This breaks person detection in two ways: (1) the floor cluster consumes the majority of processing time and may exceed max_cluster_size, triggering rejection; (2) floor points near the person's feet are spatially connected to the person cluster, causing person+floor to merge into a single cluster whose centroid is at ankle height and whose bounding box spans the room. Person detection sees no person-shaped cluster; only a room-shaped blob. Fix: run RANSAC ground removal before clustering, every time.
2FollowBot reliably detects a person at 1.5 m but loses them at 3.0 m. The LIDAR reports valid returns at that range. What pipeline parameter is the likely culprit, and how would you fix it?›
The likely culprit is the voxel leaf size being too coarse, combined with a minimum cluster size that is too large. At 3 m range, the angular density of a 360° LIDAR scan means a 0.5 m-wide person subtends fewer returns than at 1.5 m range. After voxel downsampling at, say, 0.10 m leaf size, the person at 3 m may collapse to 10–15 points — below a min_cluster_size of 50. The person disappears from the cluster output even though the sensor sees them. Fixes: (1) reduce leaf size to 0.05 m or finer for near-person range; (2) reduce min_cluster_size to 15–20 points (accepting more false positives, which can be filtered by size/shape checks); (3) implement range-adaptive voxel sizing. The correct diagnosis requires plotting point count inside the expected person bounding box as a function of range — not just looking at cluster counts.
3What is motion distortion in a spinning LIDAR, and under what FollowBot operating conditions is it most problematic?›
Motion distortion occurs because a spinning LIDAR acquires different angular sectors of its scan at different instants in time. For a 10 Hz scanner, the first and last returns in a single "scan" are 100 ms apart. If the robot is moving during that window, each return was captured from a different pose — but all are timestamped identically (or treated as simultaneous). The result is a geometrically distorted point cloud: a straight wall appears curved, a circular environment appears sheared. For FollowBot, motion distortion is most problematic during sharp turns at speed — for example, when the tracked person steps sideways quickly and FollowBot spins to track them. At 60°/s angular velocity, the sensor moves ~6° during one scan revolution, producing visible scan distortion and degraded SLAM scan matching. It is least problematic during slow forward-following motion (small heading change during scan window). Fix: enable motion compensation in laser_geometry by providing the TF buffer with a high-rate odometry or IMU transform.
4RANSAC is described as "robust to outliers." What does that mean concretely, and when does RANSAC fail for ground segmentation?›
RANSAC is robust to outliers because it fits its model using a small random sample (3 points for a plane), counts inliers at each iteration, and selects the model with the most inliers — never requiring all points to agree. Even if 40–50% of the cloud is object points (non-floor), RANSAC can still find the ground plane because random sampling will eventually draw 3 floor points, and the floor will score more inliers than any other plane. Concretely: on a 1,000-point cloud where 600 are floor, 100 RANSAC iterations have a ~99.9% probability of drawing a 3-floor-point sample at least once. RANSAC fails for ground segmentation when: (1) the "ground" is not a single plane — a ramped floor, a terrain step, or a sloped outdoor surface means no single plane model fits the majority of floor points; (2) the distance threshold d is poorly chosen — too small causes floor roughness to be classified as objects, too large absorbs low objects into the ground set; (3) the point cloud has been heavily downsampled before RANSAC, reducing the inlier count below statistical reliability (run RANSAC before voxel downsampling, or reduce the voxel size for the ground-fit step). On multi-plane surfaces, a height-threshold approach (keep only points with z above a fixed height in a stabilized frame) is more reliable than RANSAC.