Sensor Fusion

Definition

Sensor fusion is the process of combining data from multiple sensors to produce a more accurate, consistent, and reliable estimate of a system's state than any single sensor could provide alone. In robotics, it is essential for tasks like localization, navigation, and perception, where individual sensors have complementary strengths and weaknesses.

Formula

\hat{x} = \hat{x}^- + K(z - H\hat{x}^-)K = P^- H^\top (H P^- H^\top + R)^{-1}

In-Depth Explanation

No single sensor is perfect. Each sensor type has limitations: - GPS: Accurate globally but unavailable indoors, high latency - IMU: High-frequency motion data but drifts over time - LiDAR: Accurate depth but no color/texture, affected by rain - Camera: Rich visual info but requires depth estimation, affected by lighting - Wheel odometry: Simple, low-cost but accumulates error (slip) Sensor fusion combines these complementary sources to compensate for individual weaknesses. Common sensor fusion architectures: 1. Complementary filter: - Simple, computationally efficient - Classic use: Fuse accelerometer (low-freq accurate) + gyroscope (high-freq, drifts) for attitude estimation 2. Kalman Filter (KF): - Optimal estimator for linear systems with Gaussian noise - Predicts state using a motion model, then corrects with sensor measurements - State update: x̂ = x̂⁻ + K(z - Hx̂⁻) - Gain: K = P⁻Hᵀ(HP⁻Hᵀ + R)⁻¹ 3. Extended Kalman Filter (EKF): - Linearizes nonlinear systems around current estimate - Standard for robot localization (EKF-SLAM, GPS/IMU fusion) 4. Unscented Kalman Filter (UKF): - Better handles highly nonlinear systems - Uses sigma points instead of Jacobians 5. Particle Filter: - Non-parametric, handles multimodal distributions - Used in Monte Carlo Localization (MCL/AMCL) for mobile robots 6. Deep learning fusion: - Neural networks learn fusion weights from data - End-to-end sensor fusion for autonomous driving perception Practical example — IMU + LiDAR fusion (LIO-SAM): A drone's IMU provides 400 Hz motion estimates but drifts after seconds. A 3D LiDAR provides accurate point clouds at 10 Hz. LIO-SAM tightly couples both: the IMU predicts motion between LiDAR scans, and LiDAR scans correct IMU drift. The result is accurate, high-frequency odometry robust to aggressive maneuvers. ROS packages for sensor fusion: - robot_localization: EKF/UKF-based state estimation (GPS + IMU + odometry) - cartographer: LiDAR + IMU SLAM - msckf_vio / VINS-Mono: Visual-inertial odometry (camera + IMU)

Related Terms

LiDAR

LiDAR (Light Detection and Ranging) is a remote sensing technology that measures distances by emitting laser pulses and detecting the reflected light. In robotics, LiDAR sensors generate 2D or 3D point clouds of the surrounding environment, enabling obstacle detection, mapping, and localization.

Reinforcement Learning (RL) in Robotics

Reinforcement Learning (RL) is a machine learning paradigm where an agent learns to make decisions by interacting with an environment, receiving rewards for desirable actions and penalties for undesirable ones. In robotics, RL enables robots to learn complex behaviors — such as locomotion, manipulation, and navigation — without explicit programming of every motion.

ROS (Robot Operating System)

ROS (Robot Operating System) is an open-source middleware framework for robot software development. Despite its name, ROS is not a traditional operating system — it provides tools, libraries, and conventions that simplify the creation of complex and reusable robot software across a wide variety of robotic platforms.

SLAM (Simultaneous Localization and Mapping)

SLAM (Simultaneous Localization and Mapping) is the computational problem of constructing or updating a map of an unknown environment while simultaneously tracking a robot's location within it. It is a foundational capability for autonomous mobile robots operating without GPS or pre-built maps.