Object Tracking

Object tracking is one of the core functions in autonomous driving perception systems. Building upon object detection, it interprets and predicts the dynamic behaviour of surrounding road users by establishing and maintaining the correspondence of the same entity across sequential frames. Put simply, while object detection answers "what objects are in the scene right now", object tracking answers "how has this object moved across the past, present, and future".

🎯 Why Is Object Tracking Crucial?

Detection data from an isolated moment only provides snapshot-style information, which is insufficient to support safe and efficient driving decisions. The true value of object tracking lies in:

• Understanding dynamic behaviour: By tracking an object's trajectory, the system can analyse its speed, acceleration, and heading changes to infer intent (such as "this car is preparing to filter into another lane").

• Maintaining identity consistency: In congested or occluded scenarios, tracking algorithms ensure the system does not mix up "Pedestrian A" with "Pedestrian B", which is critical for future trajectory forecasting.

• Informing decision-making and motion planning: Precise tracking data serves as the foundation for predicting future trajectories (such as whether the vehicle ahead will brake abruptly), evaluating collision risks, and planning navigation paths.

⚙️ Core Algorithm Framework

The mainstream paradigm in modern object tracking adopts the "tracking-by-detection" approach, with an algorithm pipeline that typically comprises:

1. Object detection: A detection network (such as the YOLO series) first identifies all objects of interest and their bounding coordinates within the current frame.

2. Motion state prediction: A kinematic motion model (such as the Kalman filter) predicts the expected position of each tracked target in the subsequent frame.

3. Data association: Matching detected targets in the current frame against previously tracked objects forms the most critical stage of the process. It resolves which bounding box corresponds to which existing track.

   • Hungarian algorithm: Frequently employed to solve this global optimal assignment problem by formulating cost matrices and iterative loops to find the lowest-cost match.

   • Mahalanobis distance: Used to gauge the statistical distance between detection boxes and predicted states, effectively factoring in the uncertainty of target motion.

4. State update: The state parameters of each tracked object are updated based on the association results.

🔧 Key Technologies and Challenges

• Multi-sensor fusion tracking: Standalone vision-based tracking is prone to failure in low-light, rainy, or hazy driving conditions. Consequently, advanced autonomous driving suites fuse data from cameras, LiDAR, and millimetre-wave radar for joint tracking—harnessing the unique strengths of each (semantic recognition from cameras, range and speed measurement from radar, and high-precision 3D geometry from LiDAR) to bolster operational robustness. For instance, a study introducing a multi-modal collaborative perception tracking method demonstrated an improvement in tracking accuracy (AMOTA) of 6.33% over existing approaches.

• Prediction-oriented tracking: The ultimate objective of tracking is to facilitate prediction. Advanced architectures (such as the Tracklet-Trajectory-Intent (TRI) framework developed by Huawei) integrate long-term trajectory forecasting with intent classification (such as lane changing or proceeding straight), realising a seamless "tracking-as-prediction" paradigm.

• Key engineering challenges: Systems must continuously tackle hurdles including target occlusion, visual overlap among similar entities, shifting ambient lighting, and computational overhead from high-density traffic. For example, preserving track IDs without dropouts during severe visual obstruction remains an industry-wide challenge, while the finite compute capacity of in-vehicle ECUs necessitates highly optimised, lightweight algorithms. A standard mitigation strategy against occlusion relies on joint matching via motion models and appearance features, where appearance signatures typically stay valid across a brief time window just prior to occlusion.

On the whole, object tracking equips autonomous driving systems with a temporal dimension to comprehend the dynamic driving environment, establishing itself as an indispensable cornerstone for safe and effective vehicular decision-making.