A Benchmark and Simulator for UAV Tracking
Matthias MuellerNeil G. SmithBernard Ghanem
Presents a dedicated low-altitude aerial tracking benchmark of 123 fully annotated HD video sequences alongside an Unreal Engine-based simulator for real-time evaluation and synthetic data generation.
Deploying automated visual tracking on unmanned aerial vehicles (UAVs) offers significant utility across applications such as search and rescue, surveillance, wildlife monitoring, and navigation. However, existing tracking algorithms have primarily been evaluated on ground-based datasets that fail to capture the specific conditions encountered during airborne flight, such as abrupt camera motion, rapid aspect ratio shifts, and drastic scale changes.
The article establishes a dedicated baseline for low-altitude aerial tracking by evaluating state-of-the-art computer vision trackers against a new high-definition aerial dataset and introducing a photo-realistic, closed-loop UAV simulator for real-time evaluation.
To achieve this, the authors curated the UAV123 dataset, comprising 123 fully annotated video sequences and over 110,000 frames captured from low-altitude flights, small low-cost drones, and synthetic sequences. They benchmarked 14 leading tracking algorithms using standard precision and success metrics across diverse visual attributes. In addition, they developed an Unreal Engine 4 simulator to evaluate how tracking latency and accuracy directly interact with UAV flight dynamics and control systems in real time.
The investigation produced several key findings: First, trackers performed substantially worse on aerial video than on standard ground-based benchmarks, exposing distinct vulnerabilities to scale variation, aspect ratio changes, and low resolution. Second, correlation-filter-based algorithms incorporating scale adaptation, specifically SRDCF, achieved top performance on the overall aerial benchmark, while algorithms with memory-retention mechanisms like MUSTER excelled in long-term tracking scenarios. Third, tracker processing speed emerged as a critical operational bottleneck; when video processing dropped to 10 frames per second to emulate compute constraints, tracker accuracy degraded by 7% to 36%. Finally, closed-loop simulation revealed that tracker lag creates instability during rapid maneuvers, though increasing UAV flight altitude or fine-tuning flight controllers helped mitigate tracking errors caused by computational delay.
These results demonstrate that standard computer vision trackers cannot be directly deployed onto aerial platforms without substantial performance degradation. For operational systems, algorithm latency introduces physical safety risks and mission failure when drones attempt to follow dynamic targets, meaning speed is just as critical as baseline accuracy.
For future development and system deployment, engineering teams should prioritize integrating scale and aspect-ratio adaptation into lightweight, correlation-based trackers. System designers should utilize closed-loop simulation environments to validate tracking algorithms against flight control parameters before field testing, while considering dynamic flight tactics—such as temporarily increasing altitude—to maintain target locks when tracking fast-moving objects.
While the benchmark and simulator provide robust, highly controlled testing environments, the evaluations rely on standardized parameter sets and a specific drone model. Readers should note that performance may vary under harsher field conditions, diverse sensor payloads, or unmodeled weather dynamics.
- Paper: Object Tracking Benchmark, Yi Wu et al. (2015). Its standardized single-object tracking protocols and attribute-based evaluation provide the benchmark conventions that this paper adapts to reveal how aerial conditions change tracker performance.
- Paper: Learning Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking, Yongxin Li et al. (2024). Building on the aerial benchmarks and real-time constraints established here, AVTrack evaluates adaptive tracking on UAV123 and UAV123@10fps and advances toward efficient onboard deployment.
- Paper: TCTrack: Temporal Contexts for Aerial Tracking, Ziang Cao et al. (2022). Using the aerial-tracking benchmark setting this paper established, TCTrack extends the work by exploiting temporal context to improve robust, real-time tracking on resource-limited UAVs.
