Multi-modal adaptive sensor fusion for UAS odometry : a deep learning approach with hybrid temporal modeling

Abstract

Accurate motion estimation is critical for autonomous unmanned aerial systems (UAS) in global navigation satellite system (GNSS) denied environments. This thesis presents a deep learning framework for six degrees of freedom (6-DOF) pose estimation by integrating RGB cameras, depth sensors, and inertial measurement units (IMU) through adaptive multi-modal sensor fusion.

The architecture comprises two core components: a visual-inertial-depth fusion layer and a hybrid temporal module. The fusion layer achieves adaptive sensor fusion through cross-modal attention and learned importance weighting, dynamically prioritizing the most informative sensors for pose prediction. Poses are predicted using a hybrid temporal modeling unit combining self-attention mechanisms with long short-term memory (LSTM) networks. The system is trained and evaluated on the Mid-Air dataset, incorporating diverse environmental conditions including sunny, cloudy, foggy, and sunset (low-light) scenarios to ensure resilience against varying weather conditions. Experimental evaluation demonstrates competitive performance with absolute trajectory error (ATE) of 0.2032 meters, relative pose error of 0.0821 meters for translation (RPE_trans) and 0.0116 radians for rotation (RPE_rot). The proposed multi-modal approach achieves an 85% improvement in ATE compared to single-modal baselines. The adaptive fusion demonstrates robust performance under sensor degradation, with a 74% degradation in foggy conditions while maintaining functionality. The hybrid temporal modeling outperforms individual LSTM or attention-only approaches, indicating strong potential for real-world UAS deployment.