Abstract:

To address the perception inaccuracies of autonomous mining trucks in open-pit mines under low-visibility and low-illumination conditions—issues that may lead to obstacle collisions. This paper was proposed a multimodal fusion-based obstacle detection method to enhance detection accuracy and operational safety in complex environments. Firstly, local Feature matching at light speed (LightGlue), was employed to achieve spatial alignment between thermal infrared and visible light images, thereby avoiding spatial misalignment and geometric distortion prior to fusion. Secondly, in the modality feature extraction and fusion stage, a Dual-Modality Feature Fusion (DMFF) module was incorporated into the improved dual-branch backbone network. The extraction capability of dual-modality features was enhanced and fusion was performed through feature compression and cross-modal feature enhancement. An iterative learning method was then introduced to effectively match the complementary information between modalities, generating a fused dual-modality feature map and improving multimodal detection performance. Finally, the fused feature maps at multiple scales were input into the detection head. They were combined with bounding box regression and classification prediction for precise detection. Experimental results demonstrate that the proposed method achieves excellent obstacle detection performance in challenging scenarios with low visibility. Specifically, it achieves a mean Average Precision (mAP@0.5) of 90.8%, and an F₁-score of 0.887, outperforming existing methods in both accuracy and speed. Moreover, the proposed approach exhibits lower false positive and miss detection rates, effectively ensuring the safe navigation of autonomous mining trucks in complex operational environments.

Key words: open-pit mine, low visibility, unmanned driving truck, multi-modal fusion, obstacle detection, perceptual warning system