Obstacle detection on mining roads based on multi-scale feature fusion and attention mechanism

doi:10.16265/j.cnki.issn1003-3033.2024.09.1631

Abstract

Abstract:

In order to solve the problem of travelling obstacle detection in the context of complex open pit mines, a mining road obstacle detection algorithm based on improved cross-scale feature fusion is proposed. Firstly, to address the problem of unbalanced small target sample categories in the original mine dataset, a data enhancement method based on geometric transformation and weighted Poisson fusion is used to expand the number of samples.Secondly, a cross-stage connectivity network that is more suitable for obstacle detection is proposed in the feature extraction stage to increase the detection scale and improve the algorithm's learning ability of the small target features, and then a 3D parameterless attention (SimAM) and de-weighted Bi-directional feature fusion pyramid network (Bi-FPN) are proposed in the feature fusion stage to improve the multi-scale detection performance by enlarging the predicted feature map and feature receptive field. Finally, to address the problems of sample imbalance and imprecise obstacle bounding box localisation in the training, the quality focal loss function (QFL) and the scalable Intersection and combination ratio loss function (SIoU), which combines the classification score with the quality prediction of the position to improve the localisation accuracy for dense occlusion targets. The results show that the improved method can effectively identify unstructured road obstacles in open pit mining area under complex background, and in practical application, the detection accuracy reaches 91.88% and the detection speed reaches 68.7 f/s, which has a better performance of small-target and multi-scale detection compared with the mainstream detection methods, and it can satisfy the requirements of obstacle safety detection in the travelling of unmanned mine cards in open pit mining area.

Key words: multi-scale, feature fusion, attention mechanism, mining roads, obstacle detection, data enhancement

CLC Number:

X948

LI Gang, DU Yabo, YANG Qinghe, MAO Mengying, JIA Dongping. Obstacle detection on mining roads based on multi-scale feature fusion and attention mechanism[J]. China Safety Science Journal, 2024, 34(9): 87-98.

Figures/Tables 16

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Table 1

Fig.7

Table 2

Table 3

Fig.8

Fig.9

Table 4

Table 5

Fig.10

Table 6

Ablation experiment

数据扩增	数据均衡	多尺度	Bi-FPN	SimAM	QFL	SIoU	VOC数据精度 $/ %$	矿山数据精度 $/ %$	坑洞精度 $/ %$
—	—	—	—	—	—	—	88.71	73.67	34.24
√	—	—	—	—	—	—	88.71	83.69	65.52
—	√	—	—	—	—	—	88.71	87.82	72.69
—	√	√	—	—	—	—	88.90	88.49	73.30
—	√	√	√	—	—	—	89.30	88.89	73.65
—	√	√	√	√	—	—	90.89	89.40	74.98
—	√	√	√	√	√	—	92.13	90.56	75.66
—	√	√	√	√	√	√	93.21	91.88	77.12

Table 6

References 20

[1]	付恩三, 刘光伟, 白润才, 等. 大数据时代矿山协同监管监察新思路[J]. 中国安全科学学报, 2023, 33(8): 39-44. doi: 10.16265/j.cnki.issn1003-3033.2023.08.2035
	FU Ensan, LIU Guangwei, BAI Runcai, et al. Exploration and research of mine multi-element cooperative supervision new thinking in era of big data[J]. China Safety Science Journal, 2023, 33(8): 39-44. doi: 10.16265/j.cnki.issn1003-3033.2023.08.2035
[2]	范冰倩, 董秉聿, 王彪, 等. 基于深度学习的地铁施工作业人员不安全行为识别与应用[J]. 中国安全科学学报, 2023, 33(1): 41-47. doi: 10.16265/j.cnki.issn1003-3033.2023.01.0874
	FAN Bingqian, DONG Bingyu, WANG Biao, et al. Identification and application of unsafe behaviors of subway construction workers based on deep learning[J]. China Safety Science Journal, 2023, 33(1): 41-47. doi: 10.16265/j.cnki.issn1003-3033.2023.01.0874
[3]	陈志发, 余贵珍, 张传莹, 等. 露天矿区无人驾驶行车风险评估及防控策略仿真研究[J]. 煤炭学报, 2023, 48(4): 1782-1797.
	CHEN Zhifa, YU Guizhen, ZHANG Chuanying, et al. Driving risk assessment and prevention strategies for autonomous vehicle in open-pits[J]. Journal of China Coal Society, 2023, 48(4): 1782-1797.
[4]	张彦斐, 封子晗, 张嘉恒, 等. 基于特征融合的果园非结构化道路识别方法[J]. 农业机械学报, 2023, 54(7):35-44,67.
	ZHANG Yanfei, FENG Zihan, ZHANG Jiaheng, et al. Recognition method of orchard unstructured road based on feature fusion[J]. Transactions of the Chinese Society for Agricultural Machinery, 2023, 54(7):35-44, 67.
[5]	卢才武, 马玲, 阮顺领. 矿区复杂道路边缘检测[J]. 测绘科学, 2020, 45(9):123-131.
	LU Caiwu, MA Ling, RUAN Shunling. Mining area complex road edge detection based on HSV space[J]. Science of Surveying and Mapping, 2020, 45(9): 123-131.
[6]	郝明, 林惠晶, 高彦彦. 基于改进主动轮廓模型的无人机影像矿区地裂缝提取[J]. 地球信息科学学报, 2022, 24(12):2448-2457. doi: 10.12082/dqxxkx.2022.220376
	HAO Ming, LIN Huijing, GAO Yanyan. Ground fissure extraction method based on improved active contour model for UAV images in mining areas[J]. Journal of Geo-information Science, 2022, 24(12): 2448-2457.
[7]	CHENG Yong, WANG Wei, ZHANG Wenjie, et al. A multi-feature fusion and attention network for multi-scale object detection in remote sensing images[J]. Remote Sensing, 2023, 15(8): 2096-2115.
[8]	马永杰, 马芸婷, 程时升, 等. 基于改进YOLOv3模型与Deep-SORT算法的道路车辆检测方法[J]. 交通运输工程学报, 2021, 21(2): 222-231.
	MA Yongjie, MA Yunting, CHENG Shisheng, et al. Road vehicle detection method based on improved YOLOv3 model and Deep-SORT algorithm[J]. Journal of Traffic and Transportation Engineering, 2021, 21(2): 222-231.
[9]	REDMON J, FARHADI A. Yolov3: an incremental improvement[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2018: 89-95.
[10]	张炳力, 秦浩然, 江尚, 等. 基于 RetinaNet 及优化损失函数的夜间车辆检测方法[J]. 汽车工程, 2021, 43(8): 1195-1202.
	ZHANG Bingli, QIN Haoran, JIANG Shang, et al. A method of vehicle detection at night based on RetinaNet and optimized loss functions[J]. Automotive Engineering, 2021, 43(8): 1195-1202.
[11]	HE Kaiming, ZHANG Xiangyu, REN Saoqing, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. doi: 10.1109/TPAMI.2015.2389824 pmid: 26353135
[12]	NAJIBI M, SAMANGOUEI P, CHELLAPPA R, et al. Ssh: single stage headless face detector[C]. Proceedings of the IEEE International Conference on Computer Vision, 2017: 4875-4884.
[13]	YANG Lingxiao, ZHANG Ruyuan, LI Lida, et al. SimAM: a simple, parameter-free attention module for convolutional neural networks[C]. International Conference on Machine Learning. PMLR, 2021: 11 863-11 874.
[14]	HU Jie, SHEN LI,SUN Gang. Squeeze-and-excitation networks[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7132-7141.
[15]	WOO S, PARK J, LEE J Y, et al. Cbam: convolutional block attention module[C]. Proceedings of the European Conference on Computer Vision (ECCV),2018: 3-19.
[16]	LI Xiang, WANG Wenhai, WU Lijun, et al. Generalized focal loss: learning qualified and distributed bounding boxes for dense object detection[J]. Advances in Neural Information Processing Systems, 2020, 33: 21 002-21 012.
[17]	石祥滨, 贾贺林. 基于控制点网格优化的航拍图像拼接算法[J]. 计算机应用与软件, 2020, 37(5): 303-308.
	SHI Xiangbin, JIA Helin. Aerial image stitching algorithm based on control point grid optimization[J]. Computer Applications and Software, 2020, 37(5): 303-308.
[18]	HAO Qibin, ZHOU Daquan, FENG Jiashi. Coordinate attention for efficient mobile network design[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 13 713-13 722.
[19]	WANG Qilong, WU Banggu, ZHU Pengfei, et al. ECA-Net: efficient channel attention for deep convolutional neural networks[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 11 534-11 542.
[20]	SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-cam: visual explanations from deep networks via gradient-based localization[C]. Proceedings of the IEEE International Conference on Computer Vision, 2017: 618-626.

类别	样本数量/个
类别	卡车	挖机	推土机	行人	汽车	坑洞	总图片数
训练集数据增强前	7 739	2 786	779	387	997	331	4 909
训练集数据增强后	38 695	13 930	3 895	1 935	4 985	1 655	24 545
训练集数据均衡后	9 240	4 779	4 887	8 605	7 385	3 906	24 204

训练数据	预训练数据	文中数据
图像输入尺寸	640×640×3	640×640×3
冻结训练batchsize	—	16
冻结训练Epoch	—	20
冻结训练学习率	—	0.001 25
解冻训练batchsize	16	16
解冻训练Epoch	50	280
解冻训练学习率	0.005	0.01
动量参数	0.937	0.937
优化方法	SGD	SGD
NMS阈值	0.65	0.65

模型	输入尺寸	精确率/%	速度/(帧·s^-1)	参数量/MB	计算量/GB	坑洞精度
SSD	300×300	61.37	176.3	24.41	30.68	41.74
Faster-RCNN	800×800	80.36	48.0	41.37	134.09	61.57
RetinaNet	600×600	84.50	38.1	36.43	82.45	65.54
EfficientDet-d3	896×896	78.72	42.5	18.44	108.20	58.70
RepVGG-A2	512×512	83.99	52.4	41.12	57.66	67.39
YOLOX-m	640×640	82.04	95.6	25.28	36.76	61.68
YOLOX-tiny	640×640	77.54	133.0	5.03	7.58	56.88
Our Model	640×640	91.88	68.7	17.80	20.77	76.51

主干网络	VOC精度/%	矿山精度/%	参数量/ MB	坑洞精度/%
Base	88.71	87.82	7.06	72.69
ResNet50	91.47	88.61	25.60	73.01
RepVGG A0	74.33	77.70	7.03	56.81
CSPDarknnet+	88.90	88.49	7.79	73.30

注意力模块	VOC精度/%	矿山精度/%	坑洞精度/%	参数量/ MB	计算量/ MB
Baseline	89.30	88.89	73.65	+0	+0
SENet	90.95	89.39	74.80	+0.021	3.072
CBAM	90.78	88.54	72.67	+0.003	6.144
CA	90.87	88.98	73.56	+0.012	7.168
ECA	90.97	89.40	75.05	+0	2.048
SimAM	90.89	89.40	74.92	+0	+0