Unsafe behavior recognition of air traffic controllers based on ST-GCN

doi:10.16265/j.cnki.issn1003-3033.2023.05.1396

Abstract

Abstract:

In order to prevent and supervise the violation behavior in ATC, an unsafe behavior recognition model suitable for controllers' sitting posture was studied by intelligent video analysis technology. Firstly, the hidden characteristics of unsafe working behaviors of air traffic controllers were analyzed, and five typical unsafe behaviors of air traffic controllers were summarized, including stretching, dozing, falling asleep, crooked asleep and half lying asleep, and the video data set of controllers unsafe working status (CUWS) was constructed. Secondly, a skeleton key point expansion algorithm that can describe the controller's sitting posture was proposed. Based on ST-GCN, an ATC-ST-GCN model suitable for sitting posture and legs hidden from view was built,and the workflow of the controller's unsafe behavior recognition was given. Finally, the ATC-ST-GCN model was trained and tested using the CUWS dataset, and the verification test was carried out using the actual surveillance video of the control room. The results show that the model can identify five typical unsafe behaviors on the limited validation dataset, and the accuracy rate reaches 93.65%. The experimental results show that the model is scientific and effective.

Key words: spatial temporal graph convolutional network (ST-GCN), air traffic control(ATC), unsafe behavior, air traffic controller, behavior recognition

WANG Chao, XU Chuxin, DONG Jie, WANG Zhifeng. Unsafe behavior recognition of air traffic controllers based on ST-GCN[J]. China Safety Science Journal, 2023, 33(5): 42-48.

Figures/Tables 11

Fig.1

Tab.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Tab.2

Fig.9

References 16

[1]	DOC9966-2016, Manual for the oversight of fatigue management approaches[S].
[2]	靳慧斌, 张静, 吕川. HRV在管制员疲劳检测中的适用性[J]. 北京航空航天大学学报, 2018, 44(11): 2292-2298.
	JIN Huibin, ZHANG Jing, LYU Chuan. Application of HRV in air traffic controllers' fatigue detection[J]. Journal of Beijing University of Aeronautics and Astronautics, 2018, 44(11): 2292-2298.
[3]	王莉莉, 朱敏. 基于脑电数据的管制架次对管制员疲劳影响研究[J]. 中国安全科学学报, 2021, 31(2): 173-178. doi: 10.16265/j.cnki.issn 1003-3033.2021.02.024
	WANG Lili, ZHU Min. Research on influence of controlled sorties on controllers' fatigue based on EEG data[J]. China Safety Science Journal, 2021, 31(2): 173-178. doi: 10.16265/j.cnki.issn 1003-3033.2021.02.024
[4]	JI Yingyu, WANG Shigang, LU Yang, et al. Eye and mouth state detection algorithm based on contour feature extraction[J]. Journal of Electronic Imaging, 2018, 27(5): DOI: 10.1117/1.JEI.27.5.051205. doi: 10.1117/1.JEI.27.5.051205
[5]	YANG Hao, LIU Li, MIN Weidong, et al. Driver yawning detection based on subtle facial action recognition[J]. IEEE Transactions on Multimedia, 2020, 23: 572-583. doi: 10.1109/TMM.2020.2985536
[6]	HUANG Rui, WANG Yan, LI Zijian, et al. RF-DCM: Multi-granularity deep convolutional model based on feature recalibration and fusion for driver fatigue detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(1): 630-640. doi: 10.1109/TITS.2020.3017513
[7]	卜建, 刘银鑫, 王艳军. 空中交通管制员的眼动行为与疲劳关系[J]. 航空学报, 2017, 38(增1): 57-62.
	BU Jian, LIU Yinxin, WANG Yanjun. Relationship between air traffic controllers' eye movement and fatigue[J]. Acta Aeronautica et Astronautica Sinica, 2017, 38(S1): 57-62.
[8]	刘忠育. 基于深度学习的矿工不安全行为识别方法研究[D]. 徐州: 中国矿业大学, 2021.
	LIU Zhongyu. Research on recognition methods of miners' unsafe behavior based on deep learning[D]. Xuzhou: China University of Mining and Technology, 2021.
[9]	崔铁军, 王凌霄. YOLOv4目标检测算法在煤矿工人口罩佩戴监测工作中的应用研究[J]. 中国安全生产科学技术, 2021, 17(10): 66-71.
	CUI Tiejun, WANG Lingxiao. Research on application of YOLOv4 object detection algorithm in monitoring on masks wearing of coal miners[J]. Journal of Safety Science and Technology, 2021, 17(10): 66-71.
[10]	刘耀, 焦双健. ST-GCN在建筑工人不安全动作识别中的应用[J]. 中国安全科学学报, 2022, 32(4): 30-35. doi: 10.16265/j.cnki.issn1003-3033.2022.04.005
	LIU Yao, JIAO Shuangjian. Application of ST-GCN in unsafe action identification of construction workers[J]. China Safety Science Journal, 2022, 32(4): 30-35. doi: 10.16265/j.cnki.issn1003-3033.2022.04.005
[11]	FANG Weili, LOVE P E D, LUO Hanbin, et al. Computer vision for behaviour-based safety in construction: a review and future directions[J]. Advanced Engineering Informatics, 2020, 43: DOI: 10.1016/j.aei.2019.100980. doi: 10.1016/j.aei.2019.100980
[12]	KWON Seongkyung, SEO Jihwan, YUN Junyoung, et al. Driving behavior classification and sharing system using CNN-LSTM approaches and V2X communication[J]. Applied Sciences, 2021, 11(21): 8856-8878. doi: 10.3390/app11198856
[13]	QI Mengshi, WANG Yunhong, QIN Jie, et al. StagNet: an attentive semantic RNN for group activity and individual action recognition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(2): 549-565. doi: 10.1109/TCSVT.76
[14]	DING Lieyun, FANG Weili, LUO Hanbin, et al. A deep hybrid learning model to detect unsafe behavior: integrating convolution neural networks and long short-term memory[J]. Automation in Construction, 2018, 86: 118-124. doi: 10.1016/j.autcon.2017.11.002
[15]	温廷新, 王贵通, 孔祥博, 等. 基于迁移学习与残差网络的矿工不安全行为识别[J]. 中国安全科学学报, 2020, 30(3):41-46. doi: 10.16265/j.cnki.issn1003-3033.2020.03.007
	WEN Tingxin, WANG Guitong, KONG Xiangbo, et al. Identification of miners' unsafe behaviors based on transfer learning and residual network[J]. China Safety Science Journal, 2020, 30(3): 41-46. doi: 10.16265/j.cnki.issn1003-3033.2020.03.007
[16]	YAN Sijie, XIONG Yuanjun, LIN Dahua. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]. Thirty-second AAAI Conference on Artificial Intelligence, 2018: 8561-8568.

序号	不安全工作行为	行为表现
1	伸懒腰	在5~7 s内出现双臂打开行为
2	瞌睡状态	在5~7 s时间段内频繁点头
3	低头入睡	保持低头状态持续5~7 s
4	歪头入睡	头部靠在椅背上保持静止5~7 s
5	半躺入睡	上半身躺在椅子里保持静止5~7 s

识别模型	Top-1准确率	Top-5准确率
ST-GCN^[16]	71.73	96.83
ATC-ST-GCN	93.65	100.00