Recognition of personnel fatigue state and unsafe behavior based on computer vision

doi:10.16265/j.cnki.issn1003-3033.2025.03.0749

Abstract

Abstract:

Taking improving the safety and efficiency of tower crane operation as an example, a method of integrated identification of fatigue state and unsafe behavior was proposed in order to detect the potential safety hazards of drivers in real time. A live video stream was captured via a camera, and the video was analyzed and pre-processed to extract critical information for identifying subsequent fatigue and unsafe behavior. In terms of fatigue state recognition, the analysis method based on the state of eyes and mouth was used to monitor the physiological indicators such as the state of eyes opening and closing, the blink frequency and yawn frequency. In terms of unsafe behavior identification, computer vision and deep learning technology were combined to detect the potential dangerous operations of drivers in real time, thus ensuring timely detection of safety risks. The results show that the performance of the optimized YOLOv5-ECA(Efficient Channel Attention) model is significantly improved in fatigue state and unsafe behavior recognition. The accuracy rate and recall rate of the model on the test set are more than 90%, showing good recognition ability.

Key words: computer vision, fatigue state, unsafe behavior detection, tower crane driver, YOLOv5

CLC Number:

X912安全生理学

LI Hua, WU Lizhou, ZHONG Xingrun, GUO Liangwei, CUI Yuxin. Recognition of personnel fatigue state and unsafe behavior based on computer vision[J]. China Safety Science Journal, 2025, 35(3): 28-35.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Fig.5

Fig.6

Table 2

Fig.7

Fig.8

Fig.9

References 13

[1]	SHAIK M. A systematic review on detection and prediction of driver drowsiness[J]. Transportation Research Interdisciplinary Perspectives, 2023, 21: DOI:10.1016/j.trip.2023.100864.
[2]	MEHMOOD L, LI Heng, UMER W, et al. Non-invasive detection of mental fatigue in construction equipment operators through geometric measurements of facial features[J]. Journal of Safety Research, 2024, 89: 234-250. doi: 10.1016/j.jsr.2024.01.013 pmid: 38858047
[3]	KUWAHARA A, NISHIKAWA K, HIRAKAWA R, et al. Eye fatigue estimation using blink detection based on eye aspect ratio mapping(EARM)[J]. Cognitive Robotics, 2022, 2: 50-59.
[4]	LIU Pengkun, CHI Hunglin, LI Xiao, et al. Effects of dataset characteristics on the performance of fatigue detection for crane operators using hybrid deep neural networks[J]. Automation in Construction, 2021, 132: DOI:10.1016/j.autcon.2021.103901.
[5]	孙世梅, 孙祖航, 冯子阳, 等. 基于行为安全“2-4”模型理论的建筑施工事故行为原因分析[J]. 中国安全科学学报, 2023, 33(11): 30-37. doi: 10.16265/j.cnki.issn1003-3033.2023.11.2450
	SUN Shimei, SUN Zuhang, FENG Ziyang, et al. Behavioral causes analysis of construction accidents based on behavior-based accident causation 24Model theory[J]. China Safety Science Journal, 2023, 33(11): 30-37. doi: 10.16265/j.cnki.issn1003-3033.2023.11.2450
[6]	蔡闯闯, 刘庆华. 基于多特征数据融合的疲劳驾驶检测研究[J]. 江苏科技大学学报:自然科学版, 2023, 37(5): 52-57.
	CAI Chuangchuang, LIU Qinghua. Research on fatigue driving detection based on multi-feature data fusion[J]. Journal of Jiangsu Normal University: Natural Science Edition, 2023, 37(5): 52-57.
[7]	辛嵩, 宋明达, 王泽明, 等. 针对特定驾驶员的疲劳驾驶检测方法[J]. 安全与环境学报, 2023, 23(1): 147-152.
	XIN Song, SONG Mingda, WANG Zeming, et al. Fatigue driving detection method for specific drivers[J]. Journal of Safety and Environment, 2023, 23(1): 147-152.
[8]	王红君, 白浩, 赵辉, 等. 基于计算机视觉的驾驶员疲劳状态检测预警技术[J]. 科学技术与工程, 2022, 22(12): 4887-4894.
	WANG Hongjun, BAI Hao, ZHAO Hui, et al. Driver fatigue state detection and early warning technology based on computer vision[J]. Science Technology and Engineering, 2022, 22(12): 4887-4894.
[9]	方浩杰, 董红召, 林少轩, 等. 多特征融合的驾驶员疲劳状态检测方法[J]. 浙江大学学报:工学版, 2023, 57(7): 1287-1296.
	FANG Haojie, DONG Hongzhao, LIN Shaoxuan, et al. Driver fatigue state detection method based on multi-feature fusion[J]. Journal of Zhejiang University: Engineering Science, 2023, 57(7): 1287-1296.
[10]	田水承, 郭谦, 张江江, 等. 基于蒙特卡罗法的建筑工人不安全行为风险评估[J]. 西安科技大学学报, 2023, 43(2): 373-379.
	TIAN Shuicheng, GUO Qian, ZHANG Jiangjiang, et al. Risk assessment of unsafe behaviors of construction workers based on Monte Carlo method[J]. Journal of Xi'an University of Science and Technology, 2023, 43(2): 373-379.
[11]	李华, 吴立舟, 薛曦澄, 等. 基于计算机视觉的高处临边作业安全巡检[J]. 中国安全科学学报, 2023, 33(9): 69-75. doi: 10.16265/j.cnki.issn1003-3033.2023.09.0202
	LI Hua, WU Lizhou, XUE Xicheng, et al. Computer vision based safety detection of high abutting edges[J]. China Safety Science Journal, 2023, 33(9): 69-75. doi: 10.16265/j.cnki.issn1003-3033.2023.09.0202
[12]	叶贵, 王妍, 任梦雪, 等. 体力疲劳对建筑工人不安全行为的影响效应研究[J]. 中国安全生产科学技术, 2023, 19(1): 122-127.
	YE Gui, WANG Yan, REN Mengxue, et al. Study on influence effect of physical fatigue on unsafe behavior of construction workers[J]. Journal of Safety Science and Technology, 2023, 19(1): 122-127.
[13]	史玉芳, 卢吉发. 基于SEM的建筑工人疲劳对不安全行为影响机理[J]. 西安科技大学学报, 2020, 40(4): 712-719.
	SHI Yufang, LU Jifa. Research on the impact of construction workers fatigue on unsafe behaviors based on SEM[J]. Journal of Xi'an University of Science and Technology, 2020, 40(4): 712-719.

Perclos取值	疲劳等级
[0,0.1)	清醒
[0.1,0.3)	临界状态
[0.3,0.4)	轻度疲劳
[0.4,0.5)	中度疲劳
[0.5,1.2)	严重疲劳

训练方法	P	R	mAP@ 0.5	mAP@ 0.5:0.95	模型内存/MB
YOLOv5	0.914	0.897	0.934	0.726	13.7
YOLOv5-CA	0.923	0.901	0.942	0.723	15.2
YOLOv5-CBAM	0.911	0.909	0.941	0.722	14.7
YOLOv5-ECA	0.923	0.905	0.940	0.723	13.7
YOLOv5-SE	0.904	0.904	0.934	0.715	14.7

[1]	ZHANG Hao, HAIMUDULA Aierken. Enterprise safety risk management model of machinery manufacturing industry based on improved YOLOv5 [J]. China Safety Science Journal, 2025, 35(3): 52-59.
[2]	ZHANG Zehui, ZHANG Qianlong, XU Xiaobin, ZHAO Zuguo, WANG Haiquan, LI Hao. Unsafe behavior recognition model of high climbing workers based on vision [J]. China Safety Science Journal, 2025, 35(2): 144-151.
[3]	LIU Huilin, FANG Qiong, JIANG Yu, WEI Huazhang, WANG Tao, ZHANG Shuchuan. A lightweight forest fire detection algorithm based on YOLOv5s [J]. China Safety Science Journal, 2025, 35(1): 75-83.
[4]	ZHAO Jiangping, WANG Xinran, WU Lizhou. Model of pavement pothole target detection with improved YOLOv5s [J]. China Safety Science Journal, 2025, 35(1): 67-74.
[5]	LEI Bin, YAN Langlang, YU Hua, WEN Yan, ZHANG Liang, LI Zhexu. Construction of rail transit emergency early warning perception system under YOLOv5 detection algorithm [J]. China Safety Science Journal, 2024, 34(9): 225-233.
[6]	JI Zhi'an, ZHOU Yunyi, ZHANG Yuyuan, GUO Xinran, SHI Kun. Industrial site unsafe behavior detection based on improved YOLOv5 [J]. China Safety Science Journal, 2024, 34(7): 38-43.
[7]	WU Lizhou, LI Hua, LI Dianbin, WU Yujin, LIU Panwang, XUE Xicheng. Crane danger zone intrusion warning based on computer vision [J]. China Safety Science Journal, 2024, 34(7): 139-146.
[8]	ZHANG Guangyuan, DENG Long, WANG Yawei, SUN Ziwei, LI Sha, CHEN Cheng. Recognition of fatigue state of high-speed rail dispatchers based on EEG signal characteristics [J]. China Safety Science Journal, 2024, 34(6): 235-246.
[9]	ZHANG Shulin, ZHANG Ya'nan, TIAN Chao, YAN Xiang, LU Yi, SHI Shiliang. Study on flame image recognition of chemical industrial park fires based on convolutional neural network [J]. China Safety Science Journal, 2024, 34(1): 179-186.
[10]	WANG Yaohan, SONG Zeyang, ZHANG Lidong. Research on safety sign classification based on CNN [J]. China Safety Science Journal, 2023, 33(S1): 263-269.
[11]	LI Hua, WU Lizhou, XUE Xicheng, ZHONG Xingrun. Computer vision based safety inspection of high abutting edges [J]. China Safety Science Journal, 2023, 33(9): 69-75.
[12]	ZHANG Lei, LI Xiwei, YAN Qianru, WANG Haosheng, LEI Weiqiang. Personnel detection algorithm in fully mechanized coal face based on improved YOLOv5s [J]. China Safety Science Journal, 2023, 33(7): 82-89.
[13]	WANG Zhe, LI Xiang, YANG Dongliang, LIU Dan. Fire detection model of wildland-urban interface based on YOLOv5s [J]. China Safety Science Journal, 2023, 33(6): 152-158.
[14]	ZHAO Jiangping, LIU Xingxing, ZHANG Xiangzhuo. Research on image recognition technology for external scaffold hidden danger based on improved YOLOv5s [J]. China Safety Science Journal, 2023, 33(12): 60-66.
[15]	ZHANG Jun, YIN Liu, GONG Xinfei, XU Hehua. Small target smoke detection algorithm based on improved YOLOv5 [J]. China Safety Science Journal, 2023, 33(11): 75-81.