Overview of recognition methods of pedestrian abnormal behaviors in public places

doi:10.16265/j.cnki.issn1003-3033.2024.02.1125

Abstract

Abstract:

The purpose of this research is to clarify the research progress of the theory and technology of pedestrian abnormal behavior recognition in public places. Firstly, with the help of China National Knowledge Infrastructure (CNKI) and the Web of Science (WOS), a broad definition and universal characteristics of abnormal pedestrian behavior in public places were given. The existing research results related to abnormal behaviors were divided into three categories: harmful behaviors, dissociable behaviors and violations. Then, from the perspective of data and technological foundations, the existing abnormal behavior recognition methods were divided into four categories: artificial design, human skeleton, Red Geen Blue(RGB) images and wearable sensors. Secondly, this study sorted out the abnormal behavior datasets of mainstream populations both domestically and internationally, and analyzed the performance of relevant algorithms on the datasets. Finally, the limitations of existing research methods in available datasets and data fusion detection were summarized, and future research directions and optimization suggestions were provided. The results indicate that these four types of abnormal behavior recognition methods have their own advantages and disadvantages. It is necessary to construct a diversified, well-defined and high-quality international benchmark dataset of abnormal behaviors among the crowd. Future research should focus on robust and accurate methods, models, and algorithms for identifying abnormal behaviors, explore multi-dimensional data fusion complementary detection methods, improve the application scenario consistency and adaptability of the theoretical results of abnormal behavior recognition, and eventually enhance the level of public place crowd safety governance.

Key words: public places, pedestrian abnormal behaviors, recognition methods, wearable sensor, abnormal behavior dataset

CLC Number:

ZHAO Rongyong, WEI Bingyu, ZHU Wenjie, ZHENG Chengyuan, LI Haonan. Overview of recognition methods of pedestrian abnormal behaviors in public places[J]. China Safety Science Journal, 2024, 34(2): 83-93.

Figures/Tables 7

Fig.1

Tab.1

Fig.2

Fig.3

Fig.4

Fig.5

Tab.2

References 68

[1]	KOK V J, LIM M K, CHAN C S. Crowd behavior analysis: a review where physics meets biology[J]. Neurocomputing, 2016, 177: 342-362. doi: 10.1016/j.neucom.2015.11.021
[2]	张萌, 韩豫, 刘泽锋. 深度学习下建筑工人高空安全防护装备检测方法[J]. 中国安全科学学报, 2022, 32(5): 140-146. doi: 10.16265/j.cnki.issn1003-3033.2022.05.1141
	ZHANG Meng, HAN Yu, LIU Zefeng. Detection method of high-altitude safety protective equipment for construction workers based on deep learning[J]. China Safety Science Journal, 2022, 32(5): 140-146. doi: 10.16265/j.cnki.issn1003-3033.2022.05.1141
[3]	张晓平, 纪佳慧, 王力, 等. 基于视频的人体异常行为识别与检测方法综述[J]. 控制与决策, 2022, 37(1): 14-27.
	ZHANG Xiaoping, JI Jiahui, WANG Li, et al. Overview of video based human abnormal behavior recognition and detection methods[J]. Control and Decision, 2022, 37(1): 14-27.
[4]	李建更, 谢海征. 基于姿态估计的人体异常行为识别算法[J]. 北京工业大学学报, 2022, 48(7): 710-720.
	LI Jiangeng, XIE Haizheng. Human abnormal behavior recognition algorithm based on pose estimation[J]. Journal of Beijing University of Technology, 2022, 48(7): 710-720.
[5]	徐涛, 田崇阳, 刘才华. 基于深度学习的人群异常行为检测综述[J]. 计算机科学, 2021, 48(9): 125-134. doi: 10.11896/jsjkx.201100015
	XU Tao, TIAN Chongyango, LIU Caihua. Deep learning for abnormal crowd behavior detection: a review[J]. Computer Science, 2021, 48(9): 125-134. doi: 10.11896/jsjkx.201100015
[6]	郭毅博, 孟文化, 范一鸣, 等. 基于可穿戴传感器数据的人体行为识别数据特征提取方法[J]. 计算机辅助设计与图形学学报, 2021, 33(8): 1246-1253.
	GUO Yibo, MENG Wenhua, FAN Yiming, et al. Wearable sensor data based human behavior recognition: a method of data feature extraction[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(8): 1246-1253.
[7]	张烈平, 匡贞伍, 李昆键, 等. 基于加速度传感器和神经网络的人体活动行为识别[J]. 现代电子技术, 2019, 42(16): 71-74.
	ZHANG Lieping, KUANG Zhenwu, LI Kunjian, et al. Human activity behavior recognition based on acceleration sensor and neural network[J]. Modern Electronics Technique, 2019, 42(16): 71-74.
[8]	陈波, 余秋婷, 陈铁明. 基于传感器人体行为识别深度学习模型的研究[J]. 浙江工业大学学报, 2018, 46(4): 375-381.
	CHEN Bo, YU Qiuting, CHEN Tieming. On deep-learning-model-based sensor activity recognition[J]. Journal of Zhejiang University of Technology, 2018, 46(4): 375-381.
[9]	艾达, 王倩, 樊炜鑫, 等. 智能手机传感器的人体行为识别技术[J]. 西安邮电大学学报, 2020, 25(1): 42-48.
	AI Da, WANG Qian, FAN Weixin, et al. A review of human behavior recognition based on multi-sensor of smart phone[J]. Journal of Xi'an University of Posts and Telecommunications, 2020, 25(1): 42-48.
[10]	SODEMANN A A, ROSS M P, BORGHETTI B J. A review of anomaly detection in automated surveillance[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 2012, 42(6): 1257-1272.
[11]	POPOOLA O P, KEJUN W. Video-based abnormal human behavior recognition: a review[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 2012, 42(6): 865-878.
[12]	MABROUK A B, ZAGROUBA E. Abnormal behavior recognition for intelligent video surveillance systems: a review[J]. Expert Systems with Applications, 2018, 91: 480-491. doi: 10.1016/j.eswa.2017.09.029
[13]	DHIMAN C, VISHWAKARMA D K. A review of state-of-the-art techniques for abnormal human activity recognition[J]. Engineering Applications of Artificial Intelligence, 2019, 77: 21-45. doi: 10.1016/j.engappai.2018.08.014
[14]	LI Weixin, MAHADEVAN V, VASCONCELOS N. Anomaly detection and localization in crowded scenes[J]. IEEE Trans Pattern Anal Mach Intell, 2014, 36(1): 18-32. doi: 10.1109/TPAMI.2013.111
[15]	杜鉴豪, 许力. 基于区域光流特征的异常行为检测[J]. 浙江大学学报:工学版, 2011, 45(7): 1161-1166.
	DU Jianhao, XU Li. Abnormal behavior detection based on regional optical flow[J]. Journal of Zhejiang University: Engineering Science, 2011, 45(7): 1161-1166.
[16]	LU Cewu, SHI Jianping, JIA Jiaya. Abnormal event detection at 150 fps in Matlab[C]. 2013 IEEE International Conference on Computer Vision, 2013: 2720-2727.
[17]	LIU Wen, LUO Weixin, LIAN Dongze, et al. Future frame prediction for anomaly detection: a new baseline[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 6536-6545.
[18]	CHATHURAMALI K G M, RAMASINGHE S, RODRIGO R. Abnormal activity recognition using spatio-temporal features[C]. The 7^th International Conference on Information and Automation for Sustainability, 2014: 1-5.
[19]	SULTANI W, CHEN C, SHAH M. Real-world anomaly detection in surveillance videos[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 18-23.
[20]	LIU Shuo, XUE Hao, XU Chunjie, et al. Abnormal behavior detection based on the motion-changed rules[C]. 2020 15^th IEEE International Conference on Signal Processing (ICSP), 2020: 146-149.
[21]	KRATZ L, NISHINO K. Anomaly detection in extremely crowded scenes using spatio-temporal motion pattern models[C]. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009: 20-25.
[22]	CONG Yang, YUAN Junsong, LIU Ji. Sparse reconstruction cost for abnormal event detection[C]. 2011 IEEE Conference on Computer Vision and Pattern Recognition, 2011: 20-25.
[23]	LI Ce, HAN Zhenjun, YE Qixiang, et al. Visual abnormal behavior detection based on trajectory sparse reconstruction analysis[J]. Neurocomputing, 2013, 119: 94-100. doi: 10.1016/j.neucom.2012.03.040
[24]	马露, 裴伟, 朱永英, 等. 基于深度学习的跌倒行为识别[J]. 计算机科学, 2019, 46(9): 106-112.
	MA Lu, PEI Wei, ZHU Yongying, et al. Fall action recognition based on deep learning[J]. Computer Science, 2019, 46(9): 106-112. doi: 10.11896/j.issn.1002-137X.2019.09.014
[25]	杨晨晨, 马春梅, 朱金奇. 基于智能手机的跌倒行为识别算法研究[J]. 计算机工程, 2019, 45(2): 178-183. doi: 10.19678/j.issn.1000-3428.0048883
	YANG Chenchen, MA Chunmei, ZHU Jinqi. Study of fall behavior identification algorithm based on smartphone[J]. Computer Engineering, 2019, 45(2): 178-183. doi: 10.19678/j.issn.1000-3428.0048883
[26]	桑海峰, 陈禹, 何大阔. 基于整体特征的人群聚集和奔跑行为检测[J]. 光电子·激光, 2016, 27(1): 52-60.
	SANG Haifeng, CHEN Yu, HE Dakuo, et al. Crowd gathering and running behavior detection based on overall features[J]. Journal of Optoelectronics·Laser, 2016, 27(1): 52-60.
[27]	何鹏, 陈跃跃, 扈啸. 基于智能手表加速度传感器的人体行为识别[J]. 电脑与信息技术, 2015, 23(5): 6-8,33.
	HE Peng, CHEN Yueyue, HU Xiao. Human behavior recognition based on the accelerator sensor of intelligent watch[J]. Computer and Information Technology, 2015, 23(5): 6-8, 33.
[28]	王冰, 康增建, 吕晓军. 铁路客运站视频监控系统中的行人逆行异常事件检测算法研究[J]. 铁路计算机应用, 2012, 21(4): 19-22.
	WANG Bing, KANG Zengjian, LYU Xiaojun. Research on algorithm of abnormal event detection to pedesorian opposing flow in video monitoring system of railway passenger station[J]. Railway Computer Application, 2012, 21(4): 19-22.
[29]	柏万胜, 孙鹏, 郎宇博, 等. 复杂动态背景下视频中逆行检测技术[J]. 警察技术, 2023(3): 44-47.
[30]	MO Xuan, MONGA V, BALA R, et al. Adaptive sparse representations for video anomaly detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(4): 631-645. doi: 10.1109/TCSVT.2013.2280061
[31]	GUO Shuqiang, BAI Qianlong, GAO Song, et al. An analysis method of crowd abnormal behavior for video service robot[J]. IEEE Access, 2019, 7: 169 577-169 585.
[32]	CHANDRAKALA S, DEEPAK K, VIGNESH L K P. Bag-of-Event-Models based embeddings for detecting anomalies in surveillance videos[J]. Expert Systems with Applications, 2022, 190: 1-10.
[33]	ROSHTKHARI M J, LEVINE M D. Online dominant and anomalous behavior detection in videos[C]. 2013 IEEE Conference on Computer Vision and Pattern Recognition, 2013: 2611-2618.
[34]	DORETTO G, CHIUSO A, WU Y N, et al. Dynamic textures[J]. International Journal of Computer Vision, 2003, 51: 91-109. doi: 10.1023/A:1021669406132
[35]	SINGH K, RAJORA S, VISHWAKARMA D K, et al. Crowd anomaly detection using aggregation of ensembles of fine-tuned ConvNets[J]. Neurocomputing, 2020, 371: 188-98. doi: 10.1016/j.neucom.2019.08.059
[36]	CAO Zhe, SIMON T, WEI Shihen, et al. Realtime multi-person 2d pose estimation using part affinity fields[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: 1302-1310.
[37]	FANG Haoshu, LI Jiefeng, TANG Hongyang, et al. AlphaPose: whole-body regional multi-person pose estimation and tracking in real-time[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(6): 7157-7173. doi: 10.1109/TPAMI.2022.3222784 pmid: 37145952
[38]	WANG Pichao, LI Wanqing, LI Chuankun, et al. Action recognition based on joint trajectory maps with convolutional neural networks[J]. Knowledge-Based Systems, 2018, 158: 43-53. doi: 10.1016/j.knosys.2018.05.029
[39]	BO Li, DAI Yuchao, CHENG Huahui, et al. Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN[C]. 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2017: 601-604.
[40]	WANG Hongsong, WANG Liang. Learning robust representations using recurrent neural networks for skeleton based action classification and detection[C]. 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2017: 591-596.
[41]	YAN Sijie, XIONG Yuanjun, LIN Dahua. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]. AAAI Conference on Artificial Intelligence, 2018: 7444-7452.
[42]	YANG C L, SETYOKO A, TAMPUBOLON H, et al. Pairwise adjacency matrix on spatial temporal graph convolution network for skeleton-based two-person interaction recognition[C]. 2020 IEEE International Conference on Image Processing (ICIP), 2020: 2166-2170.
[43]	FENG Dong, WU Zhongcheng, ZHANG Jun, et al. Multi-scale spatial temporal graph neural network for skeleton-based action recognition[J]. IEEE Access, 2021, 9: 58 256-58 265.
[44]	ZHENG Yin, ZHANG Dongping, YANG Li, et al. Fall detection and recognition based on GCN and 2D Pose[C]. 2019 6^th International Conference on Systems and Informatics (ICSAI), 2019: 558-562.
[45]	刘耀, 焦双健. St-Gcn在建筑工人不安全动作识别中的应用[J]. 中国安全科学学报, 2022, 32(4): 30-35. doi: 10.16265/j.cnki.issn1003-3033.2022.04.005
	LIU Yao, JIAO Shuangjian. Application of ST-GCN in unsafe action identification of construction workers[J]. China Safety Science Journal, 2022, 32(4):30-35. doi: 10.16265/j.cnki.issn1003-3033.2022.04.005
[46]	NAR R, SINGAL A, KUMAR P. Abnormal activity detection for bank ATM surveillance[C]. 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2016: 2042-2046.
[47]	PANG Guansong, SHEN Chunhua, CAO Longbing, et al. Deep learning for anomaly detection[J]. ACM Computing Surveys, 2021, 54(2): 1-38.
[48]	HASAN M, CHOI J, NEUMANN J, et al. Learning temporal regularity in video sequences[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016: 733-742.
[49]	GONG Dong, LIU Lingqiao, LE Vuong, et al. Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection[C]. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019: 1705-1714.
[50]	LI Jing, HUANG Qingwang, DU Yingjun, et al. Variational abnormal behavior detection with motion consistency[J]. IEEE Trans Image Process, 2022, 31: 275-286. doi: 10.1109/TIP.2021.3130545
[51]	CHONG Y S, TAY Y H. Abnormal event detection in videos using spatiotemporal autoencoder[M]. Berlin: Advances in Neural Networks, 2017:189-196.
[52]	RAVANBAKHSH M, NABI M, SANGINETO E, et al. Abnormal event detection in videos using generative adversarial nets[C]. 2017 IEEE International Conference on Image Processing (ICIP), 2017:1577-1581.
[53]	ZHAO Yiru, DENG Bing, SHEN Chen, et al. Spatio-temporal autoencoder for video anomaly detection[C]. 25^th ACM International Conference on Multimedia, 2017: 1933-1941.
[54]	RUMELHART D E, HINTON G E, WILLIAMS R J. Learning representations by back-propagating errors[J]. Nature, 1986, 323: 533-536. doi: 10.1038/323533a0
[55]	LIU Wen, LUO Weixin, LI Zhengxin, et al. Margin learning embedded prediction for video anomaly detection with a few anomalies[C]. International Joint Conference on Artificial Intelligence, 2019: 3023-3030.
[56]	WANG Xuanzhao, CHE Zhengping, JIANG Bo, et al. Robust unsupervised video anomaly detection by multipath frame prediction[J]. IEEE Trans Neural Netw Learn Syst, 2022, 33(6): 2301-2312. doi: 10.1109/TNNLS.2021.3083152
[57]	PANG Guansong, SHEN Chunhua, VAN DEN HENGEL A. Deep anomaly detection with deviation networks[C]. 25^th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019: 353-362.
[58]	PANG Guansong, YAN Cheng, SHEN Chunhua, et al. Self-trained deep ordinal regression for end-to-end video anomaly detection[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 12 170-12 179.
[59]	肖进胜, 申梦瑶, 江明俊, 等. 融合包注意力机制的监控视频异常行为检测[J]. 自动化学报, 2022, 48(12): 2951-2959.
	XIAO Jinsheng, SHEN Mengyao, JIANG Mingjun, et al. Abnormal behavior detection algorithm with video-bag attention mechanism in surveillance video[J]. Acta Automatica Sinica, 2022, 48(12): 2951-2959.
[60]	GEORGESCU M I, BǍRBǍLǍU A, IONESCU R T, et al. Anomaly detection in video via self-supervised and multi-task learning[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021: 12 737-12 747.
[61]	TANG Yao, ZHAO Lin, ZHANG Shanshan, et al. Integrating prediction and reconstruction for anomaly detection[J]. Pattern Recognition Letters, 2020, 129: 123-130. doi: 10.1016/j.patrec.2019.11.024
[62]	SABOOR A, KASK T, KUUSIK A, et al. Latest research trends in gait analysis using wearable sensors and machine learning: a systematic review[J]. IEEE Access, 2020, 8: 167 830-167 864.
[63]	WU Donghui, WANG Zhelong, CHEN Ye, et al. Mixed-kernel based weighted extreme learning machine for inertial sensor based human activity recognition with imbalanced dataset[J]. Neurocomputing, 2016, 190: 35-49. doi: 10.1016/j.neucom.2015.11.095
[64]	MOHAMMADIAN R N, VAN LAARHOVEN T, FURLANELLO C, et al. Novelty detection using deep normative modeling for imu-based abnormal movement monitoring in parkinson's disease and autism spectrum disorders[J]. Sensors (Basel), 2018, 18(10): 3533-3550. doi: 10.3390/s18103533
[65]	周晓芳, 吴松洋, 韩玮, 等. 管控区域人员异常行为检测系统与方法研究[J]. 计算机应用与软件, 2022, 39(5): 92-97.
	ZHOU Xiaofang, WU Songyang, HAN Wei, et al. Detection system and method for abnormal behavior of personnel in control areas[J]. Computer Applications and Software, 2022, 39(5): 92-97.
[66]	KIPRIJANOVSKA I, GJORESKI H, GAMS M. Detection of gait abnormalities for fall risk assessment using wrist-worn inertial sensors and deep learning[J]. Sensors (Basel), 2020, 20(18): 5373-5394. doi: 10.3390/s20185373
[67]	CHAKRABORTY J, NANDY A. Discrete wavelet transform based data representation in deep neural network for gait abnormality detection[J]. Biomedical Signal Processing and Control, 2020, 62: 102 076-102 088.
[68]	CAVIAR. Context aware vision using image-based active recognition[EB/OL]. (2005-09-30). https://homepages.inf.ed.ac.uk/rbf/CAVIAR/.

场景	异常行为	行为类型	相关文献
校园人行道	奔跑、跌倒、追逐等	危害行为	[14,16-17]
场景	异常行为	行为类型	相关研究文献
(高速) 公路	行走、骑行、奔跑等	不合群行为	[18-19]
火车站、地铁站	跌倒、奔跑等	危害行为	[20⇓-22]
超市、商店	偷窃等	违规行为	[16,23]

类别	方法	年份	数据集	性能/%
人工设计的特征	STVs描述符+聚类^[33]	2013	UCSD Ped1	ERR-15.00%(f), 29.00%(p)
	轨迹特征+稀疏重构^[23]	2013	CAVIAR^[68]	Accuracy-90.42%
	光流特征+轮廓特征+SVM^[18]	2014	UMN	Accuracy-95.83%
人体骨架	Openpose+ST-GCN^[44]	2019	自建数据集	Accuracy-100.00%
	Alphapose+ST-GCN^[45]	2022	自建数据集	Accuracy-98.48%
	Kinect 3D+逻辑回归^[46]	2016	自建数据集	—
RGB 图像	帧重构+ST-AEs^[51]	2017	CUHK, UCSD Ped1,Ped2	AUC-80.30%, 89.90%, 87.40%
	帧重构+MemAE^[49]	2019	UCSD Ped1, CUHK, SH.Tech	AUC-94.10%, 83.30%, 71.20%
	帧预测+U-Net^[17]	2018	CUHK, UCSD Ped1,Ped2	AUC-84.90%, 83.1%, 95.40%
	帧预测+边际学习^[55]	2019	CUHK, SH.Tech	AUC-92.80%, 76.80%
	端对端+深度多实例排序^[19]	2018	自建数据集	AUC-75.41%
	端对端+自训练学习^[58]	2020	UCSD Ped1,Ped2, UMN	AUC-71.70%, 83.20%, 97.25%
	自监督+多任务学习^[60]	2021	CUHK, UCSD Ped2, SH.Tech	AUC-92.80%, 99.8%, 92.80%
	帧重构+帧预测^[61]	2020	USCD Ped1,Ped2, CUHK, SH.Tech	AUC-82.60%, 96.20%,83,70%, 71.50%
可穿戴传感器	惯性传感器+SVM^[65]	2022	自建数据集	F₁-96.50%
可穿戴传感器	惯性传感器+CNN^[67]	2020	自建数据集	Accuracy -96.40%