基于BERT-BiLSTM-CRF的隧道施工安全领域命名实体识别

doi:10.16265/j.cnki.issn1003-3033.2024.12.0713

摘要/Abstract

摘要：

为解决隧道施工安全领域传统命名实体识别(NER)方法存在的实体边界模糊、小样本学习困难、特征信息提取不够全面准确等问题,提出一种基于变换器的双向编码器表征(BERT)-双向长短时记忆(BiLSTM)网络-条件随机场 (CRF)模型的隧道施工事故文本实体识别方法。首先,利用BERT模型将隧道施工事故文本编码得到蕴含语义特征的词向量;然后,将BERT模型训练后输出的词向量输入BiLSTM模型进一步获取隧道施工事故文本的上下文特征并进行标签概率预测;最后,利用CRF层的标注规则的约束,修正BiLSTM模型的输出结果,得到最大概率序列标注结果,从而实现对隧道施工事故文本标签的智能分类。将该模型与其他4种常用的传统NER模型在隧道施工安全事故语料数据集上进行对比试验,试验结果表明:BERT-BiLSTM-CRF模型的识别准确率、召回率和F₁值分别达到88%、89%和88%,实体识别效果优于其他基准模型。利用所建立的NER模型识别实际隧道施工事故文本中的实体,验证了其在隧道施工安全领域中的应用效果。

关键词: 变换器的双向编码器表征(BERT), 双向长短时记忆(BiLSTM)网络, 条件随机场(CRF), 隧道施工, 安全领域, 命名实体识别(NER), 深度学习

Abstract:

To solve the problems existing in the traditional NER methods in the domain of tunnel construction safety, such as fuzzy entity boundary, difficulty in small-sample learning, and insufficiently comprehensive extraction of feature information, an entity recognition method for tunnel construction accident text based on the BERT-BiLSTM-CRF model was proposed. Firstly, the BERT model was used to encode the tunnel construction accident text to obtain word vectors containing semantic features. Then, the word vectors output after the training of the BERT model were input into the BiLSTM model to further obtain the context feature of the tunnel construction accident text and conduct label probability prediction. Finally, by utilizing the constraints of the annotation rules of the CRF layer, the output result of the BiLSTM model was corrected, and the maximum probability sequence annotation result was obtained, so as to realize the intelligent classification of the labels of the tunnel construction accident texts. Comparative experiments were conducted between this model and other four commonly used traditional NER models on the tunnel construction safety accident corpus dataset. The results show that the recognition accuracy rate, recall rate and F₁ value of the BERT-BiLSTM-CRF model are 88%, 89% and 88% respectively, and the entity recognition effect is better than other benchmark models. By using the established NER model to recognize the entities in the actual tunnel construction accident texts, its application effect in the domain of tunnel construction safety is verified.

Key words: bidirectional encoder representations from transformers (BERT), bidirectional long short-term memory(BiLSTM), conditional random fields (CRF), tunnel construction, safety field, named entity recognition (NER), deep learning

中图分类号:

X928事故调查与分析(工伤事故分析与预防)

张念, 周彩凤, 万飞, 刘非, 王耀耀, 徐栋梁. 基于BERT-BiLSTM-CRF的隧道施工安全领域命名实体识别[J]. 中国安全科学学报, 2024, 34(12): 56-63.

ZHANG Nian, ZHOU Caifeng, WAN Fei, LIU Fei, WANG Yaoyao, XU Dongliang. Tunnel construction safety domain named entity recognition based on BERT-BiLSTM-CRF[J]. China Safety Science Journal, 2024, 34(12): 56-63.

图/表 10

图1

图2

图3

表1

表2

表3

图4

图5

表4

图6

参考文献 18

[1]	薛亚东, 董宏鑫, 李彦杰. 山岭公路隧道施工安全风险评估理论体系[J]. 天津大学学报:自然科学与工程技术版, 2019, 52(增1):84-91.
	XUE Yadong, DONG Hongxin, LI Yanjie. Theoretical System for the safety risk assessment of mountain tunnel construction[J]. Journal of Tianjin University: Science and Technology, 2019, 52(S1): 84-91.
[2]	周泽林, 张凯, 张恒, 等. 属性识别理论下的岩溶隧道地表塌陷风险评价[J]. 中国安全科学学报, 2022, 32(11): 105-112. doi: 10.16265/j.cnki.issn1003-3033.2022.11.1867
	ZHOU Zelin, ZHANG Kai, ZHANG Heng, et al. Risk assessment of surface subsidence in karst tunnels under attribute recognition theory[J]. China Safety Science Journal, 2022, 32(11): 105-112. doi: 10.16265/j.cnki.issn1003-3033.2022.11.1867
[3]	康怡琳, 孙璐冰, 朱容波, 等. 深度学习中文命名实体识别研究综述[J]. 华中科技大学学报:自然科学版, 2022, 50(11):44-53.
	KANG Yilin, SUN Lubing, ZHU Rongbo, et al. Survey on Chinese named entity recognition with deep learning[J]. Journal of Huazhong University of Science and Technology: Nature Science Edition, 2022, 50(11): 44-53.
[4]	高翔, 王石, 朱俊武, 等. 命名实体识别任务综述[J]. 计算机科学, 2023, 50(增1):26-33.
	GAO Xiang, WANG Shi, ZHU Junwu, et al. Overview of Named Entity Recognition Tasks[J]. Computer Science, 2023, 50(S1): 26-33.
[5]	HAMMER B. Learning with recurrent neural networks[J]. Assembly Automation, 2001, 21(2): 178-183.
[6]	CUN Y L, BOSER B, DENKER J S, et al. Handwritten digit recognition with a back-propagation network[J]. Advances in Neural Information Processing Systems, 1990, 2(2): 396-404.
[7]	BARCZ A, SZYMAŃSKI Z, JANKOWSKI S. Implementation aspects of graph neural networks[J]. Proceedings of SPIE-the International Society for Optical Engineering, 2013, 8903(12): 405-408.
[8]	SUNDERMEYER M, NEY H, SCHLUTER R. From feedforward to recurrent LSTM neural networks for language modeling[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015, 23(3): 517-529.
[9]	DIESNER J, CARLEY K M. Conditional random fields for entity extraction and ontological text coding[J]. Computational and Mathematical Organization Theory, 2008, 14(3): 248-262.
[10]	LI Fei, JIN Yonghao, LIU Weisong, et al. Fine-tuning bidirectional encoder representations from transformers (BERT)-based models on large-scale electronic health record notes: an empirical study[J]. JMIR Medical Informatics, 2019, 7(3):DOI: 10.2196/14830.
[11]	高国忠, 李宇, 华远鹏, 等. 基于BERT-BiLSTM-CRF模型的油气领域命名实体识别[J]. 长江大学学报:自然科学版, 2024, 21(1):57-65.
	GAO Guozhong, LI Yu, HUA Yuanpeng, et al. Named entity recognition in oil and gas field based on the BERT-BiLSTM-CRF model[J]. Journal of Yangtze University: Natural Science Edition, 2024, 21(1):57-65.
[12]	王莉, 王建平, 许娜, 等. 基于知识图谱的地铁工程事故知识建模与分析[J]. 土木工程与管理学报, 2019, 36(5):109-114,122.
	WANG Li, WANG Jianping, XU Na, et al. Knowledge graph-based metro engineering accidents knowledge modeling and analysis[J]. Journal of Civil Engineering and Management, 2019, 36(5): 109-114,122.
[13]	胡珉, 胡星辰, 宋云, 等. 基于知识图谱的盾构施工辅助决策系统设计和开发[J]. 现代隧道技术, 2023, 60(1):66-75.
	HU Min, HU Xingchen, SONG Yun, et al. Design and development of the shield construction assistant decision-making system based on the knowledge graph[J]. Modern Tunnelling Technology, 2023, 60(1): 66-75.
[14]	张鹏翔. 多维字符特征表示的铁路设备事故信息抽取方法[J]. 中国安全科学学报, 2022, 32(6): 109-114. doi: 10.16265/j.cnki.issn1003-3033.2022.06.2732
	ZHANG Pengxiang. Information extraction method for railway equipment accidents based on multi-dimensional character feature representation[J]. China Safety Science Journal, 2022, 32(6): 109-114. doi: 10.16265/j.cnki.issn1003-3033.2022.06.2732
[15]	常弘. 数据驱动的地铁施工安全风险评估与应对研究[D]. 徐州: 中国矿业大学, 2023.
	CHANG Hong. Research on data-driven safety risk assessment and response in metro construction[D]. Xuzhou: China University of Mining and Technology, 2023.
[16]	JATNIKA D, BIJAKSANA A M, SURYANI A A. Word2Vec model analysis for semantic similarities in English words[J]. Procedia Computer Science, 2019, 157: 160-167.
[17]	YUAN Wei, HU Fei, LU Liangfu. A new non-adaptive optimization method: stochastic gradient descent with momentum and difference[J]. Applied Intelligence, 2021, 52(4): 1-15.
[18]	夏成魁, 李少波. 基于BERT-BiLSTM-MHA-CRF的中文命名实体识别方法[J]. 计算机与数字工程, 2023, 51(9):2087-2091,2 102.
	XIA Chengkui, LI Shaobo. Chinese named entity recognition method based on BERT-BiLSTM-MHA-CRF model[J]. Computer & Digital Engineering, 2023, 51(9): 2087-2091,2 102.

试验环境名称	配置
操作系统	64位Windows10系统
编程语言	Python3.7
深度学习框架	Pytorch1.10.2+cu113
GPU	NVIDIA GeForce GTX 1650
GPU加速器	cuDA 11.3,CUDNN 8.0
内存/GB	16
代码编写平台	PyCharm Professional Edition 2021.2.4

参数名称	参数值	参数注释
max_seq_len	256	最大序列长度
lstm_hidden	128	LSTM隐藏层单元
dropout	0.5	随机失活率
learning rate	3×10^-5	学习率
optimizer	Adam	优化器
batch_size	6	每批次训练样本数

模型	P	R	F₁
BERT	0.87	0.87	0.87
BiLSTM-CRF	0.82	0.78	0.80
BiLSTM	0.70	0.70	0.70
BERT-CRF	0.88	0.88	0.88
BERT-BiLSTM-CRF	0.88	0.89	0.88

实体类别	P	R	F₁
事故地点	0.81	0.86	0.86
事故时间	0.91	0.94	0.92
事故类型	0.84	0.95	0.84
伤亡损失	0.88	0.90	0.89
隧道名称	0.93	0.93	0.93