An image-text multimodal intelligent identification method for construction safety hazards in hydropower engineering

doi:10.16265/j.cnki.issn1003-3033.2026.03.0881

Abstract

Abstract:

To address the problems of incomplete unimodal feature representation and low image-text fusion efficiency in construction safety hazard identification for hydropower projects, an intelligent image-text multimodal intelligent identification method was proposed. First, 12 categories of construction safety hazards were defined according to hydropower construction characteristics, and an image-text multimodal dataset was established. Second, bidirectional encoder representations from transformers (BERT) model and vision transformer (ViT) model were employed to extract hazard text and image features respectively. GFN was then introduced to dynamically adjust the contribution of image and text features and capture cross-modal correlated feature information, while a multi-layer perceptron was used to improve classification accuracy. Comparative experiments were conducted to verify the model's accuracy and reliability. The results show the method optimizes the contribution of multimodal features by enhancing identification stability. The multimodal hazard identification accuracy reaches 84.99%, representing an improvement of 1.73% over the text-based model and 12.24% over the image-based model.. The proposed approach outperforms existing benchmark models in hazard classification and improves the robustness of intelligent hazard identification.

Key words: hydropower project, construction safety hazard, multimodal, gated fusion network(GFN), intelligent identification

CLC Number:

X948

NIE Benwu, CHEN Shu, CHEN Yun, TIAN Xueqi, CAO Kunyu, LI Zhi. An image-text multimodal intelligent identification method for construction safety hazards in hydropower engineering[J]. China Safety Science Journal, 2026, 36(3): 104-112.

Figures/Tables 11

Table 1

Table 2

Fig.1

Fig.2

Table 3

Table 4

Fig.3

Table 5

Table 6

Fig.4

Table 7

References 16

[1]	卢冰, 陈述, 曹坤煜, 等. 水电工程施工安全隐患类别辅助校正方法[J]. 水力发电学报, 2025, 44(4): 42-49.
	LU Bing, CHEN Shu, CAO Kunyu, et al. Auxiliary correction methods for categories of potential safety hazards in hydropower project construction[J]. Journal of Hydroelectric Engineering, 2025, 44(4): 42-49.
[2]	陈述, 王典学, 杨应柳, 等. 水电工程施工安全隐患语义匹配模型[J]. 中国安全科学学报, 2024, 34(12): 40-47. doi: 10.16265/j.cnki.issn1003-3033.2024.12.0795
	CHEN Shu, WANG Dianxue, YANG Yingliu, et al. Semantic matching model of potential safety hazards in hydroelectric project construction[J]. China Safety Science Journal, 2024, 34(12): 40-47. doi: 10.16265/j.cnki.issn1003-3033.2024.12.0795
[3]	杨阳蕊, 潘世峰, 刘雪梅, 等. 多模态知识图谱与大模型协同的水利工程风险应对决策推荐[J]. 水利学报, 2025, 56(4): 519-530.
	YANG Yangrui, PAN Shifeng, LIU Xuemei, et al. Multimodal knowledge graph collaborated with large model fordecision recommendation of water projects risk response[J]. Journal of Hydraulic Engineering, 2025, 56(4): 519-530.
[4]	张泽辉, 张乾隆, 徐晓滨, 等. 基于视觉的工人高处攀爬不安全行为识别模型[J]. 中国安全科学学报, 2025, 35(2): 144-151. doi: 10.16265/j.cnki.issn1003-3033.2025.02.0278
	ZHANG Zehui, ZHANG Qianlong, XU Xiaobin, et al. Unsafe behavior recognition model of high climbing workers based on vision[J]. China Safety Science Journal, 2025, 35(2): 144-151. doi: 10.16265/j.cnki.issn1003-3033.2025.02.0278
[5]	ZHONG Botao, SHEN Luoxin, PAN Xing, et al. Visual attention framework for identifying semantic information from construction monitoring video[J]. Safety Science, 2023, 163:DOI:10.1016/J.SSCI.2023.106122.
[6]	FANG Weili, LOVE P E D, DING Lieyun, et al. Computer vision and deep learning to manage safety in construction: matching images of unsafe behavior and semantic rules[J]. IEEE Transactions on Engineering Management, 2023, 70:4 120-4 132. doi: 10.1109/TEM.2021.3093166
[7]	陈述, 张超, 陈云, 等. 基于命名实体识别的水电工程施工安全规范实体识别模型[J]. 中国安全科学学报, 2024, 34(9): 19-26. doi: 10.16265/j.cnki.issn1003-3033.2024.09.0008
	CHEN Shu, ZHANG Chao, CHEN Yun, et al. Model of identifying entities of safety specification for hydropower engineering construction[J]. China Safety Science Journal, 2024, 34(9): 19-26. doi: 10.16265/j.cnki.issn1003-3033.2024.09.0008
[8]	周佳一, 郑霞忠, 田丹, 等. 水电工程施工安全隐患多标签文本智能分类方法[J]. 水力发电学报, 2024, 43(11): 114-124.
	ZHOU Jiayi, ZHENG Xiazhong, TIAN Dan, et al. Multi-label text intelligent classification method for construction safety hazards of hydropower projects[J]. Journal of Hydroelectric Engineering, 2024, 43(11): 114-124.
[9]	LIU Jiajing, FANG Weili, PETER E D L, et al. Detection and location of unsafe behaviour in digital images: a visual grounding approach[J]. Advanced Engineering Informatics, 2022, 53:DOI:10.1016/j.aei.2022.101688.
[10]	田丹, 许仁乐, 邵波, 等. 强化特征表达的水电工程施工安全隐患自动辨识方法[J/OL]. 河海大学学报:自然科学版: 1-11.[2025-05-28]. https://kns.cnki.net/kcms2/article.html.
	TIAN Dan, XU Renle, SHAO Bo, et al. Automatic identification method for construction safety hazards in hydropower engineering with enhanced feature expression[J/OL]. Journal of Hohai University: Natural Sciences Edition: 1-11.[2025-05-28]. https://kns.cnki.net/kcms2/article.html.
[11]	陈岩松, 张乐, 张雷瀚, 等. 基于跨模态注意力和门控单元融合网络的多模态情感分析方法[J]. 数据分析与知识发现, 2024, 8(7): 67-76. doi: 10.11925/infotech.2096-3467.2023.0591
	CHEN Yansong, ZHANG Le, ZHANG Leihan, et al. Multimodal sentiment analysis method based on cross-modal attention and gated unit fusion network[J]. Data Analysis and Knowledge Discovery, 2024, 8(7): 67-76. doi: 10.11925/infotech.2096-3467.2023.0591
[12]	WANG Yiheng, XIAO Bo, BOUFERGUENE A, et al. Proactive safety hazard identification using visual-text semantic similarity for construction safety management[J]. Automation in Construction, 2024, 166:DOI:10.1016/j.autcon.2024.105602.
[13]	KIM H, YI J S. Image generation of hazardous situations in construction sites using text-to-image generative model for training deep neural networks[J]. Automation in Construction, 2024, 166:DOI:10.1016/j.autcon.2024.105615.
[14]	AREVALO J, SOLORIO T, MONTES M, et al. Gated multimodal networks[J]. Neural Computing and Applications, 2020, 32(14): 10 209-10 228. doi: 10.1007/s00521-019-04559-1
[15]	CAO Kunyu, CHEN Shu, CHEN Yun, et al. Decision analysis of safety risks pre-control measures for falling accidents in mega hydropower engineering driven by accident case texts[J]. Reliability Engineering & System Safety, 2025,261: DOI:10.1016/j.ress.2025.111120.
[16]	TIAN Dan, LIU Hao, CHEN Shu, et al. Human error analysis for hydraulic engineering: comprehensive system to reveal accident evolution process with text knowledge[J]. Journal of Construction Engineering and Management, 2022, 148(9): DOI:10.1061/(ASCE)CO.1943-7862.0002366.

隐患类型	隐患特征
高处坠落	存在从坠落高度基准面2 m以上(含2 m)处坠落的隐患
触电事故	电流有通过人体造成伤害或死亡的隐患
物体打击	失控物体存在(如工具、材料)击中人体造成伤害的隐患
起重伤害	存在因吊具、吊物等引发的伤害的隐患
机械伤害	存在机械设备运动部件(如齿轮、皮带)导致的夹击、切割等伤害的隐患
火灾事故	易燃物堆积、电气短路或明火管理不当等发生燃烧的隐患
车辆伤害	存在机动车辆(如自卸车、挖掘机)碰撞、碾压导致伤害的隐患
爆炸事故	存在压力容器、危险化学品等因瞬间能量释放造成破坏的隐患
坍塌事故	存在土方、模板或建筑结构塌落的隐患
设备损坏	生产设备非正常损毁和机械故障、超负荷运行或维护不足引发连锁的隐患
文明施工	因现场管理混乱(如杂物堆放、通道堵塞)等脏乱差导致的间接隐患
其他事故	未涵盖在上述分类中的其他风险或隐患

序号	图片描述	隐患描述	隐患类型	隐患照片
1	挖掘机的机械臂停放在空中	挖掘机机械臂在停止作业时未放置地面	起重伤害
2	一个打开的电源箱, 未作接地连接	二级配电箱一闸多机且电缆出线未作接地连接	触电事故
3	一个没有连接扣件的脚手架	边坡支护排架剪刀撑钢管与立杆相交处未使用扣件连接	坍塌事故
︙	︙	︙	︙	︙
2282	高处作业人员未系挂安全带	排导槽上游高处作业人员未系挂安全带	高处坠落

隐患类型	训练集	验证集	测试集	总数	占比/%	隐患类型	训练集	验证集	测试集	总数	占比/%
爆炸事故	57	16	8	81	4	起重伤害	85	24	12	121	5
车辆伤害	15	4	2	22	1	设备损坏	20	6	3	29	1
触电事故	505	144	72	721	32	坍塌事故	41	12	6	59	3
高处坠落	192	55	27	274	12	文明施工	246	70	35	352	15
火灾事故	205	59	29	293	13	物体打击	44	13	6	63	3
机械伤害	41	12	6	58	3	正常照片	700	200	100	1 000	—
其他事故	146	42	21	209	9	合计	2 297	656	328	3 282	100

模型名称	输入类型	优点	缺点	适用领域
BERT^[7]	文本	双向上下文预训练	计算资源消耗大、生成能力弱	文本分类、命名实体识别、阅读理解
ViT^[9]	图像	序列化+全局自注意力机制	计算复杂度高	大规模图像分类/检测
BERT-ViT^[12]	图像-文本	跨模态交互(交叉注意力机制/特征融合)	训练复杂、数据稀缺、模态不平衡	图文交互任务

模型	准确率	精确率	召回率	F₁值
文本模型	84.49	84.43	69.96	84.21
图像模型	76.58	76.82	57.61	75.32
图文模型	85.30	87.92	70.44	84.45
门控融合图文模型	85.95	85.98	71.01	84.99