融合多层注意力机制与BiLSTM的知识图谱补全算法研究

doi:10.11871/jfdc.issn.2096-742X.2023.03.009

数据与计算发展前沿 ›› 2023, Vol. 5 ›› Issue (3): 123-137.

CSTR: 32002.14.jfdc.CN10-1649/TP.2023.03.009

doi: 10.11871/jfdc.issn.2096-742X.2023.03.009

融合多层注意力机制与BiLSTM的知识图谱补全算法研究

张晓帆(),孙海春,李欣^*()

中国人民公安大学，信息网络安全学院，北京 100038

收稿日期:2022-05-17 出版日期:2023-06-20 发布日期:2023-06-21
通讯作者: *李欣（E-mail: lixin@ppsuc.edu.cn）
作者简介:张晓帆，中国人民公安大学，硕士研究生，主要研究方向为知识库、知识图谱、自然语言处理。
本文主要承担工作为文献调研、论文撰写和设计，修改全文。
ZHANG Xiaofan is a master’s student at the People’s Public Security University of China. His research interests include knowledge base, knowledge graph, and natural language processing.
In this paper, he is responsible for literature research, thesis design, writing and modification.
E-mail: 201621440015@stu.ppsuc.edu.cn|李欣，中国人民公安大学，副教授，博士，主要研究方向为网络安全、视频网络、人工智能等。
在本文中负责论文修改与指导相关工作。
LI Xin, Ph.D., is an Associate Professor at the People’s Public Security University of China. He has been engaged in research on Cyber Security, Big Data, and Artificial Intelligence.
In this paper, she is mainly responsible for the revision and guidance of the paper.
E-mail: lixin@ppsuc.edu.cn
基金资助:
国家自然科学基金“面向视频监控的跨时空行人轨迹分析方法”(62076246);公安部技术研究计划项目(2020JSYJC22)

Hierarchical Attention-Based Bidirectional Long Short-Term Memory Networks for Knowledge Graph Completion

ZHANG Xiaofan(),SUN Haichun,LI Xin^*()

Information and Network Security College, People’s Public University of China, Beijing 100038, China

Received:2022-05-17 Online:2023-06-20 Published:2023-06-21

摘要/Abstract

摘要：

【目的】 针对目前大多数知识图谱补全算法无法兼顾局部与全局特征的问题，本文提出一种对实体间的关系路径进行层级划分，并利用双向长短期记忆网络和多层注意力机制进行特征提取的算法，以对知识图谱进行补全。【方法】 首先，结合关系路径上的实体类型和关系得到关系路径序列的向量表示；然后，利用多层注意力机制和双向长短期记忆网络分层级提取序列关键信息；最终通过计算关系路径特征向量与候选关系向量间的相似度得出预测结果。【结果】 在NELL-995和FB15k-237数据集上进行链路预测实验，结果表明，该算法与已有基于关系路径的知识图谱补全算法CNN-BiLSTM等相比，MAP值提高了1.8%，Hits@1指标提高了1.4%；在Kinship数据集上，其Hits@3值达到了0.988。【结论】 本文通过实验证明了所提出的HAN-BiLSTM算法能有效提取关系路径的整体特征和局部特征，从而提高知识图谱补全效果。

关键词: 知识图谱补全, 关系路径推理, 多层注意力机制, 双向长短期记忆网络

Abstract:

[Objective] In order to utilize both local and global features hidden in relation paths between two entities in knowledge graphs which are presently often neglected in most path-based embedding methods for knowledge graph completion. In this paper, we propose a new bidirectional long short-term memory network approach based on hierarchical attention mechanisms, which process relation paths at both entity-relation level and relation path level. [Methods] After scanning the relation paths between two entities, these relation paths are vectorized considering the entity types and the relations on the paths. Then we introduce the paths into a low-dimensional space using a BiLSTM model which has hierarchical attention layers to capture the important facts at different levels. Finally, a prediction is made based on the similarity between the feature vector and the potentially possible relations. [Results] The model conducts link prediction tasks over several datasets including NELL-995 and FB15k-237. The results show that the MAP score of the HAN-BiLSTM model is 1.8% better than traditional methods such as CNN-BiLSTM, as well as an improvement in Hits@1 by 1.4%. The model achieved a Hits@3 score of 0.988 over the Kinship dataset.[Conclusions] Experiment results show that the proposed algorithm can effectively extract both global and local features of the relation paths, so as to improve the effect of knowledge graph completion

Key words: knowledge graph completion, path-based reasoning, hierarchical attention networks, bidirectional long short-term memory networks

张晓帆, 孙海春, 李欣. 融合多层注意力机制与BiLSTM的知识图谱补全算法研究[J]. 数据与计算发展前沿, 2023, 5(3): 123-137.

ZHANG Xiaofan, SUN Haichun, LI Xin. Hierarchical Attention-Based Bidirectional Long Short-Term Memory Networks for Knowledge Graph Completion[J]. Frontiers of Data and Computing, 2023, 5(3): 123-137, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2023.03.009.

图/表 8

图1

图2

表1

表2

表3

表4

表5

表6

参考文献 34

[1]	BELLO-ORGAZ G, JUNG J J, CAMACHO D. Social big data: Recent achievements and new challenges[J/OL]. Information Fusion, 2016, 28: 45-59. DOI:10.1016/j.inffus.2015.08.005. doi: 10.1016/j.inffus.2015.08.005
[2]	PUJARA J, AUGUSTINE E, GETOOR L. Sparsity and Noise: Where Knowledge Graph Embeddings Fall Short[C]// Proceedings of the 2017 Conference on Emp-irical Methods in Natural Language Processing, 2017: 1751-1756.
[3]	BORDES A, USUNIER N, GARCIA-DURAN A. Tran-slating Embeddings for Modeling Multi-relational Data[C]// Advances in Neural Information Processing Sys-tems:Volume 26, Curran Associates, Inc., 2013: 1-9.
[4]	NICKEL M, TRESP V, KRIEGEL H P. A Three-Way Model for Collective Learning on Multi-Relational Data[C]// ICML, 2011: 1-8.
[5]	TROUILLON T, WELBL J, RIEDEL S. Complex Emb-eddings for Simple Link Prediction[C]// Proceedings of The 33rd International Conference on Machine Learning, PMLR, 2016: 2071-2080.
[6]	LAO N, MITCHELL T, COHEN W W. Random Walk Inference and Learning in A Large Scale Knowledge Base[C]// Proceedings of the 2011 Conference on Empiri-cal Methods in Natural Language Processing. Edinburgh, Scotland, UK.: Association for Computational Lingui-stics, 2011: 529-539.
[7]	NEELAKANTAN A, ROTH B, MCCALLUM A. Comp-ositional Vector Space Models for Knowledge Base Comp-letion[C/OL]// Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1:Long Papers), Beijing, China: Association for Computational Linguistics, 2015: 156-166. DOI:10.3115/v1/P15-1016. doi: 10.3115/v1/P15-1016
[8]	DAS R, NEELAKANTAN A, BELANGER D. Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks[C]// Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Valencia, Spain:Association for Computational Lingui-stics, 2017: 132-141.
[9]	JIANG X, WANG Q, QI B. Attentive Path Combination for Knowledge Graph Completion[C]// Proceedings of the Ninth Asian Conference on Machine Learning, PMLR, 2017: 590-605.
[10]	JAGVARAL B, LEE W K, ROH J S. Path-based reason-ing approach for knowledge graph completion using CNN-BiLSTM with attention mechanism[J/OL]. Expert Systems with Applications, 2020, 142: 112960. DOI: 10.1016/j.eswa.2019.112960. doi: 10.1016/j.eswa.2019.112960
[11]	Huang Z, Wei X, Kai Y. Bidirectional LSTM-CRF Mo-dels for Sequence Tagging[J]. Computer Science, 2015: 1-10. arXiv:1508.01991.
[12]	JI S, PAN S, CAMBRIA E. A Survey on Knowledge Graphs: Representation, Acquisition and Applications[J/OL]. IEEE Transactions on Neural Networks and Lea-rning Systems, 2021: 1-21. DOI:10.1109/TNNLS.2021.3070843. doi: 10.1109/TNNLS.2021.3070843
[13]	WANG Q, MAO Z, WANG B. Knowledge Graph Emb-edding: A Survey of Approaches and Applications[J/OL]. IEEE Transactions on Knowledge and Data Engi-neering, 2017, 29(12): 2724-2743. DOI:10.1109/tkde.2017.2754499. doi: 10.1109/tkde.2017.2754499
[14]	BORDES A, WESTON J, COLLOBERT R. Learning Structured Embeddings of Knowledge Bases[C]// Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011: 301-306.
[15]	WANG Z, ZHANG J, FENG J. Knowledge Graph Emb-edding by Translating on Hyperplanes[J]. Proceedings of the AAAI Conference on Artificial Intell-igence, 2014, 28(1):1112-1119.
[16]	LIN Y, LIU Z, SUN M. Learning Entity and Relation Embeddings for Knowledge Graph Completion[C]// Twenty-Ninth AAAI Conference on Artificial Intellige-nce, 2015: 2081-2187.
[17]	JI G, HE S, XU L. Knowledge Graph Embedding via Dynamic Mapping Matrix[C/OL]// Proceedings of the 53rd Annual Meeting of the Association for Compu-tational Linguistics and the 7th International Joint Conf-erence on Natural Language Processing (Volume 1:Long Papers), Beijing, China: Association for Computational Linguistics, 2015: 687-696. DOI:10.3115/v1/P15-1067. doi: 10.3115/v1/P15-1067
[18]	SOCHER R, CHEN D, MANNING C D. Reasoning With Neural Tensor Networks for Knowledge Base Comp-letion[C]// Advances in Neural Information Processing Systems:Volume 26, Curran Associates, Inc., 2013: 1-10.
[19]	YANG B, YIH W tau, HE X. Embedding Entities and Rel-ations for Learning and Inference in Knowledge Bases[J]. arXiv:1412.6575 [cs], 2015: 1-12.
[20]	DETTMERS T, MINERVINI P, STENETORP P. Con-volutional 2D Knowledge Graph Embeddings[J]. Pro-ceedings of the AAAI Conference on Artificial Intelligen-ce, 2018, 32(1): 1811-1818.
[21]	GARDNER M, TALUKDAR P P, KISIEL B. Improving Learning and Inference in a Large Knowledge-Base using Latent Syntactic Cues[C]// Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, Washington, USA: Association for Computational Linguistics, 2013: 833-838.
[22]	GARDNER M, MITCHELL T. Efficient and Expressive Knowledge Base Completion Using Subgraph Feature Extraction[C/OL]// Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal: Association for Computational Ling-uistics, 2015: 1488-1498. DOI:10.18653/v1/D15-1173. doi: 10.18653/v1/D15-1173
[23]	WANG P, XU B, WU Y. Link prediction in social net-works: the state-of-the-art[J/OL]. Science China Infor-mation Sciences, 2015, 58(1): 1-38. DOI:10.1007/s11-432-014-5237-y. doi: 10.1007/s11-432-014-5237-y
[24]	Lan Y, He S, Liu K, et al. Path-based knowledge reaso-ning with textual semantic information for medical knowledge graph completion[J]. BMC Medical Inform-atics and Decision Making, 2021, 21(9): 1-12.
[25]	TREISMAN A M. Strategies and models of selective atte-ntion[J/OL]. Psychol Rev, 1969, 76(3): 282-299. DOI: 10.1037/h0027242. doi: 10.1037/h0027242
[26]	BAHDANAU D, CHO K, BENGIO Y. Neural Machine Translation by Jointly Learning to Align and Translate[J/OL]. Computer Science, 2014: 1-15. DOI:10.48550/ar-Xiv.1409.0473. doi: 10.48550/ar-Xiv.1409.0473
[27]	YANG Z, YANG D, DYER C. Hierarchical Attention Networks for Document Classification[C/OL]. 2016: 1480-1489. DOI:10.18653/v1/N16-1174. doi: 10.18653/v1/N16-1174
[28]	TREISMAN A M. Selective Attention in Man[J/OL]. Br Med Bull, 1964, 20(1): 12-16. DOI:10.1093/oxfor-djournals.bmb.a070274. doi: 10.1093/oxfor-djournals.bmb.a070274
[29]	BOUCHARD G, SINGH S, TROUILLON T. On Appr-oximate Reasoning Capabilities of Low-Rank Vec-tor Spaces[C]// 2015 AAAI Spring Symposium Series, 2015: 6-9.
[30]	KOK S, DOMINGOS P. Statistical predicate invention[C/OL]// Proceedings of the 24th international conference on Machine learning. New York, NY, USA: Association for Computing Machinery, 2007: 433-440. DOI: 10.1145/1273496.1273551. doi: 10.1145/1273496.1273551
[31]	XIONG W, HOANG T, WANG W Y. DeepPath: A Rein-forcement Learning Method for Knowledge Graph Rea-soning[C/OL]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark: Association for Computational Linguistics, 2017: 564-573. DOI:10.18653/v1/D17-1060. doi: 10.18653/v1/D17-1060
[32]	TOUTANOVA K, CHEN D, PANTEL P. Representing Text for Joint Embedding of Text and Knowledge Bases[C/OL]// Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portu-gal: Association for Computational Linguistics, 2015: 1499-1509. DOI:10.18653/v1/D15-1174. doi: 10.18653/v1/D15-1174
[33]	Das R, Dhuliawala S, Zaheer M, et al. Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Kno-wledge Bases using Reinforcement Learning[C]// Inter-national Conference on Learning Representations, 2018:1-18.
[34]	杜会芳, 王昊奋, 史英慧, 王萌. 知识图谱多跳问答推理研究进展、挑战与展望[J]. 大数据, 2021, 7(03): 60-79.

Datasets	entities	relations	train	valid	test
Countries	272	2	1158	68	72
Kinship	104	26	6926	769	1069
NELL995	63917	198	137645	5000	5000
FB15k-237	14541	237	272115	17535	20466

Dataset	NELL995				FB15k-237
Metric	MAP	MRR	Hits@1	Hits@3	MAP	MRR	Hits@1	Hits@3
PRA	0.696	0.696	0.637	0.747	0.412	0.412	0.322	0.331
Distmult	-	0.863	0.801	0.907	-	0.541	0.413	0.554
Single-Model	0.855	0.859	0.788	0.914	0.574	0.575	0.512	0.567
DeepPath	0.809	0.835	0.744	0.89	0.449	0.459	0.343	0.436
MINERVA	0.876	0.879	0.813	0.931	0.606	0.615	0.490	0.659
ConvE	-	0.909	0.904	0.929	-	0.567	0.444	0.629
CNN-BiLSTM	0.894	0.898	0.838	0.951	0.652	0.662	0.544	0.709
Our Model	0.899	0.910	0.905	0.957	0.664	0.665	0.546	0.718

Dataset	Kinship				Countries
Metric	MAP	MRR	Hits@1	Hits@3	MAP	MRR	Hits@1	Hits@3
PRA	0.724	0.799	0.699	0.896	0.687	0.739	0.577	0.900
Single-Model	0.804	0.804	0.814	0.885	0.941	0.941	0.918	0.956
MINERVA	0.816	0.824	0.710	0.937	0.960	0.960	0.925	0.995
CNN-BiLSTM	0.946	0.952	0.918	0.984	0.947	0.947	0.916	0.985
Our Model	0.948	0.955	0.918	0.989	0.964	0.962	0.923	0.993

Task(relation)	#Q	PRA	TransE	DeepPath	MINERVA	CNN-BiLSTM	Our Model
athletePlaysInLeague	381	0.904	0.773	0.951	0.954	0.978	0.978
worksFor	513	0.619	0.676	0.721	0.816	0.838	0.839
orgHiredPerson	486	0.512	0.722	0.797	0.866	0.855	0.872
athletePlaysInSport	603	0.533	0.963	0.894	0.985	0.980	0.983
teamPlaysSport	112	0.787	0.760	0.729	0.739	0.864	0.866
bornLocation	193	0.734	0.604	0.782	0.757	0.810	0.810
athleteHomeStadium	201	0.861	0.717	0.843	0.924	0.914	0.932
orgHeadquaterCity	249	0.855	0.619	0.807	0.942	0.937	0.943
athletePlaysForTeam	387	0.600	0.626	0.765	0.839	0.860	0.867
Overall	3125	0.711	0.717	0.809	0.876	0.894	0.896

组别	关系路径编码方式	关系路径特征向量计算方法
A	RNN	平均池化
B	RNN	多层注意力机制
C	BiLSTM	平均池化
D	BiLSTM	多层注意力机制

融合多层注意力机制与BiLSTM的知识图谱补全算法研究

Hierarchical Attention-Based Bidirectional Long Short-Term Memory Networks for Knowledge Graph Completion

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 34

相关文章 0

编辑推荐

Metrics

本文评价