基于服务依赖图的微服务系统故障根因定位方法

doi:10.11871/jfdc.issn.2096-742X.2024.04.007

数据与计算发展前沿 ›› 2024, Vol. 6 ›› Issue (4): 87-95.

CSTR: 32002.14.jfdc.CN10-1649/TP.2024.04.007

doi: 10.11871/jfdc.issn.2096-742X.2024.04.007

• 专刊：面向国家科学数据中心的基础软件栈及系统 • 上一篇下一篇

基于服务依赖图的微服务系统故障根因定位方法

张齐勋^1,^*(),贾统²,杨勇³,李影⁴

1.北京大学，软件与微电子学院，北京 102600
2.北京大学，人工智能研究院，北京 100871
3.北京大学，信息科学技术学院，北京 100871
4.北京大学，软件工程国家工程研究中心，北京 100871

收稿日期:2024-02-05 出版日期:2024-08-20 发布日期:2024-08-20
通讯作者: *张齐勋（E-mail: zhangqx@pku.edu.cn）
作者简介:张齐勋，北京大学软件与微电子学院, 博士, 副教授, CCF会员，主要研究领域为软件工程，智能运维。
本文中负责论文撰写，根因定位方法设计。
ZHANG Qixun, School of Software and Microelectronics, Peking University, Ph.D., Associate Professor, CCF Member, focuses on research areas such as software engineering and Artificial Intelligence for IT Operations.
In this paper, he is responsible for the paper writing and the design of root cause localization methods.
E-mail: zhangqx@pku.edu.cn
基金资助:
国家重点研发计划(2021YFF0704202)

A Root Cause Localization Method Based on Service Dependency Graph for Microservice System Failures

ZHANG Qixun^1,^*(),JIA Tong²,YANG Yong³,LI Ying⁴

1. School of Software and Microelectronics, Peking University, Beijing 102600, China
2. Institute for Artificial Intelligence, Peking University, Beijing 100871, China
3. School of Electronics Engineering and Computer Science, Peking University, Beijing 100871, China
4. National Engineering Research Center for Software Engineering, Peking University, Beijing 100871, China

Received:2024-02-05 Online:2024-08-20 Published:2024-08-20

摘要/Abstract

摘要：

【目的】 为解决微服务架构中频繁出现的系统故障以及异常快速传播的问题，特别是由于服务粒度细、更新迭代频繁及服务依赖复杂性引起的诊断复杂性，本文提出了一种基于动态微服务依赖图的故障根因快速定位方法。【方法】 本方法基于微服务的配置信息和日志数据，动态生成服务依赖图，有效捕获服务间的动态依赖变化。在故障发生时，利用服务依赖图和异常事件数据推断异常间的因果链，构造异常因果关系图。结合服务依赖的权重，通过服务依赖图中搜寻并排序可能的根因节点，以实现异常源头的精准定位。【结果】 实验结果表明，本方法异常根因top 5平均定位精确率达到66%，优于现有其它同类方法。

关键词: 微服务, 服务依赖, 异常因果关系, 根因定位

Abstract:

[Objective] To address the frequent occurrences of system failures and the rapid propagation of anomalies within microservice architectures, particularly due to the complexity of diagnosis caused by fine service granularity, frequent updates, and complex service dependencies, this paper proposes a rapid root cause localization method based on dynamic service dependency graphs. [Methods] This method utilizes configuration information and log data of microservices to dynamically generate service dependency graphs, effectively capturing the dynamic changes in service dependencies. In the event of a failure, it uses the service dependency graph and anomaly event data to infer the causal chain of anomalies and constructs an anomaly causality graph. By considering the weight of service dependencies, it searches and ranks potential root cause nodes in the service dependency graph to accurately locate the source of the anomaly. [Results] Experimental results demonstrate that the proposed method achieves an average precision rate of 66% for top 5 root cause localization, surpassing existing similar methods.

Key words: microservice, service dependency, anomaly causal relationship, root cause localization

张齐勋, 贾统, 杨勇, 李影. 基于服务依赖图的微服务系统故障根因定位方法[J]. 数据与计算发展前沿, 2024, 6(4): 87-95.

ZHANG Qixun, JIA Tong, YANG Yong, LI Ying. A Root Cause Localization Method Based on Service Dependency Graph for Microservice System Failures[J]. Frontiers of Data and Computing, 2024, 6(4): 87-95, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.04.007.

图/表 6

图1

图2

图3

表1

表2

表3

参考文献 18

[1]	冯志勇, 徐砚伟, 薛霄, 等. 微服务技术发展的现状与展望[J]. 计算机研究与发展, 2020, 57(5): 1103-1122.
[2]	贾统, 李影, 吴中海. 基于日志数据的分布式软件系统故障诊断综述[J]. 软件学报, 2020, 31(7): 1997-2018.
[3]	OPPENHEIMER D, GANAPATHI A, PATTERSON D A. Why do internet services fail, and what can be done about it?[C]// USENIX Symposium on Internet Technologies and Systems, March 2003: 125-128.
[4]	ZAND A, VIGNA G, KEMMERER R, et al. Rippler: Delay injection for service dependency detection[C]. IEEE INFOCOM 2014-IEEE Conference on Computer Communications, IEEE, 2014: 2157-2165.
[5]	CHEN X, ZHANG M, MAO Z M, et al. Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions[C]. OSDI. Vol. 8. 2008: 117-130.
[6]	KAR G, KELLER A, CALO S. Managing application services over service provider networks: Architecture and dependency analysis[C]. NOMS 2000: 61-74.
[7]	YUAN Y, HU A, SHI W, et al. Learning-based anomaly cause tracing with synthetic analysis of logs from multiple cloud service components[C]// 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), IEEE, 2019: 66-71.
[8]	杨勇, 李影, 吴中海. 分布式追踪技术综述[J]. 软件学报, 2020, 31(7): 2019-2039.
[9]	ZHOU X, PENG X, XIE T, et al. Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study[J]. IEEE Transactions on Software Engineering, 2021, 47(2): 243-260.
[10]	GUO X, PENG X, WANG H, et al. Graph-based trace analysis for microservice architecture understanding and problem diagnosis[C]// Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020: 1387-1397.
[11]	LIU D, HE C, PENG X, et al. MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems[C]// 2021 IEEE/ACM 43rd International Conference on Software Engineering:Software Engineering in Practice (ICSE-SEIP), IEEE, 2021: 338-347.
[12]	KIM M, SUMBALY R, SHAH S. Root cause detection in a service-oriented architecture[J]. ACM SIGMETRICS Performance Evaluation Review, 2013, 41(1): 93-104.
[13]	WU L, TORDSSON J, BOGATINOVSKI J, et al. MicroDiag: Fine-grained Performance Diagnosis for Microservice Systems[C]// ICSE21 Workshop on Cloud Intelligence, 2021: 1-36.
[14]	LI Y, YU R, SHAHABI C, et al. Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting[C]// International Conference on Learning Representations, 2018: 1-16.
[15]	YU B, YIN H, ZHU Z. Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting[J]. Proceedings of the International Joint Conference on Artificial Intelligence, 2018: 3634-3640.
[16]	WU Z, PAN S, LONG G, et al. Graph WaveNet for Deep Spatial-Temporal Graph Modeling[J]. Proceedings of the International Joint Conference on Artificial Intelligence, 2019: 1907-1913.
[17]	KIM M, SUMBALY R, SHAH S. Root cause detection in a service-oriented architecture[J]. ACM SIGMETRICS Performance Evaluation Review, 2013, 41(1): 93-104.
[18]	LIN J, CHEN P, ZHENG Z. Microscope: Pinpoint Performance Issues with Causal Graphs in Micro-service Environments[C]// 16th International Conference on Service-Oriented Computing (ICSOC 2018), Hangzhou, China, November 12-15, 2018, Proceedings, Springer, Cham, 2018: 3-20.

方法	P（精确率）	R（召回率）
MicroDiag	36.1%	42.2%
DCRNN	57.2%	65.1%
STGCN	61.8%	78.8%
Graph WaveNet	70.2%	76.1%
本文方法	79.1%	89.1%

评估指标	本文方法	MicroDiag	Microscope	MonitorRank
PR@1	38%	22%	23%	25%
PR@3	75%	54%	45%	50%
PR@5	85%	65%	61%	63%
AP@5	66%	47%	61%	63%

测试项	节点数量	预测用时
异常因果关系发现	42个微服务	118毫秒
根因定位		115毫秒

基于服务依赖图的微服务系统故障根因定位方法

A Root Cause Localization Method Based on Service Dependency Graph for Microservice System Failures

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献 18

相关文章 2

编辑推荐

Metrics

本文评价

[1]	张博尧, 曹荣强, 万萌, 孙境棋, 王彦棡, 王珏, 赵永华. 垂直领域知识图谱构建及应用平台的设计与实现[J]. 数据与计算发展前沿, 2023, 5(3): 111-122.
[2]	李思毅,马诗雨,崔丽月,张圣林,孙永谦,张玉志. 微服务架构下的根因定位方法综述[J]. 数据与计算发展前沿, 2022, 4(3): 78-89.