A Root Cause Localization Method Based on Service Dependency Graph for Microservice System Failures

doi:10.11871/jfdc.issn.2096-742X.2024.04.007

Abstract

Abstract:

[Objective] To address the frequent occurrences of system failures and the rapid propagation of anomalies within microservice architectures, particularly due to the complexity of diagnosis caused by fine service granularity, frequent updates, and complex service dependencies, this paper proposes a rapid root cause localization method based on dynamic service dependency graphs. [Methods] This method utilizes configuration information and log data of microservices to dynamically generate service dependency graphs, effectively capturing the dynamic changes in service dependencies. In the event of a failure, it uses the service dependency graph and anomaly event data to infer the causal chain of anomalies and constructs an anomaly causality graph. By considering the weight of service dependencies, it searches and ranks potential root cause nodes in the service dependency graph to accurately locate the source of the anomaly. [Results] Experimental results demonstrate that the proposed method achieves an average precision rate of 66% for top 5 root cause localization, surpassing existing similar methods.

Key words: microservice, service dependency, anomaly causal relationship, root cause localization

ZHANG Qixun, JIA Tong, YANG Yong, LI Ying. A Root Cause Localization Method Based on Service Dependency Graph for Microservice System Failures[J]. Frontiers of Data and Computing, 2024, 6(4): 87-95, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.04.007.

Figures/Tables 6

Fig.1

Fig.2

Fig.3

Table 1

Table 2

Table 3

References 18

[1]	冯志勇, 徐砚伟, 薛霄, 等. 微服务技术发展的现状与展望[J]. 计算机研究与发展, 2020, 57(5): 1103-1122.
[2]	贾统, 李影, 吴中海. 基于日志数据的分布式软件系统故障诊断综述[J]. 软件学报, 2020, 31(7): 1997-2018.
[3]	OPPENHEIMER D, GANAPATHI A, PATTERSON D A. Why do internet services fail, and what can be done about it?[C]// USENIX Symposium on Internet Technologies and Systems, March 2003: 125-128.
[4]	ZAND A, VIGNA G, KEMMERER R, et al. Rippler: Delay injection for service dependency detection[C]. IEEE INFOCOM 2014-IEEE Conference on Computer Communications, IEEE, 2014: 2157-2165.
[5]	CHEN X, ZHANG M, MAO Z M, et al. Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions[C]. OSDI. Vol. 8. 2008: 117-130.
[6]	KAR G, KELLER A, CALO S. Managing application services over service provider networks: Architecture and dependency analysis[C]. NOMS 2000: 61-74.
[7]	YUAN Y, HU A, SHI W, et al. Learning-based anomaly cause tracing with synthetic analysis of logs from multiple cloud service components[C]// 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), IEEE, 2019: 66-71.
[8]	杨勇, 李影, 吴中海. 分布式追踪技术综述[J]. 软件学报, 2020, 31(7): 2019-2039.
[9]	ZHOU X, PENG X, XIE T, et al. Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study[J]. IEEE Transactions on Software Engineering, 2021, 47(2): 243-260.
[10]	GUO X, PENG X, WANG H, et al. Graph-based trace analysis for microservice architecture understanding and problem diagnosis[C]// Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020: 1387-1397.
[11]	LIU D, HE C, PENG X, et al. MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems[C]// 2021 IEEE/ACM 43rd International Conference on Software Engineering:Software Engineering in Practice (ICSE-SEIP), IEEE, 2021: 338-347.
[12]	KIM M, SUMBALY R, SHAH S. Root cause detection in a service-oriented architecture[J]. ACM SIGMETRICS Performance Evaluation Review, 2013, 41(1): 93-104.
[13]	WU L, TORDSSON J, BOGATINOVSKI J, et al. MicroDiag: Fine-grained Performance Diagnosis for Microservice Systems[C]// ICSE21 Workshop on Cloud Intelligence, 2021: 1-36.
[14]	LI Y, YU R, SHAHABI C, et al. Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting[C]// International Conference on Learning Representations, 2018: 1-16.
[15]	YU B, YIN H, ZHU Z. Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting[J]. Proceedings of the International Joint Conference on Artificial Intelligence, 2018: 3634-3640.
[16]	WU Z, PAN S, LONG G, et al. Graph WaveNet for Deep Spatial-Temporal Graph Modeling[J]. Proceedings of the International Joint Conference on Artificial Intelligence, 2019: 1907-1913.
[17]	KIM M, SUMBALY R, SHAH S. Root cause detection in a service-oriented architecture[J]. ACM SIGMETRICS Performance Evaluation Review, 2013, 41(1): 93-104.
[18]	LIN J, CHEN P, ZHENG Z. Microscope: Pinpoint Performance Issues with Causal Graphs in Micro-service Environments[C]// 16th International Conference on Service-Oriented Computing (ICSOC 2018), Hangzhou, China, November 12-15, 2018, Proceedings, Springer, Cham, 2018: 3-20.

方法	P（精确率）	R（召回率）
MicroDiag	36.1%	42.2%
DCRNN	57.2%	65.1%
STGCN	61.8%	78.8%
Graph WaveNet	70.2%	76.1%
本文方法	79.1%	89.1%

评估指标	本文方法	MicroDiag	Microscope	MonitorRank
PR@1	38%	22%	23%	25%
PR@3	75%	54%	45%	50%
PR@5	85%	65%	61%	63%
AP@5	66%	47%	61%	63%

测试项	节点数量	预测用时
异常因果关系发现	42个微服务	118毫秒
根因定位		115毫秒