Research on Unsupervised KPI Anomaly Detection Based on Deep Learning

doi:10.11871/jfdc.issn.2096-742X.2020.03.008

Abstract

Abstract:

[Objective] Automatic key performance indicator (KPI), the basis of Internet artificial intelligence operations (AIOps), is of vital importance to rapid failure detection and mitigation. [Scope of the literature] In this paper, we investigate unsupervised KPI anomaly detection methods, which are based on deep generative models. [Methods] We systematically describe the theoretic model of Donut, Bagel, and Buzz, which are all unsupervised KPI anomaly detection methods, and analyze their advantages and limitations in terms of accuracy and efficiency. [Results] We evaluate the performance of those three approaches based on real-world KPI data. [Limitations] The KPI anomaly detection methods based on deep generative model are continuously evolving, and we will explore more methods in this area. [Conclusions] Choosing a deep generative model should consider the characteristics of KPI data. Generally, if the KPI data is sensitive to timing information, we should apply Bagel to perform anomaly detection. Moreover, Buzz should be used if the data is non-seasonal and complex.

Key words: deep learning, unsupervised learning, key performance indicator, anomaly detection, generative model

Zhang Shenglin,Lin Xiaofei,Sun Yongqian,Zhang Yuzhi,Pei Dan. Research on Unsupervised KPI Anomaly Detection Based on Deep Learning[J]. Frontiers of Data and Computing, 2020, 2(3): 87-100.

Figures/Tables 16

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Table 1

Fig.8

Fig.9

Fig.10

Table 2

Table 3

Fig.11

Table 4

Fig.12

References 40

[1]	Vattikonda B C, Dave V, Guha S, et al. Empirical analysis of search advertising strategies[C]// Proceedings of the 2015 Internet Measurement Conference. 2015: 79-91.
[2]	Chen Y, Mahajan R, Sridharan B, et al. A provider-side view of web search response time[J]. ACM SIGCOMM Computer Communication Review, 2013,43(4):243-254. doi: 10.1145/2534169.2486035
[3]	Miao R, Potharaju R, Yu M, et al. The dark menace: Characterizing network-based attacks in the cloud[C]// Proceedings of the 2015 Internet Measurement Conference. 2015: 169-182.
[4]	Zhang S, Liu Y, Pei D, et al. Rapid and robust impact assessment of software changes in large internet-based services[C]// Proceedings of the 11th ACM Conference on Emerging Networking Experiments and Technologies. 2015: 1-13.
[5]	Liu D, Zhao Y, Xu H, et al. Opprentice: Towards practical and automatic anomaly detection through machine learning[C]// Proceedings of the 2015 Internet Measurement Conference. 2015: 211-224.
[6]	Xu H, Chen W, Zhao N, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal KPI in web applications[C]// Proceedings of the 2018 World Wide Web Conference. 2018: 187-196.
[7]	Zhang S, Liu Y, Pei D, et al. Funnel: Assessing software changes in web-based services[J]. IEEE Transactions on Services Computing, 2016,11(1):34-48.
[8]	Knorn F, Leith D J. Adaptive kalman filtering for anomaly detection in software appliances[C]// IEEE INFOCOM Workshops 2008. IEEE, 2008: 1-6.
[9]	Pincombe B. Anomaly detection in time series of graphs using arma processes[J]. Asor Bulletin, 2005,24(4):2.
[10]	Yan H, Flavel A, Ge Z, et al. Argus: End-to-end service anomaly detection and localization from an ISP’s point of view[C]// 2012 Proceedings IEEE INFOCOM. IEEE, 2012: 2756-2760.
[11]	Lu W, Ghorbani A A. Network anomaly detection based on wavelet analysis[J]. EURASIP Journal on Advances in Signal Processing, 2008,2009:1-16.
[12]	Laptev N, Amizadeh S, Flint I. Generic and scalable framework for automated time-series anomaly detection[C]// Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2015: 1939-1947.
[13]	Amer M, Goldstein M, Abdennadher S. Enhancing one-class support vector machines for unsupervised anomaly detection[C]// Proceedings of the ACM SIGKDD Workshop on Outlier Detection and Description. 2013: 8-15.
[14]	Sölch M, Bayer J, Ludersdorfer M, et al. Variational inference for on-line anomaly detection in high-dimensional time series[J]. arXiv preprint arXiv:1602.07109, 2016.
[15]	Chandola V, Banerjee A, Kumar V. Anomaly detection: A survey[J]. ACM computing surveys (CSUR), 2009,41(3):1-58.
[16]	Erfani S M, Rajasegarar S, Karunasekera S, et al. High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning[J]. Pattern Recognition, 2016,58:121-134. doi: 10.1016/j.patcog.2016.03.028
[17]	Fontugne R, Borgnat P, Abry P, et al. Mawilab: combining diverse anomaly detectors for automated anomaly labeling and performance benchmarking[C]// Proceedings of the 6th International COnference. 2010: 1-12.
[18]	Krishnamurthy B, Sen S, Zhang Y, et al. Sketch-based change detection: methods, evaluation, and applications[C]// Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement. 2003: 234-247.
[19]	Laxhammar R, Falkman G, Sviestins E. Anomaly detection in sea traffic-a comparison of the gaussian mixture model and the kernel density estimator[C]// 2009 12th International Conference on Information Fusion. IEEE, 2009: 756-763.
[20]	Lee S B, Pei D, Hajiaghayi M T, et al. Threshold compression for 3g scalable monitoring[C]// 2012 Proceedings IEEE INFOCOM. IEEE, 2012: 1350-1358.
[21]	Mahimkar A, Ge Z, Wang J, et al. Rapid detection of maintenance induced changes in service performance[C]// Proceedings of the Seventh COnference on emerging Networking EXperiments and Technologies. 2011: 1-12.
[22]	Nicolau M, McDermott J. One-class classification for anomaly detection with kernel density estimation and genetic programming[C]// European Conference on Genetic Programming. Springer, Cham, 2016: 3-18.
[25]	Shanbhag S, Wolf T. Accurate anomaly detection through parallelism[J]. IEEE network, 2009,23(1):22-28.
[23]	Yaacob A H, Tan I K T, Chien S F, et al. Arima based network anomaly detection[C]// 2010 Second International Conference on Communication Software and Networks. IEEE, 2010: 205-209.
[24]	Ma M, Zhang S, Pei D, et al. Robust and rapid adaption for concept drift in software system anomaly detection[C]// 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2018: 13-24.
[25]	An J, Cho S. Variational autoencoder based anomaly detection using reconstruction probability[J]. Special Lecture on IE, 2015,2(1).
[26]	Zong B, Song Q, Min M R, et al. Deep autoencoding gaussian mixture model for unsupervised anomaly detection[J]. 2018.
[27]	Li Z, Chen W, Pei D. Robust and unsupervised KPI anomaly detection based on conditional variational autoencoder[C]// 2018 IEEE 37th International Performance Computing and Communications Conference (IPCCC). IEEE, 2018: 1-9.
[28]	Chen W, Xu H, Li Z, et al. Unsupervised anomaly detection for intricate KPI via adversarial training of vae[C]// IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2019: 1891-1899.
[29]	Kingma D P Welling M. Auto-encoding variational bayes[J]. arXiv preprint arXiv:1312.6114, 2013.
[30]	Liu F T, Ting K M, Zhou Z H. Isolation forest[C]// 2008 Eighth IEEE International Conference on Data Mining. IEEE, 2008: 413-422.
[31]	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[C]// Advances in neural information processing systems. 2014: 2672-2680.
[32]	Goodfellow I, Bengio Y, Courville A. Deep learning[M]. MIT press, 2016.
[33]	Kingma D P, Mohamed S, Rezende D J, et al. Semi-supervised learning with deep generative models[C]// Advances in neural information processing systems. 2014: 3581-3589.
[34]	Sohn K, Lee H, Yan X. Learning structured output representation using deep conditional generative models[C]// Advances in neural information processing systems. 2015: 3483-3491.
[35]	Arjovsky M, Chintala S, Bottou L. Wasserstein gan[J]. arXiv preprint arXiv:1701.07875, 2017.
[36]	Sterne J A C, White I R, Carlin J B, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls[J]. Bmj, 2009,338:b2393. doi: 10.1136/bmj.b2393 pmid: 19564179
[37]	Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks[C]// Advances in neural information processing systems. 2012: 1097-1105.
[38]	Rezende D J, Mohamed S, Wierstra D. Stochastic backpropagation and approximate inference in deep generative models[J]. arXiv preprint arXiv:1401.4082, 2014.
[39]	Geweke J. Bayesian inference in econometric models using Monte Carlo integration[J]. Econometrica: Journal of the Econometric Society, 1989: 1317-1339.
[40]	AIOps Challenge. http://iops.ai/[OL]. [2020-04-20]

日期和时间	2018/7/3 16:25:13 星期二
分解信息	25(minute),16(hour),2(day of week)
one-hot编码	0…0,1,0…0\|0…0,1,0000000,\|0,1,00000（其中第一处省略23个0,第二处省略32个0,第三处省略14个0）

比较项	Donut	Bagel	Buzz
运行耗时	较短	较短	较长
运行性能要求	一般	一般	较高
需要大量数据	是	是	是
依赖异常标注	否	否	否
异常检测准确性	较好	较好	较好
适用于周期性KPI	是	是	否
适用于非周期KPI	否	否	是
对时间信息敏感	否	是	否

KPI序列数目	50
KPI数据点数目	226 200
缺失值点数	1 539
异常数据点数目	2 844

	Precision	Recall	F-Score
Donut	0.88	0.82	0.84
Bagel	0.91	0.85	0.88
Buzz	0.78	0.44	0.56