KPI异常检测方法评估

doi:10.11871/jfdc.issn.2096-742X.2022.03.004

数据与计算发展前沿 ›› 2022, Vol. 4 ›› Issue (3): 46-65.

CSTR: 32002.14.jfdc.CN10-1649/TP.2022.03.004

doi: 10.11871/jfdc.issn.2096-742X.2022.03.004

• 专刊:先进智能计算平台及应用(下) • 上一篇下一篇

KPI异常检测方法评估

孙永谦^1,²(),张茹茹¹(),林子涵¹(),张圣林^1,^2,^3,^*(),谭智元¹(),张玉志^1,^2,³()

1.南开大学,软件学院,天津 300350
2.天津市操作系统企业重点实验室,天津 300350
3.先进计算与关键软件（信创）海河实验室,天津 300350

收稿日期:2022-02-14 出版日期:2022-06-20 发布日期:2022-06-20
通讯作者: 张圣林
作者简介:孙永谦,南开大学,软件学院,博士,讲师,主要研究异常检测、根本原因定位以及数据中心的高性能切换。
本文中主要设计文章架构并修改论文。
SUN Yongqian is currently an assistant professor in the College of Software, Nan-kai University, Tianjin, China. His research interests include anomaly detection, root cause localization, and high perfor-mance switching in datacenter.
In this paper, he is mainly responsible for the design of the paper architecture and paper revision.
E-mail: sunyongqian@nankai.edu.cn|张茹茹,南开大学,软件学院,硕士研究生,主要研究异常检测和根本原因定位。
本文中主要承担文献调研、实验及评估。
ZHANG Ruru is currently a master’s student in the College of Software at Nan-kai University, Tianjin, China. Her research interests include anomaly detection and root cause localization.
In this paper, she is responsible for the related work investigation, experiments, and evaluation.
E-mail: 1852917912@qq.com|林子涵,南开大学,软件学院,硕士研究生,主要研究异常检测、根本原因定位。
本文中承担文献调研及实验。
LIN Zihan is currently a master’s stud-ent in the College of Software at Nankai University, Tianjin, China. His research interests include anomaly detection and root cause localization.
In this paper, he is responsible for the related work investigation and experiments.
E-mail: 2120210568@mail.nankai.edu.cn|张圣林,南开大学,软件学院,博士,副教授,主要研究数据中心网络中的故障检测、诊断和预测。发表SCI/EI 收录论文 15 篇以上。
本文中承担文献调研及指导。
ZHANG Shenglin is currently an associate professor in the College of Software, Nankai University, Tianjin, China. His current research interests include failure detection, diagnosis and prediction in data center networks. He has published 15 papers that are indexed by SCI/EI.
In this paper, he is responsible for the related work inves-tigation.
E-mail: zhangsl@nankai.edu.cn|谭智元,南开大学,软件学院,本科生,主要研究异常检测和深度学习。
本文中承担文献调研及实验
TAN Zhiyuan is an undergraduate in the College of Software at Nankai University, Tianjin, China. His main research inter-ests include anomaly detection and deep le-arning.
In this paper, he is responsible for the related work investigation and experiments
E-mail: bhbean42@qq.com|张玉志,南开大学,软件学院,院长,博士,讲席教授,主要研究方向为人工智能。
本文中承担文献调研。
ZHANG Yuzhi is currently a disting-uished professor and the dean of the College of Software, Nankai Univer-sity, Tianjin, China. His research interests include deep learning and other aspects of artificial intelligence.
In this paper, he is responsible for the related work investigation.
E-mail: zyz@nankai.edu.cn
基金资助:
国家重点研发计划(2018YFB0204304);天津市自然科学基金青年项目(21JCQNJC00180);国家自然科学基金青年项目(61902200);中国博士后科学基金面上项目(2019M651015)

Evaluation of KPI Anomaly Detection Methods

SUN Yongqian^1,²(),ZHANG Ruru¹(),LIN Zihan¹(),ZHANG Shenglin^1,^2,^3,^*(),TAN Zhiyuan¹(),ZHANG Yuzhi^1,^2,³()

1. College of Software, Nankai University, Tianjin 300350, China
2. Tianjin Key Laboratory of Operating System, Tianjin 300350, China
3. Haihe Laboratory of Information Technology Application Innovation, Tianjin 300350, China

Received:2022-02-14 Online:2022-06-20 Published:2022-06-20
Contact: ZHANG Shenglin

摘要/Abstract

摘要：

【目的】关键性能指标（Key Performance Indicators,KPI,如页面访问量、页面访问延迟、服务器CPU利用率、路由器内存使用率、交换机吞吐量、服务器磁盘I/O等）异常检测作为快速故障发现和修复的基础,对快速发展的云计算技术服务越来越重要。【文献范围】本文广泛调研近年来国内外KPI异常检测的相关工作。【方法】对各发展阶段的KPI异常检测方法深入研究和分析,并挑选出13个代表性方法进行实验评估。【结果】总结整理了其一般性问题、挑战和框架,使用3家国内顶尖互联网公司收集到的KPI数据集从准确性、鲁棒性和效率三个方面评估了以上方法的性能。【结论】这些方法涵盖了基于统计的方法、有监督学习方法、半监督学习方法和无监督学习方法,并各有优劣性。本文的研究和分析为将来的研究人员快速、准确地选择最适合其场景的KPI异常检测方法提供了依据。

关键词: 关键性能指标, 异常检测, 方法评估, 机器学习

Abstract:

[Objective] As the basis of rapid fault discovery and repair, key performance indicator (KPI, such as page view count, page-view delay, server CPU utilization, router memory utilization, switch throughput, server disk I/O) anomaly detection is becoming more and more critical for the rapid development of cloud computing technology services. [Coverage] We extensively investigated the related works of KPI anomaly detection at home and abroad in recent years. [Methods] We conduct in-depth research and analysis on KPI anomaly detection methods at various development stages and select 13 representative methods for experimental evaluation. [Results] We summarize the general problems, challenges, and frameworks. We evaluate the performance of these methods using the KPI dataset collected from three top-tier Internet companies in terms of accuracy, robustness, and efficiency. [Conclusions] These methods cover statistics-based, supervised, semi-supervised and unsupervised methods with advantages and disadvantages. Our research and analysis provide a basis for future researchers to select the most appropriate KPI anomaly detection method quickly and accurately for their scenarios.

Key words: Key performance indicator, anomaly detection, method evaluation, machine learning

孙永谦,张茹茹,林子涵,张圣林,谭智元,张玉志. KPI异常检测方法评估[J]. 数据与计算发展前沿, 2022, 4(3): 46-65.

SUN Yongqian,ZHANG Ruru,LIN Zihan,ZHANG Shenglin,TAN Zhiyuan,ZHANG Yuzhi. Evaluation of KPI Anomaly Detection Methods[J]. Frontiers of Data and Computing, 2022, 4(3): 46-65, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2022.03.004.

图/表 10

图 1

图 2

图 3

表 1

表 2

图 4

表 3

图 5

图 6

表 4

KPI异常检测方法使用的参数"

		Value	Parameter	Value
EWMA	$α$	0.1, 0.3, 0.5, 0.7, 0.9
Wavelet	win	3, 5, 7 (days)	freq	low, mid, high
Holt-winters	$α$ $γ$	0.2, 0.4, 0.6, 0.8 0.2, 0.4, 0.6, 0.8	$β$	0.2, 0.4, 0.6, 0.8
Opprentice	P	0.66	R	0.66
Opprentice	$α$	0.8	tree_count	30
ADS	min_samples	3	max_radius	0.05
ADS	tree_count	30
iForest	n_estimators max_samples	100 auto	contamination max_features	0.1 1
OCSVM	kernel	‘rbf’	nu	0.04
DeepAnT	window kernel_size	50 5	filters activation	32 ‘relu’
FuseAD	window kernel_size p q	50 5 2 2	filters activation d	32 ‘relu’ 1
Donut(Bagel)	window_size mcmc_iteration std_epsilon epoch initial_lr lr_anneal_factor (dropout_rate)	120 10 0.0001 50 0.001 0.75 (0.1)	missing_injection_rate latent_dim coefficient batch_size lr_anneal_epochs grad_clip_norm	0.01 8 0.001 256(128) 10 10.0
PUAD	max_radius	0.05
	$ω$	6	$ϵ$	9
	$π$	0.0015	$θ$	8
	$s$	0.200	$λ$	200

表 4

参考文献 52

[1]	裴丹, 张圣林, 裴昶华, 阿里巴巴公司. 基于机器学习的智能运维. 中国计算机学会通讯, 2017, 13(12): 68-72.
[2]	Barford P, Kline J, Plonka D, et al. A signal analysis of network traffic anomalies[C]// Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurement, 2002: 71-82.
[3]	Yan H, Flavel A, Ge Z, et al. Argus: End-to-end service anomaly detection and localization from an isp’s point of view[C]// 2012 Proceedings IEEE INFOCOM, IEEE, 2012: 2756-2760.
[4]	Zhang Yin, Ge Zihui, Greenberg A, et al. Network anom-ography[C]// Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement, 2005: 30-30.
[5]	Krishnamurthy B, Sen S, Zhang Yin, et al. Sketch-based change detection: methods, evaluation, and applications[C]// Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement, 2003: 234-247.
[6]	Choffnes D R, Bustamante F E, Ge Z. Crowdsourcing service-level network event monitoring[C]// Proceedings.of the ACM SIGCOMM 2010 Conference, 2010: 387-398.
[7]	Siffer A, Fouque P A, Termier A, et al. Anomaly detection in streams with extreme value theory[C]// Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017: 1067-1075.
[8]	Li J, Di S, Shen Y, et al. FluxEV: a fast and effective unsupervised framework for time-series anomaly detec-tion[C]// Proceedings of the 14th ACM International Con-ference on Web Search and Data Mining, 2021: 824-832.
[9]	Silverman B W. Density estimation for statistics and data analysis[M]. Routledge, 2018.
[10]	Cleveland R B, Cleveland W S, McRae J E, et al. STL: A seasonal-trend decomposition[J]. J. Off. Stat, 1990, 6(1): 3-73.
[11]	Wen Q, Gao J, Song X, et al. RobustSTL: A robust sea-sonal-trend decomposition algorithm for long time series[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(01): 5409-5416.
[12]	Wen Q, Zhang Z, Li Y, et al. Fast RobustSTL: Efficient and robust seasonal-trend decomposition for time series with complex patterns[C]// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Disco-very & Data Mining, 2020: 2203-2213.
[13]	Wen Q, He K, Sun L, et al. RobustPeriod: Robust Time-Frequency Mining for Multiple Periodicity Detection[C]// Proceedings of the 2021 International Conference on Management of Data, 2021: 2328-2337.
[14]	Chen Yingying, Mahajan R, Sridharan B, et al. A prov-ider-side view of web search response time[J]. ACM SIGCOMM Computer Communication Review, 2013, 43(4): 243-254. doi: 10.1145/2534169.2486035
[15]	Lee S B, Pei Dan, Hajiaghayi M T, et al. Threshold com-pression for 3g scalable monitoring[C]// 2012 Proceedings IEEE INFOCOM, IEEE, 2012: 1350-1358.
[16]	Mahimkar A, Ge Zihui, Wang Jia, et al. Rapid detection of maintenance induced changes in service performance[C]// Proceedings of the Seventh conference on emerging Networking EXperiments and Technologies, 2011: 1-12.
[17]	Liu Dapeng, Zhao Youjian, Xu Haowen, et al. Oppren-tice: Towards practical and automatic anomaly detection through machine learning[C]// Proceedings of the 2015 Internet Measurement Conference, 2015: 211-224.
[18]	Wang J, Jing Y, Qi Q, et al. ALSR: An adaptive label screening and relearning approach for interval-oriented anomaly detection[J]. Expert Systems with Applications, 2019, 136: 94-104. doi: 10.1016/j.eswa.2019.06.028
[19]	Zhang Y, Zhu Y, Li X, et al. Anomaly Detection Based on Mining Six Local Data Features and BP Neural Network[J]. Symmetry, 2019, 11(4): 571. doi: 10.3390/sym11040571
[20]	Bu Jiahao, Liu Ying, Zhang Shenglin, et al. Rapid depl-oyment of anomaly detection models for large number of emerging kpi streams[C]// 2018 IEEE 37th International Performance Computing and Communications Confer-ence (IPCCC), IEEE, 2018: 1-8.
[21]	ZHANG S, ZHAO C, SUI Y, et al. Robust KPI Anomaly Detection for Large-Scale Software Services with Partial Labels[C]// 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), Wuhan, China: IEEE, 2021: 103-114.
[22]	Qian J, Liu F, Li D, et al. Large-scale kpi anomaly dete-ction based on ensemble learning and clustering[J]. Journal of Cybersecurity, 2020, 2(4): 157-166.
[23]	Ding Zhiguo, Fei Minrui. An anomaly detection approach based on isolation forest algorithm for streaming data using sliding window[J]. IFAC Proceedings Volumes, 2013, 46(20): 12-17.
[24]	Jiang Jianmin, Yasakethu L. Anomaly detection via one class svm for protection of scada systems[C]// 2013 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, IEEE, 2013: 82-88.
[25]	Amer M, Goldstein M, Abdennadher S. Enhancing one-class support vector machines for unsupervised anomaly detection[C]// Proceedings of the ACM SIGKDD workshop on outlier detection and description, 2013: 8-15.
[26]	Munir M, Siddiqui S A, Dengel A, et al. DeepAnT: A deep learning approach for unsupervised anomaly detection in time series[J]. IEEE Access, 2018, 7: 1991-2005. doi: 10.1109/ACCESS.2018.2886457
[27]	Munir M, Siddiqui S A, Chattha M A, et al. FuseAD: unsupervised anomaly detection in streaming sensors data by fusing statistical and deep learning models[J]. Sensors, 2019, 19(11): 2451. doi: 10.3390/s19112451
[28]	Xu Haowen, Chen Wenxiao, Zhao Nengwen, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications[C]// Proce-edings of the 2018 World Wide Web Conference, 2018: 187-196.
[29]	Li Zeyan, Chen Wenxiao, Pei Dan. Robust and unsuper-vised kpi anoALSmaly detection based on conditional variational autoencoder[C]// 2018 IEEE 37th International Performance Computing and Commun-ications Confer-ence (IPCCC), IEEE, 2018: 1-9.
[30]	YU G, CAI Z, WANG S, et al. Unsupervised Online Anomaly Detection With Parameter Adaptation for KPI Abrupt Changes[J]. IEEE Transactions on Network and Service Management, 2020, 17(3): 1294-1308. doi: 10.1109/TNSM.2019.2962701
[31]	WANG Y, WANG Z, XIE Z, et al. Practical and White-Box Anomaly Detection through Unsupervised and Act-ive Learning[C]// 2020 29th International Conference on Computer Communications and Networks (ICCCN), Honolulu, HI, USA: IEEE, 2020: 1-9.
[32]	ZHAO Y, ZHANG X, SHANG Z, et al. A Novel Hybrid Method for KPI Anomaly Detection Based on VAE and SVDD[J]. Symmetry, 2021, 13(11): 2104. doi: 10.3390/sym13112104
[33]	NIE L, ZHAO L, LI K. Robust Anomaly Detection Using Reconstructive Adversarial Network[J]. IEEE Transa-ctions on Network and Service Management, 2021, 18 (2): 1899-1912.
[34]	YOKKAMPON U, CHUMKAMON S, MOWSHOWITZ A, et al. Improved Variational Autoencoder Anomaly Detection in Time Series Data[C]// 2020 IEEE Inter-national Conference on Systems, Man, and Cybernetics (SMC), Toronto, ON, Canada: IEEE, 2020: 82-87.
[35]	NIU Z, YU K, WU X. LSTM-Based VAE-GAN for Time-Series Anomaly Detection[J]. Sensors, 2020, 20 (13): 3738. doi: 10.3390/s20133738
[36]	GEIGER A, LIU D, ALNEGHEIMISH S, et al. TadGAN: Time Series Anomaly Detection Using Generative Adver-sarial Networks[C]// 2020 IEEE International Conference on Big Data (Big Data), Atlanta, GA, USA: IEEE, 2020: 33-43.
[37]	YE H, MA X, PAN Q, et al. An Adaptive Approach for Anomaly Detector Selection and Fine-Tuning in Time Series[J]. arXiv:1907.07843 [cs, stat], 2019.
[38]	REN H, XU B, WANG Y, et al. Time-Series Anomaly Detection Service at Microsoft[C]// Proceedings of the 25th ACM SIGKDD International Conference on Know-ledge Discovery & Data Mining, Anchorage AK USA: ACM, 2019: 3009-3017.
[39]	Qiu Juan, Qingfeng Du, Wei Wang, Kanglin Yin, and Liang Chen. Short-term performance metrics forecasting for virtual machine to support anomaly detection using hybrid ARIMA-WNN model[C]. In 2019 IEEE 43rd Ann-ual Computer Software and Applications Conference (CO-MPSAC), IEEE, 2019,vol. 2,pp.330-335.
[40]	LAN Z, XU L, FANG W. FDNN: Feature-based Deep Neural Network Model for Anomaly Detection of KPIs[C]//2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS), Beijing, China: IEEE, 2019: 286-289.
[41]	JIANG W, HONG Y, ZHOU B, et al. A GAN-Based Ano-maly Detection Approach for Imbalanced Industrial Time Series[J]. IEEE Access, 2019, 7: 143608-143619. doi: 10.1109/ACCESS.2019.2944689
[42]	Duan X Y, Chen N J, Xie Y S. Intelligent Detection of Large-Scale KPI Streams Anomaly Based on Transfer Learning[C]// CCF Conference on Big Data. Springer, Singapore, 2019: 366-379.
[43]	Chen W, Xu H, Li Z, et al. Unsupervised anomaly dete-ction for intricate kpis via adversarial training of vae[C]// IEEE INFOCOM 2019-IEEE Conference on Computer Communications, IEEE, 2019: 1891-1899.
[44]	Shi J, He G, Liu X. Anomaly detection for key perfor-mance indicators through machine learning[C]// 2018 International Conference on Network Infrastructure and Digital Content (IC-NIDC), IEEE, 2018: 1-5.
[45]	Al Mamun S M, Beyaz M. LSTM recurrent neural net-work (RNN) for anomaly detection in cellular mobile net-works[C]// International Conference on Machine Learning for Networking, Springer, Cham, 2018: 222-237.
[46]	Chandola V, Banerjee A, Kumar V. Anomaly detection: A survey[J]. ACM computing surveys (CSUR), 2009, 41(3): 1-58.
[47]	Shon T, Moon J. A hybrid machine learning approach to network anomaly detection[J]. Information Sciences, 2007, 177(18): 3799-3821. doi: 10.1016/j.ins.2007.03.025
[48]	Zhang J, Wang Z, Yuan J, et al. Positive and unlabeled learning for anomaly detection with multi-features[C]// Proceedings of the 25th ACM international conference on Multimedia, 2017: 854-862.
[49]	Li Zhihan, Zhao Youjian, Liu Rong, et al. Robust and rapid clustering of kpis for large-scale anomaly detection[C]// 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), IEEE, 2018: 1-10.
[50]	Qiu J, Du Q, Qian C. Kpi-tsad: A time-series anomaly detector for kpi monitoring in cloud applications[J]. Sy-mmetry, 2019, 11(11): 1350.
[51]	Paparrizos J, Gravano L. k-shape: Efficient and accu-rate clustering of time series[C]// Proceedings of the 2015 ACM SIGMOD International Conference on Managem-ent of Data, 2015: 1855-1870.
[52]	Zhang Shenglin, Lin Xiaofei, Sun Yongqian, et al. Rese-arch on Unsupervised KPI Anomaly Detection Based on Deep Learning[J]. Frontiers of Data&Computing, 2020, 2(3): 87-100.

	需要标注	人工调参	方法技术	适用KPI类型	输出	发表年份
Opprentice	√	否	随机森林	通用	离散值	2015
ADS	√	否	聚类和随机森林	通用	离散值	2018
PUAD	√	否	聚类和PU学习^[48]	通用	离散值	2021
iForest	×	部分	孤立森林	通用	离散值	2013
OCSVM	×	部分	支持向量机	通用	离散值	2013
Donut	×	否	深度生成模型	周期	连续值	2018
Bagel	×	否	深度生成模型	周期	连续值	2018
DeepAnT	×	否	卷积神经网络	周期	连续值	2018
FuseAD	×	否	统计和卷积神经网络	周期	连续值	2019
ARIMA	×	是	统计	通用	离散值	2005
EWMA	×	是	统计	通用	离散值	2003
Wavelet	×	是	统计	通用	离散值	2002
Holt-Winters	×	是	统计	通用	离散值	2012

数据集	时间间隔（分钟）	KPI曲线数量	数据点数量	异常点数量	异常点占比（%）
A	1	12	3145250	58436	1.86
B	1	7	1679741	51082	3.04
C	5	97	838064	2799	0.33

		A			B			C
	Precision	Recall	F1-score	Precision	Recall	F1-score	Precision	Recall	F1-score
ARIMA	0.91	0.55	0.68	0.57	0.55	0.56	0.70	0.81	0.75
EWMA	0.91	0.55	0.69	0.40	0.55	0.46	0.70	0.82	0.75
Wavelet	0.69	0.40	0.51	0.47	0.42	0.44	0.43	0.77	0.55
Holt-winters	0.60	0.51	0.55	0.25	0.34	0.29	0.62	0.77	0.69
Opprentice	0.72	0.69	0.70	0.98	0.85	0.91	0.76	0.83	0.79
ADS	0.55	0.74	0.63	0.60	0.73	0.66	0.81	0.96	0.88*
PUAD	0.88	0.77	0.82*	0.91	0.96	0.93*	0.78	0.95	0.86
iForest	0.54	1	0.70	0.46	1	0.63	0.30	0.89	0.45
OCSVM	0.40	0.99	0.57	0.31	0.98	0.47	0.28	1	0.44
DeepAnT	0.79	0.77	0.78	0.90	0.97	0.93*	0.69	0.90	0.78
FuseAD	0.72	0.69	0.71	0.81	0.84	0.83	0.69	0.94	0.79
Bagel	0.74	0.82	0.78	0.62	0.82	0.71	0.83	0.87	0.85
Donut	0.82	0.79	0.80	0.80	0.80	0.80	0.84	0.90	0.87

KPI异常检测方法评估

Evaluation of KPI Anomaly Detection Methods

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 52

相关文章 13

编辑推荐

Metrics

本文评价

[1]	赵忠斌,蔡满春,芦天亮. 融合多头注意力机制的网络恶意流量检测[J]. 数据与计算发展前沿, 2022, 4(5): 60-67.
[2]	危婷,张宏海,蔺小丽,张蕾蕾,王妍,贾金峰. 云服务网站用户复访行为预测模型研究[J]. 数据与计算发展前沿, 2022, 4(3): 124-130.
[3]	张怡宁,何洪波,王闰强. 热门数字音频预测技术综述[J]. 数据与计算发展前沿, 2021, 3(4): 81-92.
[4]	蒲剑苏,朱正国,邵慧,高博洋,朱焱麟,闫宗楷,向勇. 基于可视化的固态电解质材料机器学习筛选与预测[J]. 数据与计算发展前沿, 2021, 3(4): 18-29.
[5]	张舒莹,韩鑫胤,何小雨,袁丹阳,栾海晶,李瑞琳,何佳茵,牛北方. 基于机器学习的基因组微卫星状态探测方法综述[J]. 数据与计算发展前沿, 2021, 3(3): 126-135.
[6]	肖建平,龙春,赵静,魏金侠,胡安磊,杜冠瑶. 基于深度学习的网络入侵检测研究综述[J]. 数据与计算发展前沿, 2021, 3(3): 59-74.
[7]	郭佳龙,王宗国,王彦棡,赵旭山,宿彦京,刘志威. 基于计算机技术的材料研发方法概述[J]. 数据与计算发展前沿, 2021, 3(2): 120-132.
[8]	任荟颖,王婧,王彦棡. 基于AutoML的湍流建模[J]. 数据与计算发展前沿, 2020, 2(4): 121-131.
[9]	张圣林,林潇霏,孙永谦,张玉志,裴丹. 基于深度学习的无监督KPI异常检测[J]. 数据与计算发展前沿, 2020, 2(3): 87-100.
[10]	李姿昕,张能,熊斌,胡云凤,赵新鹏,黄海友. 材料科学数据库在材料研发中的应用与展望[J]. 数据与计算发展前沿, 2020, 2(2): 78-90.
[11]	钱旭,田子奇. 材料基因方法在材料设计中的应用[J]. 数据与计算发展前沿, 2020, 2(1): 128-141.
[12]	储中明, 肖邓杰, 乔予思, 万金宇. 机器学习在粒子加速器的应用[J]. 数据与计算发展前沿, 2019, 1(2): 110-120.
[13]	张智鹏,江佳伟,余乐乐,崔斌. Angel ⁺ : 基于Angel的分布式机器学习平台[J]. 数据与计算发展前沿, 2019, 1(1): 63-72.