面向SWF日志事件流数据的可视分析系统

doi:10.11871/jfdc.issn.2096-742X.2021.04.001

数据与计算发展前沿 ›› 2021, Vol. 3 ›› Issue (4): 3-17.

doi: 10.11871/jfdc.issn.2096-742X.2021.04.001

• 可视化与可视分析专题 • 上一篇下一篇

面向SWF日志事件流数据的可视分析系统

李玥¹(),杨波¹(),芦旭熠^1,²(),单桂华^1,^*()

1.中国科学院计算机网络信息中心,北京 100190
2.中国科学院大学,北京 100049

收稿日期:2021-06-22 出版日期:2021-08-20 发布日期:2021-08-30
通讯作者: 单桂华
作者简介:李玥,中国科学院计算机网络信息中心,助理工程师,主要研究方向为数据可视化与可视分析。
本文承担工作为：可视化开发与核心算法实现。
LI Yue is an assistant engineer in Com-puter Network Information Center, Chinese Academy of Sciences. His main research interests are visualization and visual analysis.
In this paper, he undertakes the following tasks: visualization development and key algorithms implementation.
E-mail: li1023@cnic.cn|杨波,中国科学院计算机网络信息中心,助理研究员,主要研究方向为可视化与可视分析。
本文承担工作为：任务及需求分析、系统框架设计。
YANG Bo is an assistant professor in Computer Network Information Center, Chinese Academy of Sciences. His main research interests are visualization and visual analysis.
In this paper, he undertakes the following tasks: task and de-mand analysis, system framework implementation.
E-mail: yangbo@cnic.cn|芦旭熠,中国科学院计算机网络信息中心,硕士研究生,主要研究方向为科学可视化、可视分析。
本文承担工作为：技术调研及相关部分撰写。
LU Xuyi is a postgraduate student in Computer Network Information Center, Chinese Academy of Sciences. Her main research interests include scientific visualization and visual analysis.
In this paper, she undertakes the following tasks: the technical investigation and paper writing.
E-mail: luxuyi@cnic.cn|单桂华,中国科学院计算机网络信息中心,研究员,主要研究方向为可视化与可视分析、智能交互。
本文承担工作为：架构设计、研究指导。
SHAN Guihua is a professor in Compu-ter Network Information Center, Chinese Academy of Sciences. Her main research interests are visualization and visual analysis, intelligent interaction.
In this paper, she undertakes the following tasks: the structure design and research guidance.
E-mail: sgh@cnic.cn
基金资助:
战略性先导科技专项(A类XDA19080102)

A Visual Analysis System for SWF Log Event Stream Data

LI Yue¹(),YANG Bo¹(),LU Xuyi^1,²(),SHAN Guihua^1,^*()

1. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China
2. University of Chinese Academy of Sciences, Beijing 100049, China

Received:2021-06-22 Online:2021-08-20 Published:2021-08-30
Contact: SHAN Guihua

摘要/Abstract

摘要：

【目的】为解决高性能计算集群的日志数据交互可视分析的问题,提供一套支持多种不同集群系统日志数据的可视分析框架,本文设计并实现了面向SWF日志数据的事件流可视分析系统SWFVis。【方法】SWFVis基于时间事件序列数据可视分析技术,结合多维可视分析、关联可视分析等技术,提出了一套针对SWF日志中的时序属性、关联属性、多维属性进行综合可视分析的方法。【结果】针对公开的iPSC/860与ForHLR II集群产生的真实数据,SWFVis直观展示了日志中整体作业的并发状态、执行情况,并对集群中存在的作业处理模式进行分析。【局限】对于事件流可视化呈现效果,在三维场景中存在的视角遮挡、分布杂乱的现象,未来考虑通过层次聚类等方法实现进一步的优化。【结论】SWFVis提供了针对SWF日志数据的交互式可视分析方法,支持多种计算机集群使用。通过对作业间存在的时序性、关联性进行展示,以及对作业多维属性进行挖掘,可以直观呈现集群工作状态、用户提交行为,并交互式发掘集群作业中存在的处理模式,为集群调度优化提供支撑。

关键词: 高性能计算集群, 时间事件序列, 事件流, 可视分析, SWF日志

Abstract:

[Objective] HPC cluster log data is typically presented by traditional statistical charts, which has the disadvantage of no intuitive presentation and no interactivity. We design and implement SWFVis, an event stream visual analysis system for SWF log data to improve the presentation of the log data. [Methods] Based on the visual analysis technology of temporal event sequence data, combined with multi-dimensional visual analysis and associated visual analysis technology, SWFVis proposes a comprehensive visual analysis method for the sequential and multidimensional attributes of the jobs in the log event stream. [Results] Based on the real data generated by the public IPSC /860 and FORHLR II clusters, SWFVis intuitively displays the concurrent status and execution of the overall job in the logs, and analyzes the job processing patterns in the clusters. [Limitations] As for the visual rendering effect of event stream, there are still some problems in the three-dimensional scenes such as view blocking and disordered distribution. In the future, we consider achieving hierarchical clustering and other methods for further optimization. [Conclusions] SWFVis provides an interactive visual analysis method for SWF log data. It intuitively presents the working state of the cluster and user submission behavior, and supports interactive exploration of the processing pattern in the cluster operation, which provides support for the optimization of cluster scheduling.

Key words: HPC cluster, temporal event sequence data, event stream, visual analysis, SWF log

李玥,杨波,芦旭熠,单桂华. 面向SWF日志事件流数据的可视分析系统[J]. 数据与计算发展前沿, 2021, 3(4): 3-17.

LI Yue,YANG Bo,LU Xuyi,SHAN Guihua. A Visual Analysis System for SWF Log Event Stream Data[J]. Frontiers of Data and Computing, 2021, 3(4): 3-17.

图/表 11

图1

图2

图3

图4

图5

图6

图7

图8

图9

图10

图11

参考文献 22

[1]	Guo S N, Xu K, Zhao R W, et al. EventThread: visual summarization and stage analysis of event sequence data[J]. IEEE Transactions on Visualization and Computer Graphics, 2018,24(1):56-65. doi: 10.1109/TVCG.2017.2745320
[2]	Monroe M, Lan R J, Lee H, et al. Temporal event sequence simplification[J]. IEEE Transactions on Visualization and Computer Graphics, 2013,19(12):2227-2236. doi: 10.1109/TVCG.2013.200
[3]	LIU Z, WANG Y, DONTCHEVA M, et al. Patterns and sequences: Interactive exploration of clickstreams to un-derstand common visitor paths[J]. IEEE Transactions on Visualization and Computer Graphics, 2016,23(1):321-330. doi: 10.1109/TVCG.2016.2598797
[4]	CHEN Y, XU P, REN L. Sequence synopsis: Optimize visual summary of temporal event data[J]. IEEE transac-tions on visualization and computer graphics, 2017,24(1):45-55.
[5]	B H Park, Y Hui, S Boehm, R A Ashraf, C Layton, C En-gelmann. A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log[C] //2018 IEEE International Conference on Cluster Computing (CLUSTER), 2018:571-579.
[6]	S Gupta, T Patel, C Engelmann, D Tiwari. Failures in large scale systems: Long-term measurement, analysis, and implications[C] //Proceedings of the International Conference for High Performance Computing, Networ-king, Storage and Analysis, ser. SC'17. New York, NY, USA: ACM, 2017: 44:1-44:12.
[7]	T Li, Y Jiang, C Zeng, B Xia, Z Liu, W Zhou, X Zhu, W Wang, L Zhang, J Wu, L Xue, D Bao. FLAP: An end-to-end event log analysis platform for system management[C] //Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD '17. New York, NY, USA: ACM, 2017: 1547-1556.
[8]	B H Park, S Hukerikar, R Adamson, C Engelmann. Big data meets HPC log analytics: Scalable approach to under-standing systems at extreme scale[C] //IEEE Cluster 2017 at Workshop on Monitoring and Analysis for High Perfor-mance Computing Systems Plus Applications, Sept 2017: 758-765.
[9]	Dror G, Feitelson, Dan Tsafrir, David Krakov. Experience with using the Parallel Workloads Archive[J]. Journal of Parallel and Distributed Computing, 2014,74(10):2967-2982. doi: 10.1016/j.jpdc.2014.06.013
[10]	彭燕妮, 樊晓平, 赵颖, 周芳芳. 时间事件序列数据可视化综述[J]. 计算机辅助设计与图形学学报, 2019,31(10):1698-1710.
[11]	Huang D D, Tory M, Staub-French S, et al. Visualization techniques for schedule comparison[J]. Computer Graphics Forum, 2009,28(3):951-958. doi: 10.1111/cgf.2009.28.issue-3
[12]	Han Y, Rozga A, Dimitrova N, et al. Visual analysis of pro-ximal temporal relationships of social and communicative behaviors[J]. Computer Graphics Forum, 2015,34(3):51-60. doi: 10.1111/cgf.12617
[13]	Krstajic M, Bertini E, Keim D A. CloudLines: compact display of event episodes in multiple time-series[J]. IEEE Transactions on Visualization and Computer Graphics, 2011,17(12):2432-2439. doi: 10.1109/TVCG.2011.179
[14]	Wongsuphasawat K, Gomez J A G, Plaisant C, et al. LifeFlow: visualizing an overview of event sequences[C]. Proceedings of the SIGCHI Conference on Human Fac-tors in Computing Systems. New York: ACM Press, 2011: 1747-1756.
[15]	Wongsuphasawat K, Gotz D. Exploring flow, factors, and outcomes of temporal event sequences with the outflow visualization[J]. IEEE Transactions on Visualization and Computer Graphics, 2012,18(12):2659-2668. doi: 10.1109/TVCG.2012.225
[16]	Gotz D, Wongsuphasawat K. Interactive intervention analysis[C] //AMIA Annual Symposium Proceedings, 2012: 274-280.
[17]	Tanahashi Y, Hsueh C H, Ma K L. An efficient frame-work for generating storyline visualizations from strea-ming data[J]. IEEE Transactions on Visualization and Computer Graphics, 2015,21(6):730-742. doi: 10.1109/TVCG.2015.2392771
[18]	Ogawa M, Ma K L. Software evolution storylines[C] //Proceedings of the 5th International Symposium on Soft-ware Visualization. New York: ACM Press, 2010: 35-42.
[19]	Tanahashi Y, Ma K L. Design considerations for optimi-zing storyline visualizations[J]. IEEE Transactions on Visualization and Computer Graphics, 2012,18(12):2679-2688. doi: 10.1109/TVCG.2012.212
[20]	Liu S X, Wu Y C, Wei E X, et al. StoryFlow: tracking the evolution of stories[J]. IEEE Transactions on Visualiza-tion and Computer Graphics, 2013,19(12):2436-2445.
[21]	Perer A, Sun J M. MatrixFlow: temporal network visual analytics to track symptom evolution during disease progre-ssion[C] // AMIA Annual Symposium Proceedings, 2012: 716-725.
[22]	Zhao J, Liu Z C, Dontcheva M, et al. MatrixWave: visual comparison of event sequence data[C]. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. New York: ACM Press, 2015: 259-268.

面向SWF日志事件流数据的可视分析系统

A Visual Analysis System for SWF Log Event Stream Data

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 22

相关文章 2

编辑推荐

Metrics

本文评价

[1]	赵凡,马小东,任芃锟. 网吧记录数据可视分析系统SRDVis[J]. 数据与计算发展前沿, 2021, 3(4): 70-80.
[2]	蒲剑苏,朱正国,邵慧,高博洋,朱焱麟,闫宗楷,向勇. 基于可视化的固态电解质材料机器学习筛选与预测[J]. 数据与计算发展前沿, 2021, 3(4): 18-29.