Analysis and Empirical Research on Performance Evaluation of Parallel Computing Software in Heterogeneous Systems

doi:10.11871/jfdc.issn.2096-742X.2024.03.013

Abstract

Abstract:

[Objective] The Performance evaluation of parallel computing software has always been an important research direction in the field of supercomputing. Real evaluation and analysis of the actual performance of computing software on heterogeneous systems can effectively promote the healthy development of the computing software ecosystem in heterogeneous systems. [Methods] This article first conducts research and analysis on parallel computing software performance evaluation methods, both domestically and internationally, through literature research, and summarizes three important stages of industry research on parallel computing software performance evaluation. Empirical analysis is conducted on the real performance of the software through the Cannon algorithm for parallel computation of matrix product, and multi-dimensional experimental analysis is conducted on important indicators such as execution time and efficiency. [Results] Under the same node, the execution time is not always reduced with more accelerator cards used. In the cases of using single or none accelerator card, the parallel efficiency of programs dealing with different matrix scales does not change significantly with the increase of nodes. [Conclusions] In heterogeneous system, the computing software focusing only on parallel efficiency cannot truly reflect the actual level of software performance. In addition to the efficiency factor of inter-node parallelism, intra-node acceleration has also become an important evaluation indicator reflecting the true level of parallel computing software.

Key words: heterogeneous systems, parallel software, performance evaluation, parallel efficiency

GU Beibei, QIU Jiyan, CHI Xuebin. Analysis and Empirical Research on Performance Evaluation of Parallel Computing Software in Heterogeneous Systems[J]. Frontiers of Data and Computing, 2024, 6(3): 116-126, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.03.013.

Figures/Tables 10

Fig.1

Table 1

Table 2

Table 3

Table 4

Table 5

Fig.2

Table 6

Table 7

Fig.3

References 33

[1]	迟学斌, 王彦棡, 王珏, 等. 并行计算与实现技术[M]. 北京: 科学出版社, 2015: 7.
[2]	MARCO Z, BROND L, STEVE T, et al. Performance Analysis Using the MIPS R10000 Performance Counters[C]. Proceedings of the 1996 ACM/IEEE conference on Supercomputing (CDROM), 1996.
[3]	BROWNE S, DONGARRA J, GARNER N, et al. A Scalable Cross-Platform Infrastructure for Application Performance Tuning Using Hardware Counters[C]. ACM/IEEE SC 2000 Conference, 2000.
[4]	RUDOLPH B, ZIEGLER H. PCL-the Performance Counter Library: A Common Interface to Access Hardware Performance Counters on Microprocessors[Z]. Germany: Research Centre Juelich GmbH, 1993.
[5]	LUC S. System and Kernel Thread Performance Monitor API Reference Guide[R]. BM RS/6000 Division, 1999.
[6]	GERNDT M, KEREKU E. Periscope: Advanced Techniques for Performance Analysis[C]. Parallel Computing: Current & Future Issues of High-End Computing, Proceedings of the International Conference ParCo 2005, 2005.
[7]	FAHRINGER T, GERNDT M. Specification of Performance Problems in MPI-Programs with ASL[C]. International Conference on Parallel Processing (ICPP’00), 2000.
[8]	JOSEPH E, WILLARD C. A new HPC technical computing benchmark: the IDC balanced rating[C]. IDC Bulletin W, 2000.
[9]	BKDG. A Programming Based Performance Counter Statistical Method for Modern High end Microprocessors[EB/OL]. [2010-04-12]. http://support.amd.com/us/Processor_TechDocs.
[10]	TALLENT N, MELLOR J, ADHIANTO L, et al. “HPCToolkit: performance tools for scientific computing.”[C]. Journal of Physics: Conference Series, 2008.
[11]	Open SpeedShop, Performance measurement tools..[EB/OL]. [2010-04-12]. http://www.openspeedshop.org/wp/.
[12]	PAPI, Performance measurement tools. [EB/OL]. [2010-04-12]. http://icl.cs.utk.edu/papi/.
[13]	SHENDE S, MALONY A. “The Tau Parallel Performance System.”[J]. International Journal of High Performance Computing Applications, 2006, 20(2): 287-311.
[14]	MARTIN B, KIM B D, JEFF D, et al. PerfExpert: An Easy-to-Use Performance Diagnosis Tool for HPC Applications[C]. 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, 2010.
[15]	GERNDT M, MOHR B, LARSSON J. Evaluating OpenMP Performance Analysis Tools with the APART Test Suite[C]. Fifth European Workshop on OpenMP (EWOMP’03), 2003.
[16]	GEIMER M, WOLF F, WYLIE B J, et al. The Scalasca Performance Toolset Architecture. Concurrency and Computation[J]. Practice and Experience, 2010, 22(6), 702-719.
[17]	KNUPFER A, BRUNST H, DOLESCHAL J, et al. The Vampir Performance Analysis Tool Set. In: Tools for High Performance Computing[J]. Springer, Berlin, 2008, 139-155.
[18]	SHENDE S S, MALONY A D. The TAU Parallel Performance System[J]. International Journal of High Performance Computing Applications. 2006, 20(2), 287-311.
[19]	DIETER M, SCOTT B. Score-P-A Unified Performance Measurement System for Petascale Applications[C]. Competence in High Performance Computing, 2010.
[20]	DONGARRA J, GANNON D, FOX G, et al. The Impact of Multicore on Computational Science Software[J]. ResearchGate, 2007, 3: 3-10.
[21]	JEFF D, MARTIN B. Evaluation and Optimization of Multicore Performance Bottlenecks in Supercomputing Applications[C]. IEEE International Symposium on Performance Analysis of Systems and Software, 2011.
[22]	CHEN D, SINGH D. Fractal video compression in opencl: An evaluation of cpus, gpus, and fpgas as acceleration platforms[C]. Asia and South Pacific Design Automation Conference (ASP-DAC), 2013.
[23]	BAKHODA A, YUAN G. Analyzing cuda workloads using a detailed gpu simulator[C]. IEEE International Symposium on Performance Analysis ofSystems and Software (ISPASS), 2009.
[24]	BOR K. K, SU B Y. clspmv: A cross-platform opencl spmv framework on gpus[C]. ACM international conference on Supercomputing (ICS), 2012.
[25]	JOHN C, SETH K, et al. Performance analysis framework for high-level language applications in reconfigurable computing[C]. ACMTrans. Reconfigurable Technology, 2010.
[26]	SILVA B, BRAEKEN A. Performance modeling for fpgas: Extending the roofline model with high-level synthesis tools[C]. International Journal ofReconfigurable Computing (IJRC), 2013.
[27]	ALTERA. Implementing fpga design with the opencl standard[R]. Altera Whitepaper, 2011.
[28]	SUBIN B, AJEESH S. Automated Performance Modeling of HPC Applications Using Machine Learning[C]. IEEE Transactions on Computers, 2020.
[29]	OZAN T, EMRE A. Diagnosing Performance Variations in HPC Applications Using Machine Learning[C]. Lecture Notes in Computer Science, 2017.
[30]	JONATHAN R, JACK D. Timemory: Modular Performance Analysis for HPC[C]. Lecture Notes in Computer Science, 2020.
[31]	ZHOU K, MARK W K. Tools for top-down performance analysis of GPU-accelerated applications[C]. Proceedings of the 34^th ACM International Conference on Supercomputing, 2020.
[32]	LARISSA S, MARCIN C. Performance-Detective: Automatic Deduction of Cheap and Accurate Performance Models[C]. International Conference on Supercomputing 2022, 2022.
[33]	FELIPE T, ALVARO W. Scalable performance analysis method for SPMD applications[J]. The Journal of Supercomputing, 2022 (78): 19346-19371.

软件/工具	支持硬件	支持语言	是否需要插装	是否需要重编译	支持并行库	依赖库
IPM	CPU、GPU		否	必须	MPI、openMP	papi
PAPI	是一个接口、具体实现各异		可选	可选		GCC等编译器
Timemory	CPU、GPU	C/C++、Python、CUDA、HIP	必须	必须	MPI、openMP	GCC等编译器
PerfExpert	CPU	C/C++	否	否	MPI、openMP	PAPI、HPCToolkit
ScoreP+Vampir	CPU、GPU	C/C++、Fortran CUDA	否	必须	MPI、SHMEM、openMP、Pthread、OpenACC、OpenCL	GCC等编译器
Extrae+Paraver	CPU、GPU	C/C++、Python CUDA	无	无	MPI、 openMP、 Pthread、	Boost、dyninst、libunwind、papi、wxwidgets

gpu-per-node	运行时间（ms）
gpu-per-node	1node	4 node	9 node	16 node
0	207,787	61,900	52,268	47,465
1	404	1,477	2,410	3,832
2	402	1,206	2,685	6,632
3	404	1,148	2,906	4,320
4	405	1,187	4,412	5,933

gpu-per-node	运行时间（ms）
gpu-per-node	1node	4 node	9 node	16 node
0	491,452	140,683	119,318	107,123
1	795	2,446	3,522	6,368
2	796	2,163	3,815	8,461
3	796	1,992	4,190	8,638
4	798	2,044	4,980	6,393

gpu-per-node	运行时间（ms）
gpu-per-node	1node	4 node	9 node	16 node
0	959,978	272,543	228,397	203,791
1	1,375	4,158	5,754	8,996
2	1,374	3,400	5,248	10,842
3	1,377	3,121	4,651	7,422
4	1,379	3,103	5,046	8,278

gpu-per-node	运行时间（ms）
gpu-per-node	1node	4 node	9 node	16 node
0	1,653,833	451,597	387,890	344,775
1	2,124	6,074	7,677	12,617
2	2,123	4,959	6,919	14,441
3	2,129	4,564	7,734	9,615
4	2,124	4,531	6,617	11,206