Porting and Optimizing Gadget-2 on a Heterogeneous Accelerator Platform

doi:10.11871/jfdc.issn.2096-742X.2022.05.012

Abstract

Abstract:

[Objective] In this paper, we present the work on porting the parallel cosmological N-body simulation software Gadget-2 to a homegrown heterogeneous accelerator platform, based on the BH-Tree and the Particle-Mesh methods. [Methods] The most time-consuming part, computation of short-range force, is ported to the accelerator with HIP, including the traversal of the local tree. The Structure of Arrays is reconstructed, the register and shared memory are fully utilized to improve the efficiency of memory access on the device. [Results] Numerical results show that the performance of the optimized software and the part of short-range force computation are accelerated up to 13.27 times and 35.67 times, respectively. The parallel efficiency reaches 57.29%. The optimized version is validated by the power spectrum. [Conclusions] The cosmological N-body simulations software Gadget-2 is ported and optimized on a heterogeneous accelerator platform, which can support large-scale cosmological simulation.

Key words: N-body problem, TreePM method, Heterogeneous Accelerator Platform, HIP

ZHAO Wenlong,WANG Wu. Porting and Optimizing Gadget-2 on a Heterogeneous Accelerator Platform[J]. Frontiers of Data and Computing, 2022, 4(5): 108-119, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2022.05.012.

Figures/Tables 12

Fig.1Part

Fig.2

Table 1

Fig. 3

Fig.4

Fig.5

Fig.6

Table 2

Fig.7

Table 3

References 29

[1]	Ishiyama T, Prada F, Klypin A A, et al. The Uchuu sim-ulations: Data Release 1 and dark matter halo concentr-ations[J]. Monthly Notices of the Royal Astronomical Society, 2021, 506(3): 4210-4231. doi: 10.1093/mnras/stab1755
[2]	Cheng S, Yu H R, Inman D, et al. CUBE-Towards an Optimal Scaling of Cosmological N-body Simulations[C]. 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID). IEEE, 2020: 685-690.
[3]	Wang Q, Cao Z Y, Gao L, et al. PHoToNs-A parallel heterogeneous and threads oriented code for cosmological N-body simulation[J]. Research in Astronomy and Astr-ophysics, 2018, 18(6): 062.
[4]	Yahagi H, Yoshii Y. N-body code with adaptive mesh refinement[J]. The Astrophysical Journal, 2001, 558(1): 463. doi: 10.1086/322457
[5]	Barnes J, Hut P. A hierarchical O(NlogN) force-calculation algorithm[J]. nature, 1986, 324(6096): 446-449. doi: 10.1038/324446a0
[6]	Greengard L, Rokhlin V. A fast algorithm for particle simulations[J]. Journal of computational physics, 1987, 73(2): 325-348. doi: 10.1016/0021-9991(87)90140-9
[7]	Hockney R W, Eastwood J W. article-particle-particle-mesh (P3M) algorithms[M]//Computer simulation using particles. Taylor & Francis, 1988: 267-304.
[8]	Bode P, Ostriker J P, Xu G. The tree particle-mesh N-body gravity solver[J]. The Astrophysical Journal Supplement Series, 2000, 128(2): 561. doi: 10.1086/313398
[9]	Wang Q. A hybrid Fast Multipole Method for cosm-ological N-body simulations[J]. Research in Astronomy and Astrophysics, 2021, 21(1): 1-24 doi: 10.1088/1674-4527/21/1/1
[10]	Nylons L, Harris M, Prins J. Fast n-body simulation with cuda[M]// Hubert Nguyen. GPU Gems 3, Addison-Wesley Professional, 2007: 62-66.
[11]	Yokota R, Barba L A. Treecode and fast multipole me-thod for N-body simulation with CUDA[M]. GPU Com-puting Gems Emerald Edition. Morgan Kaufmann, 2011: 113-132.
[12]	Hamada T, Iitaka T. The chamomile scheme: An opti-mized algorithm for n-body simulations on programm-able graphics processing units[J]. eprint arXiv:astro-ph/0703100, 2007.
[13]	Hamada T, Nitadori K. 190 tflops astrophysical n-body simulation on a cluster of gpus[C]. SC'10: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, IEEE, 2010: 1-9.
[14]	Hamada T, Nitadori K, Benkrid K, et al. A novel mul-tiple-walk parallel algorithm for the Barnes-Hut treecode on GPUs-towards cost effective, high performance N-body simulation[J]. Computer science-research and deve-lopment, 2009, 24(1-2): 21-31.
[15]	Hamada T, Narumi T, Yokota R, et al. 42 TFlops hiera-rchical N-body simulations on GPUs with applications in both astrophysics and turbulence[C]. Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, 2009: 1-12.
[16]	Yokota R, Barba L A. Treecode and fast multipole me-thod for N-body simulation with CUDA[M]. GPU Co-mputing Gems Emerald Edition. Morgan Kaufmann, 2011: 113-132.
[17]	Potter D, Stadel J, Teyssier R. PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys[J]. Computational Astrophysics and Cosmology, 2017, 4(1): 1-13. doi: 10.1186/s40668-017-0020-2
[18]	Wang Q, Meng C. PhotoNs-GPU: A GPU accelerated cosmological simulation code[J]. Research in Astronomy and Astrophysics, 2021, 21(11): 270-296 doi: 10.1088/1674-4527/21/11/270
[19]	扶月月, 王武, 王乔. 基于 FMM-PM 方法的宇宙 N 体模拟在 GPU 上的实现和优化[J]. 数据与计算发展前沿, 2020, 2(2): 155-164.
[20]	Springel V. The cosmological simulation code GADGET-2[J]. Monthly notices of the royal astronomical society, 2005, 364(4): 1105-1134. doi: 10.1111/j.1365-2966.2005.09655.x
[21]	Ragagnin A, Dolag K, Wagner M, et al. Gadget3 on GPUs with OpenACC[J]. Parallel Computing: Tech-nology Trends, 2020, 36: 209-218.
[22]	Jafary B, Jha S, Fiondella L, et al. Data-Driven App-lication-Oriented Reliability Model of a High-Perfor-mance Computing System[J]. IEEE Transactions on Reliability, 2021:1-13. DOI: 10.1109/TR.2021.3085582. doi: 10.1109/TR.2021.3085582
[23]	Nori M, Baldi M. AX-GADGET: a new code for cosmo-logical simulations of Fuzzy Dark Matter and Axion models[J]. Monthly Notices of the Royal Astronomical Society, 2018, 478(3): 3935-3951. doi: 10.1093/mnras/sty1224
[24]	Springel V, Pakmor R, Zier O, et al. Simulating cosmic structure formation with the GADGET-4 code[J]. Mon-thly Notices of the Royal Astronomical Society, 2021, 506(2): 2871-2949.
[25]	Greengard L, Lee J Y. A direct adaptive Poisson solver of arbitrary order accuracy[J]. Journal of Computational Physics, 1996, 125(2): 415-424. doi: 10.1006/jcph.1996.0103
[26]	Bode P, Ostriker J P, Xu G. The tree particle-mesh N-body gravity solver[J]. The Astrophysical Journal Supplement Series, 2000, 128(2): 561. doi: 10.1086/313398
[27]	Bagla J S. TreePM: A code for cosmological N-body simulations[J]. Journal of Astrophysics and Astronomy, 2002, 23(3): 185-196. doi: 10.1007/BF02702282
[28]	Fatica M. Accelerating linpack with CUDA on heter-ogenous clusters[C]. Proceedings of 2nd Workshop on General Purpose Processing on Graphics Processing Unit, 2009: 46-51.
[29]	AMD, AMD ROCm Platform, 2020[Online]. Ava-ialbe:https://rocmdocs.amd.com/en/latest /index.html. Accessed 18 Sep 2021.

Function	Time/sec	Ratio/%	Description
find_next_sync_point/ every_timestep_stuff	6	0.6	时间同步转移粒子
domain_decomposition	51.5	5.3	区域分解
compute_accelerations	905.1	93.6	计算粒子加速度
force_ shortrange	860.6	89	计算短程加速度
other parts	4.4	0.5	计算新的时间步
all	967	100	总耗时

规模	Np	单步总耗时(s)		短程力耗时(s)
规模	Np	Gadget-2	G2H	Gadget-2	G2H
128³	4	14.10	1.06	13.47	0.49
256³	8	31.86	2.78	30.10	1.24
512³	16	243.99	18.57	228.87	6.42

进程数	8	16	32	64	128
Total(s)	35.11	18.57	10.24	5.64	3.83
ShortRange(s)	11.98	6.42	3.54	2.11	1.39