eMD: A Large-Scale Molecular Dynamics Simulation Software Based on Heterogeneous Computing

doi:10.11871/jfdc.issn.2096-742X.2024.01.003

Abstract

Abstract:

[Objective] Heterogeneous computing has become an important part of high-performance computing. GPU heterogeneous computing can significantly accelerate computationally intensive molecular dynamics simulation applications. This paper introduces the system design of a self-developed molecular dynamics simulation software eMD and its application on heterogeneous computing systems. [Methods] This paper first introduces the design objectives of the eMD software including application functionality and computing performance; presents the outline of software design including the framework, modules and interfaces; and then focuses on the software architecture design and porting optimization technology for heterogeneous computing. [Results] The eMD software system can achieve large-scale system simulation based on GPU heterogeneous computing while providing unique molecular dynamics simulation algorithms and models. [Conclusions] eMD will fully leverage the heterogeneous computing power of GPU to improve the efficiency of molecular dynamics simulation applications, and to assist in innovative application of molecular modeling theory and methods and research of molecular science problems.

Key words: molecular dynamics, GPU heterogeneous computing, parallel computing, domestic supercomputer

XU Shun, ZHANG Baohua, LIU Qian, JIN Zhong. eMD: A Large-Scale Molecular Dynamics Simulation Software Based on Heterogeneous Computing[J]. Frontiers of Data and Computing, 2024, 6(1): 21-34, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.01.003.

Figures/Tables 13

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Fig. 7

Table 2

Table 3

Table 4

Table 5

Fig. 8

References 33

[1]	ALLEN M P, TILDESLEY D J. Computer simulation of liquids /2nd ed[M]. Oxford University Press, 2017.
[2]	FRENKEL D, SMIT B. Understanding molecular simulation: from algorithms to applications /3rd ed[M]. Academic Press, 2023.
[3]	LANDAU D P, BINDER K. A Guide to Monte Carlo Simulations in Statistical Physics /5th ed, Cambridge University Press, 2021.
[4]	GROENHOF G. Introduction to QM/MM Simulations[M/OL]// Biomolecular Simulations: Methods and Protocols: vol.924, 2013: 43-66. https://link.springer.com/10.1007/978-1-62703-017-5_3. DOI:10.1007/978-1-62703-017-5_3.
[5]	NOÉ F, TKATCHENKO A, MÜLLER K R, et al. Annual Review of Physical Chemistry: Machine Learning for Molecular Simulation[J/OL]. Annual Review of Physical Chemistry, 2020: 1-30. https://doi.org/10.1146/annurev-physchem-042018-.
[6]	ZHANG J, LEI Y K, ZHANG Z, et al. A Perspective on Deep Learning for Molecular Modeling and Simulations[J/OL]. Journal of Physical Chemistry A, 2020, 124(34): 6745-6763. DOI:10.1021/acs.jpca.0c04473. pmid: 32786668
[7]	陈美霖, 刘端阳, 徐黎明, 等. 基于机器学习的力场模型研究综述[J]. 数据与计算发展前沿, 2023, 5(4): 27-37.
[8]	GAO A, REMSING R C. Self-consistent determination of long-range electrostatics in neural network potentials[J/OL]. Nature Communications, 2022, 13(1): 1-11. DOI:10.1038/s41467-022-29243-2.
[9]	JIA W, WANG H, CHEN M, et al. Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning[C/OL]. International Conference for High Performance Computing, Networking, Storage and Analysis, SC, 2020, 2020-Novem. DOI:10.1109/SC41405.2020.00009.
[10]	WANG H, ZHANG L, HAN J, et al. DeePMD-kit: A deep learning package for many-body potential energy representation and molecular dynamics[J/OL]. Computer Physics Communications, 2018, 228: 178-184. https://doi.org/10.1016/j.cpc.2018.03.016. DOI:10.1016/j.cpc.2018.03.016.
[11]	SIDKY H, CHEN W, FERGUSON A L. Machine learning for collective variable discovery and enhanced sampling in biomolecular simulation[J/OL]. Molecular Physics, 2020, 118(5). https://doi.org/10.1080/00268976.2020.1737742. DOI:10.1080/00268976.2020.1737742.
[12]	ZHANG J, CHEN D, XIA Y, et al. Artificial Intelligence Enhanced Molecular Simulations[J/OL]. Journal of Chemical Theory and Computation, 2023. DOI:10.1021/acs.jctc.3c00214.
[13]	BARRETT R, CHAKRABORTY M, AMIRKULOVA D, et al. HOOMD-TF: GPU-Accelerated, Online Machine Learning in the HOOMD-blue Molecular Dynamics Engine[J/OL]. Journal of Open Source Software, 2020, 5(51): 2367. DOI:10.21105/joss.02367.
[14]	DOERR S, MAJEWSKI M, PÉREZ A, et al. TorchMD: A Deep Learning Framework for Molecular Simulations[J/OL]. Journal of Chemical Theory and Computation, 2021, 17(4): 2355-2363. DOI:10.1021/acs.jctc.0c01343.
[15]	SCHOENHOLZ S S, CUBUK E D. JAX, M.D. A framework for differentiable physics[J/OL]. Advances in Neural Information Processing Systems, 2020, 2020-Decem(NeurIPS). DOI:10.1088/1742-5468/ac3ae9.
[16]	HUANG Y P, XIA Y, YANG L, et al. SPONGE: A GPU-Accelerated Molecular Dynamics Package with Enhanced Sampling and AI-Driven Algorithms[J/OL]. Chinese Journal of Chemistry, 2022, 40(1): 160-168. DOI:10.1002/cjoc.202100456.
[17]	徐顺, 王武, 张鉴, 等. 面向异构计算的高性能计算算法与软件[J]. 软件学报, 2021, 32(8):12.DOI:10.13328/j.cnki.jos.006008.
[18]	HIGHLIGHTS of the TOP500 - JUNE 2023 [EB/OL]. (2023-6-1). [2023-9-18]. https://top500.org/lists/top500/2023/06/highs.
[19]	KONDRATYUK N, NIKOLSKIY V, PAVLOV D, et al. GPU-accelerated molecular dynamics: State-of-art software performance and porting from Nvidia CUDA to AMD HIP[J/OL]. International Journal of High Performance Computing Applications, 2021, 35(4): 312-324. DOI:10.1177/10943420211008288.
[20]	MA Z X, JIN Y Y, TANG S Z, et al. Unified Programming Models for Heterogeneous High-Performance Computers[J/OL]. Journal of Computer Science and Technology, 2023, 38(1): 211-218. DOI:10.1007/s11390-023-2888-4.
[21]	BROWN W, WANG P. Implementing molecular dynamics on hybrid high performance computers-short range forces[J/OL]. Computer Physics Communications, 2011[2013-10-10]. http://www.sciencedirect.com/science/article/pii/S0010465510005102.
[22]	ABRAHAM M J, MURTOLA T, SCHULZ R, et al. Gromacs: High performance molecular simulations through multi-level parallelism from laptops to supercomputers[J/OL]. SoftwareX, 2015, 1-2: 19-25. DOI:10.1016/j.softx.2015.06.001.
[23]	PHILLIPS J C, HARDY D J, MAIA J D C, et al. Scalable molecular dynamics on CPU and GPU architectures with NAMD[J/OL]. Journal of Chemical Physics, 2020, 153(4). https://doi.org/10.1063/5.0014475. DOI:10.1063/5.0014475.
[24]	EASTMAN P, SWAILS J, CHODERA J D, et al. OpenMM 7: Rapid development of high performance algorithms for molecular dynamics[J/OL]. PLoS Computational Biology, 2017, 13(7): 1-17. DOI:10.1371/journal.pcbi.1005659.
[25]	GLASER J, NGUYEN T D, ANDERSON J A, et al. Strong scaling of general-purpose molecular dynamics simulations on GPUs[J/OL]. Computer Physics Communications, 2015, 192(July): 97-107. http://linkinghub.elsevier.com/retrieve/pii/S0010465515000867. DOI:10.1016/j.cpc.2015.02.028.
[26]	ZHU Y L, LIU H, LI Z W, et al. GALAMOST: GPU-accelerated large-scale molecular simulation toolkit[J/OL]. Journal of Computational Chemistry, 2013, 34(25): 2197-2211. DOI:10.1002/jcc.23365.
[27]	Getting good performance from mdrun[EB/OL]. (2023-1-1). [2023-9-20]. https://manual.gromacs.org/current/user-guide/mdrun-performance.html.
[28]	COLBERG P H, HÖFLING F. Highly accelerated simulations of glassy dynamics using GPUs: Caveats on limited floating-point precision[J/OL]. Computer Physics Communications, 2011, 182(5): 1120-1129.[2013-12-04]. http://linkinghub.elsevier.com/retrieve/pii/S0010465511000294. DOI:10.1016/j.cpc.2011.01.009.
[29]	YANG C, ZENG Y, XU S, et al. The coherent motions of thermal active Brownian particles[J/OL]. Physical Chemistry Chemical Physics, 2023, 25(18): 13027-13032. DOI:10.1039/d2cp05984c.
[30]	WAN B, YU J. Two-phase dynamics of DNA supercoiling based on DNA polymer physics[J/OL]. Biophysical Journal, 2022, 121(4): 658-669. https://doi.org/10.1016/j.bpj.2022.01.001. DOI:10.1016/j.bpj.2022.01.001. pmid: 35016860
[31]	NVIDA CUDA Documentation[EB/OL]. (2023-5-10). [2023-9-24]. https://docs.nvidia.com/cuda/doc/index.html.
[32]	AMD ROCm Documentation[EB/OL]. (2023-8-18). [2023-9-24]. https://rocmdocs.amd.com/en/latest/index.html.
[33]	YIN L K, XU S, JEONG S, et al. Vapor-liquid coexisting morphology of all-atom water model through generalized isothermal isobaric ensemble molecular dynamics simulation[J/OL]. Wuli Xuebao/Acta Physica Sinica, 2017, 66(13). DOI:10.7498/aps.66.136102.

输入：本区域内原子坐标
输出：每个原子的邻居列表和溢出状态
1	nb_list.clear()
2	i = blockId.x+blockDim.x +threadIdx.x
3	iloc=0xffffffff
4	jnum_overflowed=0
5	jnum_add=0
6	if i < atom_local then
7	for each j in local atoms do
8	dx=pos[i]-pos[j]
9	dr2=dot3(dx,dx)
10	if dr2< cutskin_sq then
11	if iloc is 0xffffffff then
12	iloc=atomInc(&status->x, 0xffffffff)
13	jnum_add=0
14	end if
15	s=nb_lists.lappend(iloc,jnum_add,j)
16	if s is OVERFLOWED then
17	jnum_overflowed++
18	else
19	jnum_add++
20	end if
21	end if
22	end for
23	if iloc is not 0xffffffff then
24	ilist[iloc]=i
25	nb_list.lnum(iloc)=jnum_add
26	if jnum_overflowed > 0 then
27	atomicMax(&status->y,jnum_overflowed)
28	end if
29	end if
30	return ilist, nb_list,status
31	end if

性能项	1 CPU核 + 1块GPU卡	1 CPU核	32 CPU核
Pair force	4.689	291.2	9.486
Neighbor	2.257	84.34	2.807
两项和	6.946	375.54	12.293

作用势	原子数	CPU neighbor &CPU force	CPU neighbor &GPU force	GPU neighbor &GPU force
lj_cut	2,048,000	0.457	0.889	5.945
dpd	500,000	1.124	1.586	14.919
lj_gromacs_coul_gromacs	108,000	9.655	16.36	62.84

eMD	512核	1024核	2048核	4096核
CPU	2,215	1,103	555.7	271.1
CPU+GPU	132.7	68.87	34.86	19.26

测试程序	节点数	每节点CPU核	每节点GPU卡	整体耗时（秒）
eMD	2	4	1	31.4
eMD	2	4	0	123.3
LAMMPS	2	4	0	131.183