Research on Parallel Acceleration of OpenFOAM Threads for Domestic Accelerator

doi:10.11871/jfdc.issn.2096-742X.2024.02.012

Abstract

Abstract:

[Background] With the refinement of computational fluid dynamics simulations, the demand for computing power from CFD software OpenFOAM continues to increase. The new “East” supercomputing system is a new type of home-grown heterogeneous supercomputer to satisfy the demand. [Purpose] OpenFOAM is optimized and accelerated by porting it to the Oriental Supercomputer. [Methods] Firstly, by analyzing the functional architecture of the “East” supercomputing system and OpenFOAM, a solver for the domestic accelerator is developed and the ported CUSP is used to call the underlying code of the domestic accelerator to realize the sparse storage format matrix vector multiplication and diagonal matrix preprocessing. Secondly, a parallel SpMV (matrix-vector multiplication) on a single-node card with multiple domestic accelerators is implemented. [Results] The work introduced in this paper is verified by the pitzDaily example provided by OpenFOAM, and the performance is evaluated through comparison with several comparative methods of acceleration, which shows our approach achieves a 19.7 times speedup. [Limitations] This study only realizes a parallel optimization of OpenFOAM on a single-node. [Conclusions] The results of this study are of great significance for unleashing the advantages of OpenFOAM in fluid mechanics and expanding the scope of the application of supercomputer software.

Key words: OpenFOAM, parallel computing, heterogeneous computing, domestic accelerator, porting optimization

SHANG Xiaomin, LI Qiang, GAO Lingyun, TAO Shunan, ZHOU Quan, YUAN Wu, LU Zhonghua. Research on Parallel Acceleration of OpenFOAM Threads for Domestic Accelerator[J]. Frontiers of Data and Computing, 2024, 6(2): 134-144, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.02.012.

Figures/Tables 22

Fig.1

Table 1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Algorithm 1

Fig.8

Fig.9

Table 2

Fig.10

Fig.11

Table 3

Table 4

Fig.12

Fig.13

Table 5

Table 6

Fig.14

Table 7

References 14

[1]	KRISHNASWAMY H, MUTHUKRISHNAN S, THANIKODI S, et al. Investigation of air conditioning temperature variation by modifying the structure of passenger car using computational fluid dynamics[J]. Thermal Science, 2020, 24(1 Part B): 495-498. doi: 10.2298/TSCI190409397K
[2]	李强, 陆忠华, 王彦棡, 等. 基于高性能计算的 SWAT 参数敏感度分析并行框架[J]. 计算机应用研究, 2015, 32(1): 41-44.
[3]	田在荣, 李强, 聂宁明, 等. 并发 L-BFGS 异构率定算法设计与实现[J]. 青岛大学学报(自然科学版), 2021, 34(3): 43-50.
[4]	QI Y, LI Q, ZHAO Z, et al. Heterogeneous parallel implementation of large-scale numerical simulation of saint-Venant equations[J]. Applied Sciences, 2022, 12(11): 5671-5688. doi: 10.3390/app12115671
[5]	应智. 基于GPU的OpenFOAM并行加速研究[D]. 上海: 上海交通大学, 2012.
[6]	BELL N, GARLAND M. Efficient sparse matrix-vector multiplication on CUDA[R]. Nvidia Technical Report NVR-2008-004, Nvidia Corporation, 2008.
[7]	KRASNOPOLSKY B, MEDVEDEV A. Acceleration of large scale OpenFOAM simulations on distributed systems with multicore CPUs and GPUs[M]//Parallel Computing:On the Road to Exascale. IOS Press, 2016: 93-102.
[8]	张玉春欣, 冯鲁彬, 梁建国. 采用曙光超级计算机的海冰模型并行性能调优研究[J]. 软件导刊, 2021, 20(6): 80-85.
[9]	谢景明, 胡伟方, 韩林, 等. 基于“嵩山”超级计算机系统的量子傅里叶变换模拟[J]. 计算机科学, 2021, 48(12): 36-42. doi: 10.11896/jsjkx.201200023
[10]	吴建平, 王正华, 李晓梅. 稀疏线性方程组的高效求解与并行计算[M]. 长沙: 湖南科学技术出版社, 2004:5-6.
[11]	CHAN S, PHOON K, LEE F. A modified Jacobi preconditioner for solving ill-conditioned Biot's consolidation equations using symmetric quasi-minimal residual method[J]. International Journal for Numerical and Analytical Methods in Geomechanics, 2001, 25(10): 1001-1025. doi: 10.1002/nag.v25:10
[12]	KERSHAW D S. The incomplete Cholesky—conjugate gradient method for the iterative solution of systems of linear equations[J]. Journal of computational physics, 1978, 26(1): 43-65. doi: 10.1016/0021-9991(78)90098-0
[13]	孟德龙, 文敏华, 韦建文, 等. 神威太湖之光上OpenFOAM的移植与优化[J]. 计算机科学, 2017, 44(10):64-70. doi: 10.11896/j.issn.1002-137X.2017.10.012
[14]	XIAOMIN S, QIANG L, YONGMENG Q, et al. SLIC Research and Implementation of a Parallel Optimization Algorithm[C]// 2022 IEEE 2nd International Conference on Computer Systems (ICCS), IEEE, 2022: 67-74.

硬/软件环境	名称	详情/版本
硬件环境（单个节点）	CPU	32核心国产x86处理器*1
	内存	16GBDDR4*8
	加速设备	国产GPGPU加速器*4，16GBHBM2显存、带宽1TB/s
	网络	InfiniBandHDR网络、200Gbps
软件环境	gcc/g++	4.8.5
软件环境	OpenMP Thrust Cusp	3.1 dtk-22.01 0.5.1

网格数	128*128	256*256	512*512	1024*1024	2048*2048	4096*4096	8192*8192
ELL	4.596	4.746	4.785	4.766	5.224	9.218	12.801
CSR	4.332	4.284	4.764	4.435	4.982	8.234	11.902
COO(256)	0.128	0.203	0.519	1.808	7.019	27.739	109.907
COO(512)	0.121	0.182	0.408	1.336	5.123	20.095	79.321
COO(1024)	0.117	0.158	0.355	1.116	4.184	16.237	63.234

网格数	128*128	256*256	512*512	1024*1024	2048*2048	4096*4096	8192*8192
ELL	51.8	115.2	244.2	853.4	3,975	27,614	201,398
CSR	53.1	118.4	254.6	863.5	4,063	28,101	223,640
COO(256)	54.9	161.9	628.5	4,168.8	305,26	274,676	2,437,970
COO(512)	50.1	140.4	503.3	3,144.9	22,527	203,003	1,789,690
COO(1024)	49.5	133.3	449.5	2,656.5	18,519	165,598	1,456,670

LDU_DIA	180*180	2880*2880	12225*12225
Amul_time	0.005	0.071	0.256
Preconditioner	0.001	0.014	0.05
PCG_time	0.9	15.83	143.89
Iteration	18	81	174
ELL_DIA	180*180	2880*2880	12225*12225
Amul_time	0.002	0.01	0.036
Preconditioner	0.0001	0.002	0.005
PCG_time	0.09	5.43	36.263
Iteration	18	80	169

Multiply(单次)		100*100	250*250	500*500	750*750	1000*1000
solve_ELL	CPU	0.277	0.823	3.075	7.405	14.503
solve_ELL	国产加速卡	0.035	0.034	0.036	0.037	0.031
solve_COO	CPU	0.147	0.811	6.802	14.403	27.617
solve_COO	国产加速卡	0.242	0.305	0.472	1.062	8.42
solve_CSR	CPU	0.085	0.408	1.526	3.413	6.444
solve_CSR	国产加速卡	0.012	0.012	0.013	0.013	0.345