面向国产加速卡的OpenFOAM线程并行加速研究

doi:10.11871/jfdc.issn.2096-742X.2024.02.012

数据与计算发展前沿 ›› 2024, Vol. 6 ›› Issue (2): 134-144.

CSTR: 32002.14.jfdc.CN10-1649/TP.2024.02.012

doi: 10.11871/jfdc.issn.2096-742X.2024.02.012

面向国产加速卡的OpenFOAM线程并行加速研究

尚小敏¹(),李强^1,^*(),高凌云^2,³,陶顺安¹,周全¹,袁武^2,³,陆忠华^2,³

1.青岛大学计算机科学技术学院，山东青岛 266071
2.中国科学院计算机网络信息中心，北京 100083
3.中国科学院大学，北京 100049

收稿日期:2023-04-28 出版日期:2024-04-20 发布日期:2024-04-26
通讯作者: *李强（E-mail: lq.sxt@163.com）
作者简介:尚小敏，青岛大学计算机科学技术学院，硕士研究生，主要研究方向为高性能计算。硕士期间发表EI会议论文1篇。
本文中负责代码调试，实验验证以及论文的编写。
SHANG Xiaomin is a master’s student in the School of Computer Science and Technology at Qingdao University Her main research interests include high-performance computing. During her master's program, she published one conference paper in EI.
In this paper, she was responsible for code debugging, experimental verification, and paper writing.
E-mail: 1415470940@qq.com|李强，青岛大学计算机科学技术学院，博士，讲师，2012年于美国明尼苏达大学访问工作半年，《Hydrology Research》期刊审稿人。主要研究方向为高性能计算、异构并行计算、水循环模拟等。参与国家自然科学基金3项，发表学术论文5篇，其中2篇被SCI检索，专利1项，译著1部。
本文中负责论文实验指导、文章润色。
LI Qiang, Ph.D., is a lecturer at the School of Computer Science and Technology of Qingdao University. In 2012, he worked as a visiting scholar at the University of Minnesota in the United States for six months and served as a reviewer for the journal “Hydrology Research”. His main research fields/directions include high-performance computing, heterogeneous parallel computing, and water cycle simulation. He has participated in three projects funded by the National Natural Science Foundation, published five academic papers, of which two have been indexed by SCI, obtained one patent, and translated one book.
In this paper, he is responsible for guiding the experimental work and revising the article.
E-mail: lq.sxt@163.com
基金资助:
国家重点研发计划(2020YFB1709500);山东省自然科学基金面上项目(ZR201910310143)

Research on Parallel Acceleration of OpenFOAM Threads for Domestic Accelerator

SHANG Xiaomin¹(),LI Qiang^1,^*(),GAO Lingyun^2,³,TAO Shunan¹,ZHOU Quan¹,YUAN Wu^2,³,LU Zhonghua^2,³

1. School of Computer Science and Technology, Qingdao University, Qingdao, Shandong 266071, China
2. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100083, China
3. University of Chinese Academy of Sciences, Beijing 100049, China

Received:2023-04-28 Online:2024-04-20 Published:2024-04-26

摘要/Abstract

摘要：

【背景】随着流体力学模拟的精细化，CFD流体模拟软件OpenFOAM对算力的需求持续增加，新型东方超级计算系统是国产自研的新型异构超算。【目的】以新型东方超级计算系统为平台移植OpenFOAM，实现OpenFOAM的国产超算适配与加速。【方法】首先，通过分析“东方”超级计算系统和OpenFOAM的功能架构，本文制定了适用于国产加速卡的求解器，通过使用本文移植后的CUSP来调用国产加速卡底层代码从而实现稀疏存储格式矩阵向量乘以及diagonal矩阵预处理。其次，在此基础上，实现了单节点多国产加速卡的并行SpMV。【结果】本文使用了OpenFOAM自带的pitzDaily算例进行算法验证，并通过多种加速性能的对比方法对测试性能进行分析，取得了19.7倍的加速效果。【局限】本研究只实现了单节点的OpenFOAM并行优化。【结论】本研究结果对于发挥OpenFOAM在流体力学方面的优势和扩大超算软件适应面具有重要意义。

关键词: OpenFOAM, 并行计算, 异构计算, 国产加速卡, 移植优化

Abstract:

[Background] With the refinement of computational fluid dynamics simulations, the demand for computing power from CFD software OpenFOAM continues to increase. The new “East” supercomputing system is a new type of home-grown heterogeneous supercomputer to satisfy the demand. [Purpose] OpenFOAM is optimized and accelerated by porting it to the Oriental Supercomputer. [Methods] Firstly, by analyzing the functional architecture of the “East” supercomputing system and OpenFOAM, a solver for the domestic accelerator is developed and the ported CUSP is used to call the underlying code of the domestic accelerator to realize the sparse storage format matrix vector multiplication and diagonal matrix preprocessing. Secondly, a parallel SpMV (matrix-vector multiplication) on a single-node card with multiple domestic accelerators is implemented. [Results] The work introduced in this paper is verified by the pitzDaily example provided by OpenFOAM, and the performance is evaluated through comparison with several comparative methods of acceleration, which shows our approach achieves a 19.7 times speedup. [Limitations] This study only realizes a parallel optimization of OpenFOAM on a single-node. [Conclusions] The results of this study are of great significance for unleashing the advantages of OpenFOAM in fluid mechanics and expanding the scope of the application of supercomputer software.

Key words: OpenFOAM, parallel computing, heterogeneous computing, domestic accelerator, porting optimization

尚小敏, 李强, 高凌云, 陶顺安, 周全, 袁武, 陆忠华. 面向国产加速卡的OpenFOAM线程并行加速研究[J]. 数据与计算发展前沿, 2024, 6(2): 134-144.

SHANG Xiaomin, LI Qiang, GAO Lingyun, TAO Shunan, ZHOU Quan, YUAN Wu, LU Zhonghua. Research on Parallel Acceleration of OpenFOAM Threads for Domestic Accelerator[J]. Frontiers of Data and Computing, 2024, 6(2): 134-144, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.02.012.

图/表 22

图1

表1

图2

图3

图4

图5

图6

图7

算法1

图8

图9

表2

图10

图11

表3

表4

图12

图13

表5

表6

图14

表7

参考文献 14

[1]	KRISHNASWAMY H, MUTHUKRISHNAN S, THANIKODI S, et al. Investigation of air conditioning temperature variation by modifying the structure of passenger car using computational fluid dynamics[J]. Thermal Science, 2020, 24(1 Part B): 495-498. doi: 10.2298/TSCI190409397K
[2]	李强, 陆忠华, 王彦棡, 等. 基于高性能计算的 SWAT 参数敏感度分析并行框架[J]. 计算机应用研究, 2015, 32(1): 41-44.
[3]	田在荣, 李强, 聂宁明, 等. 并发 L-BFGS 异构率定算法设计与实现[J]. 青岛大学学报(自然科学版), 2021, 34(3): 43-50.
[4]	QI Y, LI Q, ZHAO Z, et al. Heterogeneous parallel implementation of large-scale numerical simulation of saint-Venant equations[J]. Applied Sciences, 2022, 12(11): 5671-5688. doi: 10.3390/app12115671
[5]	应智. 基于GPU的OpenFOAM并行加速研究[D]. 上海: 上海交通大学, 2012.
[6]	BELL N, GARLAND M. Efficient sparse matrix-vector multiplication on CUDA[R]. Nvidia Technical Report NVR-2008-004, Nvidia Corporation, 2008.
[7]	KRASNOPOLSKY B, MEDVEDEV A. Acceleration of large scale OpenFOAM simulations on distributed systems with multicore CPUs and GPUs[M]//Parallel Computing:On the Road to Exascale. IOS Press, 2016: 93-102.
[8]	张玉春欣, 冯鲁彬, 梁建国. 采用曙光超级计算机的海冰模型并行性能调优研究[J]. 软件导刊, 2021, 20(6): 80-85.
[9]	谢景明, 胡伟方, 韩林, 等. 基于“嵩山”超级计算机系统的量子傅里叶变换模拟[J]. 计算机科学, 2021, 48(12): 36-42. doi: 10.11896/jsjkx.201200023
[10]	吴建平, 王正华, 李晓梅. 稀疏线性方程组的高效求解与并行计算[M]. 长沙: 湖南科学技术出版社, 2004:5-6.
[11]	CHAN S, PHOON K, LEE F. A modified Jacobi preconditioner for solving ill-conditioned Biot's consolidation equations using symmetric quasi-minimal residual method[J]. International Journal for Numerical and Analytical Methods in Geomechanics, 2001, 25(10): 1001-1025. doi: 10.1002/nag.v25:10
[12]	KERSHAW D S. The incomplete Cholesky—conjugate gradient method for the iterative solution of systems of linear equations[J]. Journal of computational physics, 1978, 26(1): 43-65. doi: 10.1016/0021-9991(78)90098-0
[13]	孟德龙, 文敏华, 韦建文, 等. 神威太湖之光上OpenFOAM的移植与优化[J]. 计算机科学, 2017, 44(10):64-70. doi: 10.11896/j.issn.1002-137X.2017.10.012
[14]	XIAOMIN S, QIANG L, YONGMENG Q, et al. SLIC Research and Implementation of a Parallel Optimization Algorithm[C]// 2022 IEEE 2nd International Conference on Computer Systems (ICCS), IEEE, 2022: 67-74.

硬/软件环境	名称	详情/版本
硬件环境（单个节点）	CPU	32核心国产x86处理器*1
	内存	16GBDDR4*8
	加速设备	国产GPGPU加速器*4，16GBHBM2显存、带宽1TB/s
	网络	InfiniBandHDR网络、200Gbps
软件环境	gcc/g++	4.8.5
软件环境	OpenMP Thrust Cusp	3.1 dtk-22.01 0.5.1

网格数	128*128	256*256	512*512	1024*1024	2048*2048	4096*4096	8192*8192
ELL	4.596	4.746	4.785	4.766	5.224	9.218	12.801
CSR	4.332	4.284	4.764	4.435	4.982	8.234	11.902
COO(256)	0.128	0.203	0.519	1.808	7.019	27.739	109.907
COO(512)	0.121	0.182	0.408	1.336	5.123	20.095	79.321
COO(1024)	0.117	0.158	0.355	1.116	4.184	16.237	63.234

网格数	128*128	256*256	512*512	1024*1024	2048*2048	4096*4096	8192*8192
ELL	51.8	115.2	244.2	853.4	3,975	27,614	201,398
CSR	53.1	118.4	254.6	863.5	4,063	28,101	223,640
COO(256)	54.9	161.9	628.5	4,168.8	305,26	274,676	2,437,970
COO(512)	50.1	140.4	503.3	3,144.9	22,527	203,003	1,789,690
COO(1024)	49.5	133.3	449.5	2,656.5	18,519	165,598	1,456,670

LDU_DIA	180*180	2880*2880	12225*12225
Amul_time	0.005	0.071	0.256
Preconditioner	0.001	0.014	0.05
PCG_time	0.9	15.83	143.89
Iteration	18	81	174
ELL_DIA	180*180	2880*2880	12225*12225
Amul_time	0.002	0.01	0.036
Preconditioner	0.0001	0.002	0.005
PCG_time	0.09	5.43	36.263
Iteration	18	80	169

Multiply(单次)		100*100	250*250	500*500	750*750	1000*1000
solve_ELL	CPU	0.277	0.823	3.075	7.405	14.503
solve_ELL	国产加速卡	0.035	0.034	0.036	0.037	0.031
solve_COO	CPU	0.147	0.811	6.802	14.403	27.617
solve_COO	国产加速卡	0.242	0.305	0.472	1.062	8.42
solve_CSR	CPU	0.085	0.408	1.526	3.413	6.444
solve_CSR	国产加速卡	0.012	0.012	0.013	0.013	0.345

面向国产加速卡的OpenFOAM线程并行加速研究

Research on Parallel Acceleration of OpenFOAM Threads for Domestic Accelerator

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 22

参考文献 14

相关文章 8

编辑推荐

Metrics

本文评价

	180* 180	2880*2880	12225*12225	110025*110025
1块国产加速卡	1	1	1	1
2块国产加速卡	1.34	1.33	1.35	1.62
4块国产加速卡	1.63	1.59	1.69	3

[1]	徐顺, 张宝花, 刘倩, 金钟. eMD：基于异构计算的大规模分子动力学模拟软件[J]. 数据与计算发展前沿, 2024, 6(1): 21-34.
[2]	李维钊, 王伟. 基于国产异构计算平台的快速SVD算法及其在海洋资料同化的应用[J]. 数据与计算发展前沿, 2024, 6(1): 35-45.
[3]	曹义魁,陆忠华,张鉴,刘夏真,袁武,梁姗. 面向国产加速器的CFD核心算法并行优化[J]. 数据与计算发展前沿, 2021, 3(4): 93-103.
[4]	柴象海,胡寿丰,张执南,侯亮. 显式动力学子模型法在航空发动机整机瞬态冲击并行计算中的应用[J]. 数据与计算发展前沿, 2020, 2(6): 11-20.
[5]	张留莹,王鹏飞,张峰,刘海龙,林鹏飞,王涛,韦俊林,田少博,姜金荣,迟学斌. 海洋环流模式LICOM的GPU实现与优化[J]. 数据与计算发展前沿, 2020, 2(4): 92-104.
[6]	周广庆,张云泉,姜金荣,张贺,吴保东,曹杭,王天一,郝卉群,朱家文,袁良,张明华. 地球系统模式CAS-ESM[J]. 数据与计算发展前沿, 2020, 2(1): 38-54.
[7]	李晓涵,陈文光. COS：度量分布式大数据处理系统的效率[J]. 数据与计算发展前沿, 2020, 2(1): 93-104.
[8]	赵海涛, 孙家昶, 黎雷生, 杨文浩, 赵慧, 李会元. 适合一类复杂异构超算系统的HPL并行计算模型研究[J]. 数据与计算发展前沿, 2020, 2(1): 85-92.