End-to-End Workflow Framework for Cross-Center Scientific Data Analysis

doi:10.11871/jfdc.issn.2096-742X.2023.01.002

Abstract

Abstract:

[Objective] The rapid development of big data and artificial intelligence technology has led to the transformation of research paradigms. New paradigms generally require collaborative analysis. Task types are complex and the analysis process spans different scientific data centers. [Application background] Existing process-based analysis frameworks are difficult to support end-to-end cross-center scientific data analysis requirements due to the lack of the capabilities of analysis process expression, heterogeneous computing framework integration, and cross-center job scheduling. [Methods] A software framework for end-to-end cross-center analysis of scientific data is proposed, which supports cross-center heterogeneous workflow construction, cross-framework data transparent transfer, and cross-center job optimization scheduling. [Results] The function and performance of the proposed framework are verified based on the scenario of "cross-station online processing and quality control of aboveground grass biomass" in the National Ecosystem Science Data Center, which verifies the advancement and feasibility of the framework.

Key words: scientific research paradigm, analysis workflow, scientific data center, cross-center computing

ZHU Xiaojie,WANG Huajin,SHEN Zhihong,GUO Xuebing,DONG Wen. End-to-End Workflow Framework for Cross-Center Scientific Data Analysis[J]. Frontiers of Data and Computing, 2023, 5(1): 15-27, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2023.01.002.

Figures/Tables 9

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

References 26

[1]	JUMPER, J, EVANS, R, PRITZEL, A, et al. Highly acc-urate protein structure prediction with AlphaFold[J]. Nat-ure, 2021, 596(7873): 583-589.
[2]	NIU, C H, AGGARWAL, K, LI, D, et al. A repeating fast radio burst associated with a persistent radio source[J]. Nat-ure, 2022, 606(7916): 873-877.
[3]	COLLABORATION, E H T, AKIYAMA, K, ALBERDI, A, et al. First Sagittarius A* Event Horizon Telescope Re-sults. II. EHT and Multiwavelength Observations, Data Processing, and Calibration[J]. The Astrophysical Journal Letters, 2022, 930(2): L13. doi: 10.3847/2041-8213/ac6675
[4]	PETERSEN, A M, AHMED, M E, PAVLIDIS, I. Grand challenges and emergent modes of convergence science[J]. Humanities and Social Sciences Communications, 2021, 8(1): 1-15. doi: 10.1057/s41599-020-00684-8
[5]	沈志宏, 张晓林, 郑晓欢. PARIS原则: 开放协作环境下科学数据的可用性[J]. 大数据, 2023.DOI: 10.11959/j.issn.2096-0271.2023013. doi: 10.11959/j.issn.2096-0271.2023013
[6]	康建芳, 张耀南, 王家耀, 等. 黄河流域生态保护与高质量发展体系化科学数据建设与实践[J]. 中国科技资源导刊, 2022, 54(1): 56-65.
[7]	Apache NiFi[EB/OL]. [2022-11-30]. https://nifi.apache.org/.
[8]	朱小杰, 赵子豪, 杜一, 等. 模型驱动的大数据流水线框架PiFlow[J]. 计算机应用, 2020, 40(6): 1638. doi: 10.11772/j.issn.1001-9081.2019101793
[9]	Apache Airflow[EB/OL]. [2022-11-30]. https://airflow.apache.org/.
[10]	DER AUWERA G A, O’CONNOR B D. Genomics in the cloud: using Docker, GATK, and WDL in Terra[M]. O’Reilly Media, 2020.
[11]	OpenWDL[EB/OL]. [2022-11-30]. https://openwdl.org/.
[12]	DEAN J, GHEMAWAT S. MapReduce: a flexible data pro-cessing tool[J]. Communications of the ACM, 2010, 53 (1): 72-77.
[13]	LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436-444. doi: 10.1038/nature14539
[14]	BIRD I, CARMINATI F, MOUNT R, et al. Update of the Computing Models of the WLCG and the LHC Exper-iments[R]. 2014.
[15]	FOSTER I. Globus Toolkit Version 4: Software for Ser-vice-Oriented Systems[J]. Journal of Computer Science and Technology, 2006, 21(4): 513-520. doi: 10.1007/s11390-006-0513-y
[16]	MICHENER W, VIEGLAIS D, VISION T, et al. Data-ONE: Data observation network for earth-preserving data and enabling innovation in the biological and envir-onmental sciences[J]. D-Lib Magazine, 2011, 17(1-2).
[17]	GIULIANI G, RAY N, SCHWARZER S, et al. Sharing Environmental Data through GEOSS[J]. Emerging Meth-ods and Multidisciplinary Applications in Geospatial Research, 2013: 266-281.
[18]	黄铁青, 牛栋. 中国生态系统研究网络(CERN):概况、成就和展望[J]. 地球科学进展, 2005, 20(8): 895. doi: 10.11867/j.issn.1001-8166.2005.08.0895
[19]	GAIA-X - Home[EB/OL]. [2022-11-30]. https://www.data-infrastructure.eu/GAIAX.
[20]	GO FAIR initiative: Make your data & services FAIR[EB/OL]. [2022-11-30]. https://www.go-fair.org/.
[21]	ZAHARIA M, FRANKLIN M J, GHODSI A, et al. Apa-che Spark: a unified engine for big data processing[J]. Communications of the ACM, 2016, 59(11): 56-65.
[22]	CARBONE P, KATSIFODIMOS A, EWEN S, et al. Apache Flink: Stream and Batch Processing in a Single Engine[J]. Bulletin of the IEEE Computer Society Technical Com-mittee on Data Engineering, 2015, 36(4).
[23]	IHAKA R, GENTLEMAN R. R: A Language for Data Analysis and Graphics[J]. Journal of Computational and Graphical Statistics, 1996, 5(3): 299-314.
[24]	MCKINNEY W. pandas: a Foundational Python Library for Data Analysis and Statistics[J]. Python for high per-formance and scientific computing, 2011, 14(9): 1-9.
[25]	PASZKE A, GROSS S, MASSA F, et al. PyTorch: An Im-perative Style, High-Performance Deep Learning Library[J]. Advances in Neural Information Processing Systems, 2019, 32.
[26]	ABADI M, BARHAM P, CHEN J, et al. TensorFlow: A System for Large-Scale Machine Learning[C]// Pro-ceedings of the 12th USENIX Symposium on Oper-ating Systems Design and Implementation, 2016: 265-283.