Research and Design of a Basic Software Stack Architecture for Scientific Data Center

doi:10.11871/jfdc.issn.2096-742X.2023.01.001

Abstract

Abstract:

[Background] Data resources in many disciplines are growing explosively. Scientific research paradigms are undergoing profound changes. Scientific data centers have become important carriers of data app-lications in disciplines. The basic software stack of the scientific data center is currently in top priority of the scientific data center construction. [Methods] Through analyzing the software architecture of typical scientific data centers at home and abroad, this paper points out the shortcomings of the current mainstream software architecture in the storage, management, sharing, and analysis of multi-source heterogeneous data. [Results] A reference design for the software architecture of the scientific data center and its prototype implementation are proposed, which consists of the functionalities of standardized storage, heterogeneous data fusion management, big data stream processing, online interactive analysis, etc. The proposed scheme can serve as a technical reference for constructing scientific data centers in China.

Key words: scientific data center, software stack, technical architecture

WANG Jiulong,WANG Ronghao,WANG Huajin,LU Changfa,DUAN Junlei. Research and Design of a Basic Software Stack Architecture for Scientific Data Center[J]. Frontiers of Data and Computing, 2023, 5(1): 1-14, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2023.01.001.

Figures/Tables 6

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

References 36

[1]	张家榕, 颜嘉麒, 叶鹰. 科学数据分析综论:情报学新视野[J/OL]. 情报科学:1-7[2022-12-01]. http://kns.cnki.net/kcms/detail/22.1264.G2.20221010.1454.004.html.
[2]	苏靖. 大数据时代加强科学数据管理的思考与对策[J]. 中国软科学, 2022,(09):50-54.
[3]	冯媛. 科学数据开放共享的价值共创模型及运行机制研究[J]. 图书馆, 2022,(09):29-37.
[4]	高孟绪, 王瑞丹, 王超, 徐波, 石蕾. 关于国家科学数据中心建设与发展的思考[J]. 农业大数据学报, 2019, 1(3): 21-27.
[5]	European Commission High Level Expert Group on Sci-entific Data. Riding the wave: How Europe can gain from the rising tide of scientific data (Final report of the High Level Expert Group on Scientific Data. A submission to the European Commission)[N/OL]. European Union [2010]. https://www.fosteropenscience.eu/content/riding-wave-how-europe-can-gain-rising-tide-scientific-data.
[6]	NIH Sharing Policies and Related Guidance on NIH-Funded Research Resources[EB/OL]. National Institutes of Health [2016]. https://grants.nih.gov/policy/sharing.htm.
[7]	Award and Administration Guide: Data Sharing Policy[EB/OL]. National Science Foundation [2011]. https://www.nsf.gov/pubs/policydocs/pappguide/nsf11001/aag_6.jsp#VID4.
[8]	OECD Principles and Guidelines for Access to Research Data from Public Funding[EB/OL]. Paris: Organisation for Economic Co-Operation and Development, p. 24. Organisation for Economic Co-operation Development [2007]. http://www.oecd.org/dataoecd/9/61/38500813.pdf.
[9]	Shoshani, A. The Scientific Data Management Center: Providing Technologies for Large Scale Scientific Explor-ation. SSDBM[J/OL]. Lecture Notes in Computer Science, 2009, vol 5566, Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02279-1_1.
[10]	Semilof, M. and Montgomery, J. Software stack. Tech-Target[J/OL]. [2020]. https://www.techtarget.com/searchapparchitecture/definition/software-stack.
[11]	Cinque, M., Cotroneo, D., Frattini, F., & Russo, S. To Cloudify or Not to Cloudify: The Question for a Scien-tific Data Center[J/OL]. IEEE Transactions on Cloud Computing, 2016, 4(1): 90-103.DOI: 10.1109/TCC.2015.2396061. doi: 10.1109/TCC.2015.2396061
[12]	Bell, T. OpenStack and CERN[N/OL]. OpenStack Swi-tzerland User Group[2012]. https://vdocuments.net/20121115-open-stackchusergroupv12.html?page=1.
[13]	ESA. ESA AGENDA 2025-Make space for Europe[EB/OL]. [2022]. https://esamultimedia.esa.int/docs/ESA_Agenda_2025_final.pdf.
[14]	GBIF[EB/OL]. What is GBIF[2022]. https://www.gbif.org/what-is-gbif.
[15]	Downs, R.R. A Scientific Data Center Perspective on Publishing Data[N/OL]. Columbia Academic Commons [2015]. https://academiccommons.columbia.edu/doi/10.7916/D8P849P6.
[16]	Sayers, E.W., Beck, J., and Brister, J.R., et al. Database resources of the National Center for Biotechnology Infor-mation[J/OL]. Nucleic Acids Research[2017], Volume 45, Issue D1, Pages D12-D17. https://doi.org/10.1093/nar/gkw1071.
[17]	Metaj, S., Giordano, D., Paltenghi, M., Metaj, S., & Dvořák, A. Anomaly detection in the CERN cloud infrastructure[J/OL]. EPJ Web of Conferences, 2021, 251,02011. DOI: https://doi.org/10.1051/epjconf/202125102011.
[18]	About CERN[EB/OL].[2022]. https://home.cern/about.
[19]	NASA[EB/OL].[2022]. https://www.nasa.gov/about/index.html.
[20]	NOAA[EB/OL]. [2021]. https://www.noaa.gov/about-our-agency.
[21]	eXtreme Data Cloud[N/OL].[2020]. https://cordis.europa.eu/project/id/777367/results/fr.
[22]	Rousseau, H., Chan Kwok Cheong, B., Contescu, C.I., Espinal Curull, X., Iven, J., González Labrador, H., Lamanna, M., Lo Presti, G., Mascetti, L., Moscicki, J.T., & van der Ster, D. Providing large-scale disk storage at CERN[J/OL]. EPJ Web of Conferences, 2019, 214,04033. DOI: https://doi.org/10.1051/epjconf/201921404033.
[23]	Shontz, K. The NESDIS Cloud Framework-A Common, Data-Agnostic Cloud Computing Solution[J/OL]. NOAA[2020]. https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fwww.star.nesdis.noaa.gov%2Fstar%2Fdocuments%2Fmeetings%2F2020JPSSGOES%2FFriday%2FS18_4_PGRR_Summit_Feb_2020_Shontz.pptx.
[24]	Bitzes, G., Luchetti, F., Manzi, A., Patrascoiu, M., Peters, A.J., Simon, M., & Sindrilaru, E.A. EOS architectural evolution and strategic development directions[N/OL]. EPJ Web of Conferences, 2020, 245:04009.DOI: https://doi.org/10.1051/epjconf/202024504009.
[25]	Giordano, D., Paltenghi, M., Metaj, S., & Dvořák, A. Anomaly detection in the CERN cloud infrastructure[N/OL]. EPJ Web of Conferences, 2021, 251:02011.DOI: https://doi.org/10.1051/epjconf/202125102011.
[26]	Cesini, D., Donvito, G., Costantini, A., Gómez, F.A., Duma, D.C., Fuhrmann, P., Dutka, L., Viljolen, M., Battaglia, S., Poireau, V., dell'Agnello, L., Keeble, O., Lemrani, R., Ohmann, C., & Lucas, J.M. The eXtreme-DataCloud project solutions for data management ser-vices in distributed e-infrastructures[N/OL]. EPJ Web of Conferences, 2020, 245:04010.DOI: https://doi.org/10.1051/epjconf/202024504010.
[27]	Zhang, Q., Cheng, L., and Boutaba, R. Cloud computing: State-of-the-art and research challenges[J/OL]. Journal of Internet Services and Applications, 2010, 1:7-18. https://doi.org/10.1007/s13174-010-0007-6. doi: 10.1007/s13174-010-0007-6
[28]	石蕾, 高孟绪, 徐波, 等. 欧美建设发展科学数据中心的经验及对我国的启示[J]. 中国科技资源导刊, 2022, 54(03):31-36.
[29]	China Earthquake Data Center. National Seismological Science Data Resource Manual Finalized[EB/OL].[2018-3-12]. http://data.earthquake.cn/gxdt/info/2018/61557.html.
[30]	科学数据银行. ScienceDB发布[J]. 电子世界, 2021,(03):2.
[31]	GEO[EB/OL]. [2022]. https://www.earthobservations.org/geo_community.php.
[32]	Bansal, S., Patel, S., Shah, I., Patel, P., Makwana, P., & Thakker, D. R. AGDC: Automatic garbage detection and collection[J]. arXiv preprint, 2019, arXiv:1908.05849.
[33]	Terra.bio[EB/OL]. [2022]. https://terra.bio/.
[34]	Madduri, R. K., Sulakhe, D., Lacinski, L., Liu, B., Rodriguez, A., Chard, K.,... & Foster, I. T. Experiences building Globus Genomics: a next-generation sequencing analysis service using Galaxy, Globus, and Amazon Web Services[J]. Concurrency and Computation: Practice and Experience, 2014, 26(13):2266-2279. pmid: 25342933
[35]	Robertson, T., Döring, M., Guralnick, R.P., Bloom, D., Wieczorek, J., Braak, K., Otegui, J., Russell, L.A., & Desmet, P. The GBIF Integrated Publishing Toolkit: Facilitating the Efficient Publishing of Biodiversity Data on the Internet[J/OL]. PLoS ONE, 2014, 9(8):e102623.DOI:10.1371/journal.pone.0102623. doi: 10.1371/journal.pone.0102623
[36]	IVOA[EB/OL]. [2022]. https://www.aovi.education/en/about-us/.