Research and Applications of High Energy Physics Grid Data Management Based on Rucio

doi:10.11871/jfdc.issn.2096-742X.2024.03.006

Abstract

Abstract:

[Objective] In recent years, significant changes have occurred in the scale of high-energy physics Grid data and user requirements, necessitating research and the application of emerging Grid data management technologies to adapt to changing demands. [Methods] Based on the novel Grid data management system Rucio, and leveraging its characteristics of high scalability, modularity, and extensibility, the functionalities of distributed data recovery and adaptive data replication are utilized to design Grid data management solutions tailored to the experimental requirements of several China-led international collaborative experiments. [Results] Multiple functions are realized, including uniform naming of distributed data, basic management functions for data creation, modification, retrieval, and deletion, multi-site data replica management, original data distribution management, and embedding of interfaces for experiment software data management. Various tests and applications are conducted in different stages. [Conclusions] This study explores the design and development of Grid data management solutions for China-led international collaborative high-energy physics experiments. Future research can further delve into Grid architecture to achieve universal standard Grid data management solutions.

Key words: grid computing, distributed computing, grid data management, high energy physics

ZHANG Xuantong, ZHANG Xiaomei, HU Hao, WANG Haofan. Research and Applications of High Energy Physics Grid Data Management Based on Rucio[J]. Frontiers of Data and Computing, 2024, 6(3): 58-66, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.03.006.

Figures/Tables 5

Fig.1

Fig.2

Table 1

Fig.3

Fig.4

References 26

[1]	HERMANN A, KRIGE J, MERSITS U, et al. History of CERN[M]. Amsterdam: North-Holland, 1987.
[2]	SHIERS J. The worldwide LHC computing grid (worldwide LCG)[J]. Computer physics Communications, 2007, 177(1-2): 219-223.
[3]	LAMANNA M. The LHC computing grid project at CERN[J]. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2004, 534(1-2): 1-6.
[4]	EVANS L. The large hadron collider[J]. New Journal of Physics, 2007, 9(9): 335.
[5]	GREY F, LIN S C. Asia federation report on international symposium on grid computing (ISGC) 2010[C]// Data Driven e-Science: Use Cases and Successful Applications of Distributed Computing Infrastructures (ISGC 2010). Springer New York, 2011:3-36.
[6]	ABLIKIM M, AN Z H, BAI J Z, et al. Design and construction of the BESIII detector[J]. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2010, 614(3): 345-399.
[7]	ADAM T, AN F, AN G, et al. JUNO conceptual design report[J]. arXiv preprint arXiv:1508.07166, 2015.
[8]	ZHANG S N, ADRIANI O, ALBERGO S, et al. The high energy cosmic-radiation detection (HERD) facility onboard China's Space Station[C]// Space Telescopes and Instrumentation 2014:Ultraviolet to Gamma Ray. SPIE, 2014, 9144: 293-301.
[9]	BARISITS M, BEERMANN T, BERGHAUS F, et al. Rucio: Scientific data management[J]. Computing and Software for Big Science, 2019, 3(1): 1-19.
[10]	JONES M, BRADLEY J, SAKIMURA N. Json web token (jwt)[R]. 2015.
[11]	HOUSLEY R, FORD W, POLK W, et al. Internet X.509 public key infrastructure certificate and CRL profile[R]. 1999.
[12]	ALFIERI R, CECCHINI R, CIASCHINI V, et al. From gridmap-file to VOMS: managing authorization in a Grid environment[J]. Future Generation Computer Systems, 2005, 21(4): 549-558.
[13]	SAKIMURA N, BRADLEY J, JONES M, et al. Openid connect core 1.0[J]. The OpenID Foundation, 2014: S3.
[14]	BOCKELMAN B, CECCANTI A, COLLIER I, et al. WLCG Authorisation from X. 509 to Tokens[C]// EPJ Web of Conferences. EDP Sciences, 2020, 245: 03001.
[15]	FORTI A. Modernizing Third-Party-Copy Transfers in WLCG[C]// EPJ Web of Conferences. 2019.
[16]	ADYE T, BOCKELMAN B, ELLIS K, et al. XRootD Third Party Copy for the WLCG and HLLHC[C]// EPJ Web of Conferences. EDP Sciences, 2020, 245: 04034.
[17]	BOCKELMAN B, CECCANTI A, FURANO F, et al. Third-party transfers in WLCG using HTTP[C]// EPJ Web of Conferences. EDP Sciences, 2020, 245: 04031.
[18]	ALLCOCK W, BRESNAHAN J, KETTIMUTHU R, et al. The Globus striped GridFTP framework and server[C]// SC'05: Proceedings of the 2005 ACM/IEEE conference on Supercomputing. IEEE, 2005: 54-54.
[19]	SHOSHANI A, SIM A, GU J. Storage Resource Managers: Mitldleware Components for Grid Storage[C]// Tenth Goddard Conference on Mass Storage Systems and Technologies. 2002: 209.
[20]	FUHRMANN P, GÜLZOW V. dCache, storage system for the future[C]// Euro-Par 2006 Parallel Processing: 12th International Euro-Par Conference, Dresden, Germany, Springer Berlin Heidelberg, 2006: 1106-1113.
[21]	PETERS A J, SINDRILARU E A, ADDE G. EOS as the present and future solution for data storage at CERN[C]// Journal of Physics: Conference Series. IOP Publishing, 2015, 664(4): 042042.
[22]	ELMSHEUSER J, DI GIROLAMO A. Overview of the ATLAS distributed computing system[C]// EPJ Web of Conferences. EDP Sciences, 2019, 214: 03010.
[23]	陈思思, 曾珊, 李卫东, 等. 高能物理实验数据传输系统的研究和实现[J]. 计算机工程与应用, 2019, 55(13): 218-224. doi: 10.3778/j.issn.1002-8331.1712-0325
[24]	TSAREGORODTSEV A, BARGIOTTI M, BROOK N, et al. DIRAC: a community grid solution[C]// Journal of Physics: Conference Series. IOP Publishing, 2008, 119(6): 062048.
[25]	TSAREGORODTSEV A, POSS S. Dirac file replica and metadata catalog[C]// Journal of Physics: Conference Series. IOP Publishing, 2012, 396(3): 032108.
[26]	SERFON C, MASHINISTOV R, DE STEFANO J S, et al. Integration of Rucio in Belle II[C]// EPJ Web of Conferences. EDP Sciences, 2021, 251: 02057.

DID类型	举例	定义
文件	temp:/herd/user/output.root	数据文件的逻辑名，满足Linux文件路径规则。
数据集	temp:/herd/user/	某个文件目录下所有文件，以“/”号结尾。
数据容器	temp:/herd/user	某个文件目录下所有子目录的集合，不以“/”结尾。