基于Rucio的高能物理网格数据管理的研究和应用

doi:10.11871/jfdc.issn.2096-742X.2024.03.006

数据与计算发展前沿 ›› 2024, Vol. 6 ›› Issue (3): 58-66.

CSTR: 32002.14.jfdc.CN10-1649/TP.2024.03.006

doi: 10.11871/jfdc.issn.2096-742X.2024.03.006

基于Rucio的高能物理网格数据管理的研究和应用

张玄同^1,^*(),张晓梅¹,胡皓^1,²,王浩帆¹

1.中国科学院高能物理研究所，北京 100049
2.中国科学技术大学，安徽合肥 230026

收稿日期:2023-10-30 出版日期:2024-06-20 发布日期:2024-06-21
通讯作者: *张玄同（E-mail：zhangxuantong@ihep.ac.cn）
作者简介:张玄同，中国科学院高能物理研究所计算中心，助理研究员，主要研究方向为国际网格框架下分布式计算系统和数据管理系统。
负责论文初稿撰写与相关软件应用的开发。
ZHANG Xuantong, is a Research Associate of Computing Center of Institute of High Energy Physics, Chinese Academy of Sciences. His research interests include distributed computing systems and data management systems within the WLCG framework.
In this paper, he is responsible for the paper drafting and relevant software applications development.
E-mail: zhangxuantong@ihep.ac.cn

Research and Applications of High Energy Physics Grid Data Management Based on Rucio

ZHANG Xuantong^1,^*(),ZHANG Xiaomei¹,HU Hao^1,²,WANG Haofan¹

1. Institute of High Energy Physics, Chinese Academy of Sciences, Beijing 100049, China
2. University of Science and Technology of China, Hefei, Anhui 230026, China

Received:2023-10-30 Online:2024-06-20 Published:2024-06-21

摘要/Abstract

摘要：

【目的】近年来，高能物理网格数据规模和用户需求产生重大变革，需要研究和应用新兴网格数据管理技术以应对需求变化。【方法】基于新型网格数据管理系统Rucio，利用其高伸缩性、模块化和可扩展性的软件特点，发挥其分布式数据恢复、自适应的数据复制的功能特性，为多个国内主导的国际合作实验设计了面向实验需求的网格数据管理解决方案。【结果】实现了分布式数据统一命名、数据增删改查等基础管理功能、多站点数据副本管理、原始数据分发管理、实验软件数据管理接口嵌入等多种功能，并先后进入了测试应用阶段。【结论】本研究为未来国内主导的国际合作的高能物理实验网格数据管理方案的设计和开发进行了探索和尝试，希望进一步在网格架构上开展深入研究，实现国内实验通用标准的网格数据管理方案。

关键词: 网格计算, 分布式计算, 网格数据管理, 高能物理

Abstract:

[Objective] In recent years, significant changes have occurred in the scale of high-energy physics Grid data and user requirements, necessitating research and the application of emerging Grid data management technologies to adapt to changing demands. [Methods] Based on the novel Grid data management system Rucio, and leveraging its characteristics of high scalability, modularity, and extensibility, the functionalities of distributed data recovery and adaptive data replication are utilized to design Grid data management solutions tailored to the experimental requirements of several China-led international collaborative experiments. [Results] Multiple functions are realized, including uniform naming of distributed data, basic management functions for data creation, modification, retrieval, and deletion, multi-site data replica management, original data distribution management, and embedding of interfaces for experiment software data management. Various tests and applications are conducted in different stages. [Conclusions] This study explores the design and development of Grid data management solutions for China-led international collaborative high-energy physics experiments. Future research can further delve into Grid architecture to achieve universal standard Grid data management solutions.

Key words: grid computing, distributed computing, grid data management, high energy physics

张玄同, 张晓梅, 胡皓, 王浩帆. 基于Rucio的高能物理网格数据管理的研究和应用[J]. 数据与计算发展前沿, 2024, 6(3): 58-66.

ZHANG Xuantong, ZHANG Xiaomei, HU Hao, WANG Haofan. Research and Applications of High Energy Physics Grid Data Management Based on Rucio[J]. Frontiers of Data and Computing, 2024, 6(3): 58-66, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.03.006.

图/表 5

图1

图2

表1

图3

图4

参考文献 26

[1]	HERMANN A, KRIGE J, MERSITS U, et al. History of CERN[M]. Amsterdam: North-Holland, 1987.
[2]	SHIERS J. The worldwide LHC computing grid (worldwide LCG)[J]. Computer physics Communications, 2007, 177(1-2): 219-223.
[3]	LAMANNA M. The LHC computing grid project at CERN[J]. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2004, 534(1-2): 1-6.
[4]	EVANS L. The large hadron collider[J]. New Journal of Physics, 2007, 9(9): 335.
[5]	GREY F, LIN S C. Asia federation report on international symposium on grid computing (ISGC) 2010[C]// Data Driven e-Science: Use Cases and Successful Applications of Distributed Computing Infrastructures (ISGC 2010). Springer New York, 2011:3-36.
[6]	ABLIKIM M, AN Z H, BAI J Z, et al. Design and construction of the BESIII detector[J]. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2010, 614(3): 345-399.
[7]	ADAM T, AN F, AN G, et al. JUNO conceptual design report[J]. arXiv preprint arXiv:1508.07166, 2015.
[8]	ZHANG S N, ADRIANI O, ALBERGO S, et al. The high energy cosmic-radiation detection (HERD) facility onboard China's Space Station[C]// Space Telescopes and Instrumentation 2014:Ultraviolet to Gamma Ray. SPIE, 2014, 9144: 293-301.
[9]	BARISITS M, BEERMANN T, BERGHAUS F, et al. Rucio: Scientific data management[J]. Computing and Software for Big Science, 2019, 3(1): 1-19.
[10]	JONES M, BRADLEY J, SAKIMURA N. Json web token (jwt)[R]. 2015.
[11]	HOUSLEY R, FORD W, POLK W, et al. Internet X.509 public key infrastructure certificate and CRL profile[R]. 1999.
[12]	ALFIERI R, CECCHINI R, CIASCHINI V, et al. From gridmap-file to VOMS: managing authorization in a Grid environment[J]. Future Generation Computer Systems, 2005, 21(4): 549-558.
[13]	SAKIMURA N, BRADLEY J, JONES M, et al. Openid connect core 1.0[J]. The OpenID Foundation, 2014: S3.
[14]	BOCKELMAN B, CECCANTI A, COLLIER I, et al. WLCG Authorisation from X. 509 to Tokens[C]// EPJ Web of Conferences. EDP Sciences, 2020, 245: 03001.
[15]	FORTI A. Modernizing Third-Party-Copy Transfers in WLCG[C]// EPJ Web of Conferences. 2019.
[16]	ADYE T, BOCKELMAN B, ELLIS K, et al. XRootD Third Party Copy for the WLCG and HLLHC[C]// EPJ Web of Conferences. EDP Sciences, 2020, 245: 04034.
[17]	BOCKELMAN B, CECCANTI A, FURANO F, et al. Third-party transfers in WLCG using HTTP[C]// EPJ Web of Conferences. EDP Sciences, 2020, 245: 04031.
[18]	ALLCOCK W, BRESNAHAN J, KETTIMUTHU R, et al. The Globus striped GridFTP framework and server[C]// SC'05: Proceedings of the 2005 ACM/IEEE conference on Supercomputing. IEEE, 2005: 54-54.
[19]	SHOSHANI A, SIM A, GU J. Storage Resource Managers: Mitldleware Components for Grid Storage[C]// Tenth Goddard Conference on Mass Storage Systems and Technologies. 2002: 209.
[20]	FUHRMANN P, GÜLZOW V. dCache, storage system for the future[C]// Euro-Par 2006 Parallel Processing: 12th International Euro-Par Conference, Dresden, Germany, Springer Berlin Heidelberg, 2006: 1106-1113.
[21]	PETERS A J, SINDRILARU E A, ADDE G. EOS as the present and future solution for data storage at CERN[C]// Journal of Physics: Conference Series. IOP Publishing, 2015, 664(4): 042042.
[22]	ELMSHEUSER J, DI GIROLAMO A. Overview of the ATLAS distributed computing system[C]// EPJ Web of Conferences. EDP Sciences, 2019, 214: 03010.
[23]	陈思思, 曾珊, 李卫东, 等. 高能物理实验数据传输系统的研究和实现[J]. 计算机工程与应用, 2019, 55(13): 218-224. doi: 10.3778/j.issn.1002-8331.1712-0325
[24]	TSAREGORODTSEV A, BARGIOTTI M, BROOK N, et al. DIRAC: a community grid solution[C]// Journal of Physics: Conference Series. IOP Publishing, 2008, 119(6): 062048.
[25]	TSAREGORODTSEV A, POSS S. Dirac file replica and metadata catalog[C]// Journal of Physics: Conference Series. IOP Publishing, 2012, 396(3): 032108.
[26]	SERFON C, MASHINISTOV R, DE STEFANO J S, et al. Integration of Rucio in Belle II[C]// EPJ Web of Conferences. EDP Sciences, 2021, 251: 02057.

DID类型	举例	定义
文件	temp:/herd/user/output.root	数据文件的逻辑名，满足Linux文件路径规则。
数据集	temp:/herd/user/	某个文件目录下所有文件，以“/”号结尾。
数据容器	temp:/herd/user	某个文件目录下所有子目录的集合，不以“/”结尾。

基于Rucio的高能物理网格数据管理的研究和应用

Research and Applications of High Energy Physics Grid Data Management Based on Rucio

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 5

参考文献 26

相关文章 1

编辑推荐

Metrics

本文评价