Abstract:

[Objective] In order to meet the demand for remote access to massive data generated by the high energy synchrotron radiation source (HEPS), an efficient and adaptable solution based on HDF5 data format is proposed. [Background] The HEPS is expected to produce a tremendous volume of scientific data annually. By adopting HDF5 as the unified data format and employing the "centralized data storage, remotely distributed processing" computing paradigm, users are no longer required to download data in advance. Instead, computing tasks can be seamlessly run at the remote computing center. [Methods] By analyzing the current storage infrastructure of the HEPS data center and existing data access methods, the XHDF5 system is designed and implemented. Leveraging HDF5 VOL and XRootD technologies, this system facilitates seamless and transparent access to HDF5 data. Moreover, the system incorporates advanced techniques such as concurrent access and large-scale data compression to improve access performance. [Results] After comprehensive technical testing, detailed performance comparison experiments, and rigorous large-scale data processing evaluations, the XHDF5 system has demonstrated outstanding performance and remarkable stability during remote HDF5 file access. In comparison to traditional access methods that rely on file system mounting and H5serv technology, the XHDF5 system exhibits superior performance, particularly in high network latency environments. [Conclusion] The XHDF5 system effectively supports cross-region data sharing and collaboration, providing researchers with a stable and reliable data access platform to facilitate scientific research.

Key words: XRootD, HDF5, HEPS, NFS, remote data access