Abstract:

[Context] The Murchison Widefield Array (MWA) is a low-frequency precursor telescope for the Square Kilometre Array (SKA), which is widely used in the study of astronomical phenomena such as pulsars. Due to the large scale of data transmission and storage, coupled with challenges in data processing, the read-write performance is low, thereby affecting the efficiency of data processing. [Object] To enhance the data processing efficiency of the MWA, a pre-processing optimization of storage layout is proposed to alleviate the read-write bottlenecks. [Methods] By analyzing the data characteristics and computational workflows of the MWA, a vertical data layout strategy is introduced. This approach, combining local computation modes with a pipeline architecture, achieves efficient data pre-processing and layout adjustment. [Results] The proposed solution optimizes the data access strategy by incorporating packing and compression techniques that reduce the number of files by a factor of 40 and the data volume by 70%. With the local computation mode, the shared storage I/O load is reduced, significantly enhancing the efficiency of astronomical data analysis. The data pre-processing solution using local computation mode achieves threefold improvement in computational efficiency. [Conclusions] The data layout strategy and pre-processing optimization methods proposed in this study can significantly improve the storage performance of SKA-MWA astronomical data and the computational efficiency of subsequent beamforming. This approach provides high-quality data support for astronomical computations and is universally applicable with broad prospects for future application.

Key words: data preprocessing, distributed parallelism, storage optimization, local storage