Abstract:

[Background] The Lustre file system is a crucial foundation for scientific computing. With the continuous expansion of storage capacity and user workload, the load on storage systems is constantly increasing. Abnormal read and write requests from a single user often lead to storage cluster lag, affecting the overall user experience. Traditional anomaly handling typically involves maintainers browsing logs, identifying and locating abnormal users or read/write requests, and finally implementing troubleshooting strategies to restore system access speed. [Purpose] To improve the efficiency of anomaly diagnosis and replace the inefficient mode of relying on manual screening of logs by maintainers, this study introduces deep learning into the operation and maintenance process to achieve intelligent diagnosis of abnormal read and write behavior. [Method] This study builds an intelligent abnormal behavior detection system for Lustre file systems, involving user behavior data collection, time-series data processing, model construction, training, and deployment verification. The system converts user read and write information into time series data, builds a long short-term memory network and trains the model using unsupervised learning. [Conclusions] This study successfully trained and validated the model on Lustre's MDT and OST data. Experimental results show that the proposed method can significantly improve the accuracy of anomaly detection and effectively reduce the false alarm rate. The proposed method can reduce the time cost for error localization for maintainers and improve the efficiency of anomaly handling in file systems within scientific computing environments.

Key words: AIOps, storage system, time-series data processing