Abstract:

[Objective] A large-scale scientific computing task often includes multiple computing jobs or a job group, and there are execution orders and dependencies between multiple computing jobs. Users need to wait for the previous job to complete before submitting the next one. In order to reduce the user waiting time, there is an urgent need for new ways of submitting jobs that allows users to submit multiple jobs with dependencies at the same time. [Methods] This paper proposes an intelligent task orchestration scheme for high-performance computing environments, which can automatically resolve dependencies between jobs, intelligently orchestrate job submission sequences, monitor job status, and submit the subsequent job after the depending job is completed. [Results] From the perspective of practical application effects, the intelligent task orchestration service can effectively simplify user operations. [Conclusions] The scheme proposed achieves a good application effect.

Key words: high performance computing environment, job group, job dependency, intelligent task orchestration