Abstract:

[Objective] In order to effectively solve the problems of complex multi-step and high-throughput computing in scientific domain applications, this paper studies the key technologies of workflows for the scientific application platform. [Context] This paper combines the scientific application platform based on the high-performance computing environment with the concept of workflow. It is also suitable for multi-domain and multi-system scientific computing software, providing strong support for scientific research and engineering development of related high-performance computing applications. [Methods] In response to different application requirements in different fields, multi-task concatenation workflow and high-throughput application computing workflow are designed and implemented. The multi-task concatenation workflow not only realizes a general customized workflow logic scheme on the server side and client side, allowing users to design multi-task concatenation independently, but also encapsulates domain-specific workflows in the high-performance computing environment to meet more specific and proprietary requirements. The high-throughput application computing workflow improves efficiency by using multi-process concurrency and asynchronous file upload streams when tasks are independent of each other. When tasks are interrelated, batch files are generated by script, which interacts with the high-performance computing environment only once. Under the available computing resources, a two-layer master-slave mode load balancing scheme is adopted to achieve collaborative concurrency among subtasks. [Results] Compared with the common task submission method on the platform, the multi-task concatenation workflow can save users’ time up to 10 times, and the high-throughput application computing workflow can demonstrate significant advantages in terms of time consumption, ease of use, and degree of automation. [Conclusions] The scientific application platform workflow designed and implemented in this paper can meet numerous complex application requirements in a more efficient and automated way, bringing high quality high-performance computing application services to the majority of researchers.

Key words: high performance computing application services, workflows, scientific application platform