Abstract:

[Background] Computer-aided engineering (CAE) plays a crucial role in equipment design and development. Recent advances in large language models (LLMs) present new possibilities for intelligent CAE assistance. [Methods] This study explores the capability of LLMs in CAE tasks by designing a dedicated evaluation benchmark, CAE-Bench, focusing on structural mechanics simulation. The benchmark consists of three capability levels—knowledge retention, problem-solving, and simulation application—spanning six fundamental subjects and nine subfields. Based on CAE-Bench, we develop a dataset of 3,340 multiple-choice questions to systematically assess 15 prominent LLMs. [Conclusions] Although LLMs exhibit a basic understanding of CAE knowledge, their proficiency remains limited. Average accuracy reaches 70% on knowledge-retention questions but falls to 50% on problem-solving questions. For simulation-application tasks, which require reasoning and comprehensive analysis, the accuracy declines further as task difficulty increases. Performance also varies substantially across application scenarios, with accuracies differing by up to a factor of four, indicating that a considerable gap still remains before these models can be deployed in real-world engineering applications. This study provides a feasible framework for evaluating LLMs in CAE domain and can serve as a reference for future endeavors in intelligent CAE and automated simulation.

Key words: large language models, benchmark, computer-aided engineering, structural mechanics simulation