Research on the Application of Fine-Tunned Large Language Models Based on LoRA in Quality Evaluation of the Security Level Protection Assessment Reports

doi:10.11871/jfdc.issn.2096-742X.2025.06.011

Frontiers of Data and Computing ›› 2025, Vol. 7 ›› Issue (6): 111-123.

CSTR: 32002.14.jfdc.CN10-1649/TP.2025.06.011

doi: 10.11871/jfdc.issn.2096-742X.2025.06.011

• Technology and Application • Previous Articles Next Articles

Research on the Application of Fine-Tunned Large Language Models Based on LoRA in Quality Evaluation of the Security Level Protection Assessment Reports

WU Jianhua^1,^2,^*(),LIU Zhenyu^1,²,ZENG Rui^1,²,WANG Wenxuan^1,²,YI Yong^1,²,WANG Shiyi^1,²

1. Shanghai Key Laboratory of Computer Software Testing Evaluating, Shanghai 201112, China
2. Shanghai Development Center of Computer Software Technology, Shanghai 201112, China

Received:2025-02-14 Online:2025-12-20 Published:2025-12-17
Contact: WU Jianhua E-mail:wjh@sscenter.sh.cn
About author:WU Jianhua is the director of the Cybersecurity Research Institute at the Shanghai Computer Software Technology Development Center. He holds a master's degree and is a senior engineer. His main research areas include cybersecurity and testing technology.
In this paper, he is responsible for determining the research topic, formulating the paper framework, and revising the paper.
E-mail: wjh@sscenter.sh.cn

Abstract

Abstract:

[Purpose/Significance] The network security level protection assessment is of great importance in improving the security of network systems, meeting compliance requirements, and promoting continuous improvement. However, errors and contradictions in evaluation records often occur in the reports, which can seriously affect the conclusions of the reports. [Methods] This paper innovatively proposes a method applying a fine-tunned large language model based on LoRA to the review of level protection reports. First, the detection needs are converted into a multi-classification task based on logical reasoning and natural language understanding. Then, a fine-tuning dataset is constructed based on this, and the glm-4-9b-1m-chat, chatglm3-6b and deepseek-r1-distill-qwen-7b large models are fine-tuned based on the dataset. [Results and Analysis] The experimental results show that the fine-tuned large model has a significant improvement in Bleu-4 and Rouge-1 values compared to the base model, and the accuracy of the multi-classification reaches 87%. [Conclusion] After fine-tuning, the output quality of the large model is improved and can better meet the requirements of quality inspection for graded protection reports in this vertical domain. The LoRA fine-tuning of the large model proposed in this paper provides a new perspective for quality evaluation of the level protection report documents and other types of documents.

Key words: large language models, LoRA fine-tuning, network security, document quality detection, level protection evaluation

WU Jianhua,LIU Zhenyu,ZENG Rui,WANG Wenxuan,YI Yong,WANG Shiyi. Research on the Application of Fine-Tunned Large Language Models Based on LoRA in Quality Evaluation of the Security Level Protection Assessment Reports[J]. Frontiers of Data and Computing, 2025, 7(6): 111-123, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2025.06.011.

Figures/Tables 14

Fig.1

Fig.2

Fig.3

Table 1

Fig.4

Fig.5

Fig.6

Table 2

Fig.7

Table 3

Table 4

Fig.8

Fig.9

Fig.10

References 22

[1]	张海龙, 黄文锋, 路翔, 等. 知识增强大模型在信息系统故障分析中的应用研究[J]. 现代计算机, 2024, 30(6): 87-93.
[2]	艾洲. 基于大模型和区块链的电力知识问答系统设计与实现[J]. 电力大数据, 2024, 27(1): 87-96.
[3]	沈晨晨, 岳盛斌, 刘书隽, 等. 面向法律领域的大模型微调与应用[J]. 大数据, 2024, 10(5): 12-27.
[4]	王婷, 王娜, 崔运鹏, 等. 基于人工智能大模型技术的果蔬农技知识智能问答系统[J]. 智慧农业(中英文), 2023, 5(4): 105-116.
[5]	TAN Y, ZHANG Z, LI M, et al. MedChatZH: A tuning LLM for traditional Chinese medicine consultations[J]. Computers in biology and medicine, 2024: 1-13.
[6]	TRAD F, CHEHAB A. Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models[J]. Machine Learning and Knowledge Extraction, 2024: 367-384.
[7]	YEOM J, LEE H, BYUN H, et al. Tc-llama 2: fine-tuning LLM for technology and commercialization applications[J]. Journal of Big Data, 2024: 100-131.
[8]	刘烨, 杨良斌. 基于预训练模型的漏洞信息检索系统研究[J]. 情报杂志, 2024, 43(8): 84-91.
[9]	田雄军, 崔佳斌, 袁礼, 等. 大规模预训练模型在网络安全领域的应用研究[J]. 网络空间安全, 2024, 15(3):68-71.
[10]	朱玥. 大语言模型赋能网络威胁情报分析: 场景、风险和发展路径[J]. 情报杂志, 2025, 44(4): 23-30.
[11]	李娇, 张玉清, 吴亚飚. 面向网络安全关系抽取的大语言模型数据增强方法[J]. 信息网络安全, 2024, 24(10): 1477-1483.
[12]	池亚平, 吴冰, 徐子涵. 大语言模型在威胁情报生成方面的研究进展[J]. 信息安全研究, 2024, 10(11):1028-1035.
[13]	李镭, 郭志君, 罗淑丹. 网络安全威胁检测中基于大语言模型的实时分析策略研究[J]. 通信技术, 2024, 57(9): 949-954.
[14]	李晓东, 宋宜昌, 江政旦, 等. 基于大语言模型的SQL注入攻击检测方法研究[J]. 通讯世界, 2024, 31(6): 61-63.
[15]	张长琳, 仝鑫, 佟晖, 等. 面向网络安全领域的大语言模型技术综述[J]. 信息网络安全, 2024, 24(5):778-793.
[16]	张弛, 翁方宸, 张玉清. ChatGPT在网络安全领域的应用、现状与趋势[J]. 信息安全研究, 2023, 9(6):500-509.
[17]	JAMAL S, WIMMER H, SARKER H I. An improved transformer-based model for det ecting phishing, sp am and ham emails: A large language model appro-ach[J]. Security and Privacy, 2024, 7(5): e402.
[18]	HASSANIN M, KESHK M, SALIM S, et al. PLLM-CS: Pre-trained Large Language Model (LLM) for cyber threat detection in satellite networks[J]. Ad Hoc Networks, 2025, 166: 103645.
[19]	WU J, CHEN L, FANG S, et al. An Application Programming Interface (API) Sensitive Data Identification Method Based on the Federated Large Language Model[J]. Applied Sciences, 2024, 14(22): 10162.
[20]	CHEN M, ZHU K, LU B, et al. AECR: Automatic attack technique intelligence extraction based on fine-tuned large language model[J]. Computers & Security, 2025, 150: 104213.
[21]	HU E, SHEN Y, WALLIS P, et al. LoRA: Low-Rank Adaptation of Large Language Models[EB/OL]. arXiv, 2021.
[22]	HIYOUGA, BUAADREAMER, KUANGDD, et al. LLaMA-Factory[Z]. https://github.com/hiyouga/LLaMA-Factory.

组成部分	具体内容
风险测评项	设备种类、安全控制点、结果记录、符合程度
《判定指引》中与风险测评项有关的内容	标准要求、判例场景、补偿因素
等保报告记录中与风险测评项有关内容	安全问题、关联资产、危害分析结果

超参数	数值
训练步数	40
学习率	5e-5
最大梯度范数	1
LoRA的秩	32
截断长度	1,024
批处理大小	2
优化器	Adam

基座模型	评估指标
	Bleu-4		Rouge-1
	微调前	微调后	微调前	微调后
glm-4-9b-1m-chat	1.4	96.8	10.7	97.0
chatglm3-6b	1.1	90.5	8.5	90.1
deepseek-r1-distill-qwen-7b	1.3	94.5	9.6	94.8

样例	评估指标
样例	Precison	Recall	F1
1	1.00	1.00	1.00
2	0.55	0.62	0.58
3	0.64	0.57	0.60
4	1.00	1.00	1.00
5	1.00	1.00	1.00
6	1.00	1.00	1.00
Accuracy	0.87

Research on the Application of Fine-Tunned Large Language Models Based on LoRA in Quality Evaluation of the Security Level Protection Assessment Reports

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 14

References 22

Related Articles 4

Recommended Articles

Metrics

Comments

[1]	LIANG Fei,ZHANG Shixing,CHENG Zirui. A Blockchain Anomaly Transaction Detection Model Based on Threat Environment Perception and Large Language Model Feature Enhancement [J]. Frontiers of Data and Computing, 2025, 7(6): 23-34.
[2]	LIU Dianyu,LIU Qingkai,XIAO Yuyang,WANG Jie. CAE-Bench：An Evaluation of Large Language Models in Structural Mechanics Simulation [J]. Frontiers of Data and Computing, 2025, 7(4): 155-168.
[3]	MA Qiuping, ZHANG Qi, ZHAO Xiaofan. Review of Research on Chart Question Answering [J]. Frontiers of Data and Computing, 2025, 7(1): 19-37.
[4]	PEI Bingsen,LI Xin,JIANG Zhangtao,LIU Mingshuai. Research on the Generation and Evaluation of Judicial Text Summarization Based on Large Language Models [J]. Frontiers of Data and Computing, 2024, 6(6): 62-73.