Llama2-70b模型的微调技术及其在材料领域的应用研究

doi:10.11871/jfdc.issn.2096-742X.2025.01.012

数据与计算发展前沿 ›› 2025, Vol. 7 ›› Issue (1): 163-174.

CSTR: 32002.14.jfdc.CN10-1649/TP.2025.01.012

doi: 10.11871/jfdc.issn.2096-742X.2025.01.012

Llama2-70b模型的微调技术及其在材料领域的应用研究

唐雷^1,²(),陈子逸^1,²,梁锶翰^1,²,李凯¹,万萌¹,张博尧¹,刘淼³,孟胜³,王彦棡^1,²,周纯葆^1,^2,^*(),王宗国^1,^2,^*()

1.中国科学院计算机网络信息中心，北京 100083
2.中国科学院大学，北京 100049
3.中国科学院物理研究所，北京 100190

收稿日期:2024-11-14 出版日期:2025-02-20 发布日期:2025-02-21
通讯作者: *周纯葆（E-mail: zhoucb@sccas.cn）;王宗国（E-mail: wangzg@cnic.cn）
作者简介:唐雷，中国科学院计算机网络信息中心，硕士研究生，CCF会员。主要研究方向为人工智能应用、材料信息学。
本文主要承担工作为模型整体架构设计、实现、评估与论文撰写。
TANG Lei is Master’s student at the Computer Network Information Center, Chinese Academy of Sciences. He is a CCF Student Member. His research interests include Artificial Intelligence Application, and Materials Information Science.
In this paper, he is mainly responsible for overall model architecture design, implementation and evaluation, paper writing.
E-mail: ltang@cnic.cn|周纯葆，中国科学院计算机网络信息中心，博士，研究员，硕士生导师，中国科学院青促会会员。主要研究方向为异构计算、人工智能基础算法与软件。
本文主要承担工作为模型整体架构设计及应用示范。
ZHOU Chunbao, Ph.D., is a professor and master’s supervisor at the Computer Network Information Center, Chinese Academy of Sciences. He is also a member of Youth Innovation Promotion Association, Chinese Academy of Sciences. His research interests include heterogeneous computing, basic algorithms and software of artificial intelligence.
In this paper, he is mainly responsible for the overall model design.
E-mail: zhoucb@sccas.cn|王宗国，中国科学院计算机网络信息中心，博士，副研究员，硕士生导师，中国科学院青促会会员。主要研究方向为人工智能应用、材料信息学。
本文主要承担工作为模型整体架构设计及应用示范。
WANG Zongguo is an associate professor and master superviosr at the Computer Network Information Center, Chinese Academy of Sciences. She is also a member of Youth Innovation Promotion Association, Chinese Academy of Sciences. Her research interests include Artificial Intelligence Application, and Materials Information Science.
In this paper, she is mainly responsible for the overall model design and its application demonstration.
E-mail: wangzg@cnic.cn
基金资助:
中国科学院网信专项“能源材料端到端设计的信息化智能平台”(CAS-WX2023SF-0101);中国科学院前沿科学重点研究计划“Ⅲ-Ⅴ族半导体材料的‘基因图谱’研究”(ZDBS-LY-7025);中国科学院青年创新促进会(2021167)

A Study of the Fine-Tuning Technique of the Llama2-70b Model and Its Application in the Field of Materials

TANG Lei^1,²(),CHEN Ziyi^1,²,LIANG Sihan^1,²,LI Kai¹,WAN Meng¹,ZHANG Boyao¹,LIU Miao³,MENG Sheng³,WANG Yangang^1,²,ZHOU Chunbao^1,^2,^*(),WANG Zongguo^1,^2,^*()

1. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China
2. University of Chinese Academy of Sciences, Beijing 100049, China
3. Institute of Physics, Chinese Academy of Sciences, Beijing 100190, China

Received:2024-11-14 Online:2025-02-20 Published:2025-02-21

摘要/Abstract

摘要：

【目的】为降低大语言模型的使用门槛，促进大语言模型在学科领域的应用。本文系统介绍了Llama2-70b模型的微调过程及其在材料领域应用的流程。【方法】本研究利用DeepSpeed框架和无机材料合成路径的指令式数据集，采用LoRA微调技术对开源大模型Llama2-70b进行微调，并对模型的超参数进行了调优，从模型训练中的损失值和模型稳定性两个方面对调优效果进行了评估，最终确定了一组适合模型的超参数组合。【结果】通过对模型的训练和优化，最终获得了一个在稳定性和性能方面表现优异的材料合成大语言模型。【结论】该研究为大语言模型在学科领域的应用提供了宝贵的经验和方法，所训练的材料大语言模型为材料合成设计提供了有意义的参考和支持。

关键词: Llama2-70b模型, LoRA, 大模型微调, 材料合成

Abstract:

[Objective] To lower the barriers of using large language models and promote their applications in different fields, this paper systematically introduces the fine-tuning process of the Llama2-70b model and its application procedure in the field of materials science. [Methods] This study utilized the DeepSpeed framework and an instruction data set of inorganic material synthesis pathways, and employed the LoRA fine-tuning technique to fine-tune the open-source Llama2-70b model. The model’s hyperparameters were optimized, and the tuning effects were evaluated based on the loss value during model training and the model’s stability. A suitable combination of hyperparameters was finally determined. [Results] Through the training and optimization of the model, a large language model for material synthesis that performs excellently in terms of stability and performance was obtained. [Conclusions] This research provides valuable experience and methods for the application of large language models in academic fields. The trained material language model offers meaningful reference and support for material synthesis design.

Key words: Llama2-70b Model, LoRA, Large Language model, material synthesis

唐雷, 陈子逸, 梁锶翰, 李凯, 万萌, 张博尧, 刘淼, 孟胜, 王彦棡, 周纯葆, 王宗国. Llama2-70b模型的微调技术及其在材料领域的应用研究[J]. 数据与计算发展前沿, 2025, 7(1): 163-174.

TANG Lei, CHEN Ziyi, LIANG Sihan, LI Kai, WAN Meng, ZHANG Boyao, LIU Miao, MENG Sheng, WANG Yangang, ZHOU Chunbao, WANG Zongguo. A Study of the Fine-Tuning Technique of the Llama2-70b Model and Its Application in the Field of Materials[J]. Frontiers of Data and Computing, 2025, 7(1): 163-174, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2025.01.012.

图/表 11

表1

图1

图2

图3

图4

图5

图6

图7

表2

图8

图9

参考文献 29

[1]	OPENAI, ACHIAM J, ADLER S, et al. GPT-4 Technical Report[EB/OL].(2023-03-08)[2023-07-09]. https://arxiv.org/abs/2303.08774.
[2]	蔡睿, 葛军, 孙哲, 等. AI预训练大模型发展综述[J/OL]. 小型微型计算机系统, 2024, 5(15): 1-12. http://kns.cnki.net/kcms/detail/21.1106.tp.20230510.1900.010.html.
[3]	吴双. AI大模型: 以“大规模预训练+微调”范式满足多元化需求[N]. 人民邮电, 2022-06-16(005).
[4]	GUPTA T, ZAKI M, KRISHNAN N A, et al. MatSciBERT: A materials domain language model for text mining and information extraction[J]. npj Computational Materials, 2022, 8(1): 102-112.
[5]	DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis: Association for Computational Linguistics, 2019: 4171-4186.
[6]	TOUVRON H, MARTIN L, STONE K, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models[EB/OL].(2023-07-09)[2023-07-09]. https://arxiv.org/abs/2307.09288.
[7]	TOUVRON H, LAVRIL T, IZACARD G, et al. LLaMA: open and efficient foundation language models[EB/OL].(2023-02-27)[2023-07-09]. https://arxiv.org/abs/2302.13971.
[8]	YANG P, WANG J J, GAN R Y, et al. Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective[EB/OL].(2022-10-08)[2023-07-09]. https://arxiv.org/abs/2210.08590.
[9]	程乐超. 视觉大模型参数高效微调技术应用与展望[J]. 人工智能, 2024(1): 54-65.
[10]	HOULSBY N, GIURGIU A, JASTRZEBSKI S, et al. Parameter-efficient transfer learning for NLP[C]// Proceedings of the 36th International Conference on Machine Learning:Proceedings of Machine Learning Research, Vol. 97. Long Beach:PMLR, 2019: 2790-2799.
[11]	HE J, ZHOU C, MA X, et al. Towards a unified view of parameter-efficient transfer learning[C]// International Conference on Learning Representations. 2022: https://openreview.net/forum?id=0RDcd5Axok.
[12]	DING N, QIN Y J, YANG G, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models[J]. Nature Machine Intelligence, 2023, 5(3):220-235.
[13]	LESTER B, AL-RFOU R, CONSTANT N. The power of scale for parameter-efficient prompt tuning[C]// Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana: Association for Computational Linguistics, 2021: 3045-3059.
[14]	SU Y, WANG X, QIN Y, et al. On transferability of prompt tuning for natural language processing[C]// Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Seattle: Association for Computational Linguistics, 2022: 3949-3969.
[15]	LI X L, LIANG P. Prefix-Tuning: optimizing continuous prompts for generation[C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:Long Papers). Online: Association for Computational Linguistics, 2021: 4582-4597.
[16]	KARIMI M R, HENDERSON J, RUDER S. Compacter: efficient low-rank hypercomplex adapter layers[J]. Advances in Neural Information Processing Systems, 2021, 34: 1022-1035.
[17]	HU E J, YELONG S, WALLIS P, et al. LoRA: low-rank adaptation of large language models[C]// International Conference on Learning Representations.ICRL, 2022: https://openreview.net/forum?id=nZeVKeeFYf9.
[18]	PEDRO C, SAYAK P. Using LoRA for efficient stable diffusion fine-tuning[EB/OL].(2023-01-26)[2023-07-09]. https://huggingface.co/blog/LoRA.
[19]	丁鑫, 邹荣金, 潘志庚. 基于高效参数微调的生成式大模型领域适配技术[J]. 人工智能, 2023(4): 1-9.
[20]	RAFI M N, MUAAZ M. Performance Evaluation of the LoRa Protocol in the context of Smart Meter[EB/OL].(2019-07-04)[2019-07-04]. https://arxiv.org/abs/1907.12355.
[21]	SUN Y, WANG S, Li Y, et al. Ernie: Enhanced representation through knowledge integration[EB/OL].(2019-04-09)[2023-07-09]. https://arxiv.org/abs/1904.09223.
[22]	SUN Y, WANG S, Li Y, et al. Ernie 2.0: A continual pre-training framework for language understanding[C]// Proceedings of the AAAI conference on artificial intelligence, vol. 34, No. 05. Palo Alto:AAAI Press, 2020: 8968-8975.
[23]	SUN Y, WANG S, Li Y, et al. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation[EB/OL].(2021-07-02)[2023-07-09]. https://arxiv.org/abs/2107.02137.
[24]	CHEN Z Y, XIE F K, WAN M, et al. MatChat: A large language model and application service platform for materials science[J]. Chinese Physics B, 2023, 32(11):208-213.
[25]	AMINABADI R Y, RAJBHANDARI S, ZHANG M, et al. DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale[EB/OL].(2022-07-00)[2023-07-09]. https://arxiv.org/abs/2207.00032.
[26]	RAJBHANDARI S, RASLEY J, RUWASE O, et al. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models[EB/OL].(2019-10-02)[2023-07-09]. https://arxiv.org/abs/1910.02054.
[27]	SMITH S, PATWARY M, NORICK B, et al. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model[EB/OL].(2022-01-19)[2023-07-09]. https://arxiv.org/abs/2201.11990.
[28]	RAJBHANDARI S, Li C, YAO Z, et al. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale[EB/OL].(2022-01-05)[2023-07-09]. https://arxiv.org/abs/2201.05596.
[29]	KISHORE P, SALIM R, TODD W, et al. Bleu:a Method for Automatic Evaluation of Machine Translation[C]// Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, 2002: 311-318.

微调技术	优势	不足
AdapterFusion	简单易用，训练过程稳定	参数开销较高，需要更多计算资源和存储空间
Prefix-Tuning	训练速度快，计算开销小	表达能力有限，且模型需要大量的训练数据来调整前缀
P-Tuning	能够有效地优化模型参数，从而提高模型的准确率和鲁棒性	需要较大的计算资源和较长的时间来进行参数优化
LoRA	计算和存储开销较小，由良好的兼容性	需要对超参数进行更复杂的调试

超参数名称	参数取值	优化结果
learning_rate(lr)	8e-7,8e-6,8e-5, 4e-5,1e-4	8e-5
warmup_ratio	0.005,0.01,0.05,0.1	0.005
gradient_accumulation_steps	1,2,4	1
num_train_epochs(epoch)	2,4	4

Llama2-70b模型的微调技术及其在材料领域的应用研究

A Study of the Fine-Tuning Technique of the Llama2-70b Model and Its Application in the Field of Materials

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 29

相关文章 0

编辑推荐

Metrics

本文评价