An Agricultural Policy Question Answering System Based on ChatGLM2-6B

doi:10.11871/jfdc.issn.2096-742X.2024.04.010

Abstract

Abstract:

[Objective] In order to improve the transparency of the policy, reduce the information asymmetry, and provide a convenient way for stakeholders to obtain agricultural policy information and guidance, this paper constructs an agricultural policy question answering system based on ChatGLM2-6B and Langchain-Chatchat. [Methods] To construct the agricultural policy question answering dataset, this paper first leverages web crawlers to obtain the full text of guiding agricultural policies of the National Rural Revitalization Administration and Central File No. 1, as well as the full text of agricultural policies of Huanghe-Nine provincial Rural Revitalization Bureau. The collected dataset is then used to fine-tune the ChatGLM2-6B model by QLoRA and conduct model consolidation and quantification. The obtained ChatGLM2-6B-QLoRA-int4 model is further combined with Langchain-Chatchat and local agricultural policy knowledge base to construct an agricultural policy question answering system. [Results] Questions were asked to ChatGPT, ChatGLM2-6B, ChatGLM2-6B-QLORa, and our question-and-answer system, respectively, and the answer results were evaluated by expert scoring method. Our system is better than ChatGLM2-6B and CHATGLM2-6B-QLORA in the field of agricultural policy, and the overall effect is better than ChatGPT. [Conclusion] The Q&A system constructed in this research performs well in the field of agricultural policy, and can ensure the security of proprietary data and realize the local deployment of LLM-based Q&A system.

Key words: large language model (LLM), agriculture, policy, question answering system, vertical domain

WEI Yijin, FAN Jingchao. An Agricultural Policy Question Answering System Based on ChatGLM2-6B[J]. Frontiers of Data and Computing, 2024, 6(4): 116-127, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.04.010.

Figures/Tables 15

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Table 2

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Table3

References 17

[1]	白皓然, 孙伟浩, 金宁, 等. 基于改进Bi-LSTM-CRF的农业问答系统研究[J]. 中国农机化学报, 2023, 44(2): 99-105.
[2]	王东升, 王卫民, 王石, 等. 面向限定领域问答系统的自然语言理解方法综述[J]. 计算机科学, 2017, 44(8): 1-8. doi: 10.11896/j.issn.1002-137X.2017.08.001
[3]	GUU K, LEE K, TUNG Z, et al. Retrieval augmented language model pre-training[C]// Proceedings of the 37th International Conference on Machine Learning, Vienna. New York: PMLR, 2020: 3929-3938.
[4]	CHOWDHERY A, NARANG S, DEVLIN J, et al. PaLM: scaling language modeling with pathways[J/OL]. arXiv:2204.02311, 2022. [2023.11.20]https://doi.org/10.48550/arXiv.2204.02311.
[5]	WEI J, TAY Y, BOMMASANI R, et al. Emergent abilities of large language models[J/OL]. arXiv:2206. 07682, 2022. [2023-11-20] https://doi.org/10.48550/arXiv.2206.07682.
[6]	RADFORD A, NARASIMHAN K, SALIMANS T, et al. Improving language understanding by generative pre-training[J/OL]. Preprint, 2018. [2023-11-20] https://api.semanticscholar.org/CorpusID:49313245.
[7]	RADFORD A, WU J, CHILD R, et al. Language models are unsupervised multitask learners[J]. OpenAI Blog, 2019, 1(8): 9.
[8]	BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[J]. Advances in neural information processing systems, 2020, 33: 1877-1901.
[9]	DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J/OL]. arXiv preprint arXiv:1810.04805, 2018. [2023-11-20] https://doi.org/10.48550/arXiv.1810.04805.
[10]	SONG K, TAN X, QIN T, et al. Mass: Masked sequence to sequence pre-training for language generation[J/OL]. arXiv preprint arXiv:1905.02450, 2019. [2023-11-20] https://doi.org/10.48550/arXiv.1905.02450.
[11]	LEWIS M, LIU Y, GOYAL N, et al. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[J/OL]. arXiv preprint arXiv:1910.13461, 2019. [2023-11-20] https://doi.org/10.48550/arXiv.1910.13461.
[12]	BI B, LI C, WU C, et al. Palm: Pre-training an autoencoding&autoregressive language model for context-conditioned generation[J/OL]. arXiv preprint arXiv:2004.07159, 2020. [2023-11-20] https://doi.org/10.48550/arXiv.2004.07159.
[13]	MAYNEZ J, NARAYAN S, BOHNET B, et al. On faithfulness and factuality in abstractive summarization[J/OL]. arXiv:2005.00661, 2020. [2023-11-20] https://doi.org/10.48550/arXiv.2005.00661.
[14]	HU E J, SHEN Y, WALLIS P, et al. Lora: Low-rank adaptation of large language models[J/OL]. arXiv preprint arXiv:2106.09685, 2021. [2023-11-20] https://doi.org/10.48550/arXiv.2106.09685.
[15]	DETTMERS T, PAGNONI A, HOLTZMAN A, et al. Qlora: Efficient finetuning of quantized llms[J/OL]. arXiv preprint arXiv:2305.14314, 2023. [2023-11-20] https://doi.org/10.48550/arXiv.2305.14314.
[16]	ZENG A, LIU X, DU Z, et al. Glm-130b: An open bilingual pre-trained model[J/OL]. arXiv preprint arXiv: 2210.02414, 2022. [2023-11-20] https://doi.org/10. 48550/arXiv.2210.02414.
[17]	DU Z, QIAN Y, LIU X, et al. Glm: General language model pretraining with autoregressive blank infilling[J/OL]. arXiv preprint arXiv:2103.10360, 2021. [2023-11-20] https://doi.org/10.48550/arXiv.2103.10360.

项目	具体配置
操作系统	Linux-5.15.120+-x86_64-with-glibc2.35
GPU	Tesla T4
编程语言	Python 3.10.12
LangChain-Chatchat版本	0.2.6
Fastchat版本	0.2.31
分词器	ChineseRecursiveTextSplitter
LLM模型	ChatGLM2-6B-QLoRA-int4
Embeddings模型	m3e-base@cuda

问题	回答		数据集数据
问题	QLoRA微调前	QLoRA微调后	数据集数据
到2025年，农业科技整体实力力争达到什么水平?	根据《农业科技发展中心“十四五”规划》，到2025年，农业科技整体实力力争达到国内农业科技实力排名前5位水平。这意味着，农业科技在农业领域的研究、应用和创新能力将得到显著提升，成为农业领域的重要技术支撑	到2025年，农业科技整体实力力争达到世界第一方阵，生物育种、农业数字化、动物疫苗等领域居世界领先水平	到2025年，农业科技整体实力力争达到世界第一方阵，生物育种、农业5G应用、动物疫苗等领域居世界领先水平