基于大语言模型的司法文本摘要生成与评价技术研究

doi:10.11871/jfdc.issn.2096-742X.2024.06.007

数据与计算发展前沿 ›› 2024, Vol. 6 ›› Issue (6): 62-73.

CSTR: 32002.14.jfdc.CN10-1649/TP.2024.06.007

doi: 10.11871/jfdc.issn.2096-742X.2024.06.007

基于大语言模型的司法文本摘要生成与评价技术研究

裴炳森(),李欣^*(),蒋章涛,刘明帅

中国人民公安大学, 信息网络安全学院, 北京 100038

收稿日期:2024-05-24 出版日期:2024-12-20 发布日期:2024-12-20
通讯作者: 李欣
作者简介:裴炳森, 中国人民公安大学, 硕士研究生, CCF学生会员, 主要研究方向为自然语言处理。
本文中主要负责论文中模型的实现与文章撰写。
PEI Bingsen, is a Master’s student at the People’s Public Security University of China. He is a CCF student member and his main research direction is natural language processing.
In this paper, he is responsible for the implementation of the model in this paper and the writing of the article.
E-mail: 597637809@qq.com|李欣, 中国人民公安大学信息网络安全学院, 院长, 博士, 教授, CCF会员, 主要研究方向为人工智能、网络安全等。
本文中主要负责文章思路的确定与论文写作的指导。
LI Xin, Ph.D., is a full professor and the dean of the School of Information and Network Security at the People’s Public Security University of China, CCF member. His main research directions include artificial intelligence and network security.
In this paper, he is primarily responsible for defining the conceptual framework and guiding the writing of the paper.
E-mail: lixin@ppsuc.edu.cn
基金资助:
国家重点研发计划-社会治理与智慧社会-子课题“视频监控网络风险识别”(2022YFC3301101-1)

Research on the Generation and Evaluation of Judicial Text Summarization Based on Large Language Models

PEI Bingsen(),LI Xin^*(),JIANG Zhangtao,LIU Mingshuai

Information Network Security Academy, People’s Public Security University of China, Beijing 100038, China

Received:2024-05-24 Online:2024-12-20 Published:2024-12-20
Contact: LI Xin

摘要/Abstract

摘要：

【目的】随着当前社会科学技术的发展, 文本摘要技术被广泛应用在生活中的各个方面, 并发挥着重要作用。然而在司法领域中, 使用传统深度学习模型生成司法文本摘要存在冗余、信息不一致、对文本语义理解不够等问题；而且现有生成摘要的质量评价方法ROUGE较为单一, 仅关注生成摘要与参考摘要之间的重叠, 不关注深层语义信息, 需要构建新的指标多维评价生成摘要质量。【方法】本文借助知识编辑、参数微调技术构建司法领域垂直大语言模型, 生成司法文本摘要, 改善传统模型对专业司法文本理解不够的问题；并提出信息缺失因子、信息密度两种评价指标, 用大语言模型对摘要实现知识抽取, 度量文本知识, 根据度量结果计算两类指标, 进一步在内在语义层次衡量生成摘要的质量。【结论】通过实验证明, 本文提出的基于垂直领域大语言模型生成司法文本摘要的方法有助于改善信息冗余、对专业文本理解能力不够的问题, 且提出的两类摘要评价指标从信息一致性、冗余度两方面补充评价了摘要生成质量, 丰富了评价摘要的方法。

关键词: 大语言模型, 司法文本摘要生成, 摘要质量评价, 领域调优技术, 知识抽取

Abstract:

[Objective] With the development of science and technology in modern society, text summarization technology is widely used in various aspects of life and plays a significant role. However, in the judicial field, the use of traditional deep learning models to generate judicial text summaries is facing issues such as redundancy, inconsistent information, and insufficient understanding of text semantics. Moreover, the existing quality evaluation method for generated summaries, ROUGE, is relatively simplistic, focusing only on the overlap between the generated summary and the reference summary without considering deep semantic information. There is a need to construct new multi-dimensional indicators to evaluate the quality of generated summaries. [Methods] This article constructs a vertical large language model in the judicial field with the help of knowledge editing and parameter fine-tuning techniques to generate judicial text summaries, addressing the problem of traditional model’s insufficient understanding of professional judicial texts. It also proposes two evaluation indicators: information missing factor and information density. The large language model is used to extract knowledge from the summaries, measure text knowledge, and calculate the two types of indicators based on the measurement results to further evaluate the quality of the generated summaries at the intrinsic semantic level. [Conclusions] Experiments have proven that the method proposed in this article, which generates judicial text summaries based on a vertical domain large language model, helps to address issues of information redundancy and insufficient understanding of professional texts. Additionally, the two proposed summary evaluation indicators complement the evaluation of summary generation quality from the aspects of information consistency and redundancy, enriching the methods of evaluating summaries.

Key words: large language models, judicial text summary generation, text summary quality evaluation, domain fine-tuning techniques, knowledge extraction

裴炳森,李欣,蒋章涛,刘明帅. 基于大语言模型的司法文本摘要生成与评价技术研究[J]. 数据与计算发展前沿, 2024, 6(6): 62-73.

PEI Bingsen,LI Xin,JIANG Zhangtao,LIU Mingshuai. Research on the Generation and Evaluation of Judicial Text Summarization Based on Large Language Models[J]. Frontiers of Data and Computing, 2024, 6(6): 62-73, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.06.007.

图/表 7

图1

图2

表1

表2

表3

在摘要测试集中的实验结果"

模型名称	ROUGE-1	ROUGE-2	ROUGE-L	$η I D F$ （信息缺失因子）	$ρ I D$ （信息密度）
CopyNet	36.2	31.4	31.5	0.42	0.47
LDA	37.4	32.0	30.7	0.40	0.52
Pointer-Generator-Trm	40.1	34.8	32.2	0.37	0.56
Chinese Llama（原始）	40.4	33.9	33.5	0.31	0.56
Chinese Llama（仅LoRA）	42.1	35.0	36.2	0.29	0.61
Chinese Llama（仅KME）	42.6	34.6	35.9	0.20	0.73
Chinese Llama（LoRA+KME）	45.5	36.1	37.3	0.18	0.78

表3

图3

图4

参考文献 30

[1]	王骞, 雷景生, 唐小岚. 融合多层注意力表示的中文新闻文本摘要生成[J]. 计算机应用与软件, 2023, 40 (10): 191-198.
[2]	张琪, 范永胜, 金独亮. 基于MMR和WordNet的新闻文本摘要生成研究[J]. 西南师范大学学报(自然科学版), 2023, 48 (5): 77-86.
[3]	史继筠, 张驰, 王禹桥, 等. 基于知识辅助的结构化医疗报告生成[J/OL]. 计算机科学, 1-13[2024-05-18]. http://kns.cnki.net/kcms/detail/50.1075.TP.20240321.1439.003.html.
[4]	赵冠博, 张勇入领域知识的跨境民族文化生成式摘要方法[J]. 南京大学学报(自然科学), 2023, 59 (4): 620-628.
[5]	赵伟, 王文娟, 任彦凝, 等. 基于改进Transformer的生成式文本摘要模型[J]. 重庆邮电大学学报(自然科学版), 2023, 35 (1): 185-192.
[6]	施国良, 周抒, 王云峰, 等. 基于改进多头注意力机制的专利文本摘要生成研究[J]. 数据分析与知识发现, 2023, 7 (6): 61-72.
[7]	WANG Q, LIU P, ZHU Z, et al. A text abstraction summary model based on BERT word embedding and reinforcement learning[J]. Applied Sciences, 2019, 9(21): 4701.
[8]	DING J, LI Y, NI H, et al. Generative text summary based on enhanced semantic attention and gain-benefit gate[J]. IEEE Access, 2020, 8: 92659-92668.
[9]	魏鑫炀, 秦永彬, 唐向红, 等. 融合法条的司法裁判文书摘要生成方法[J]. 计算机工程与设计, 2023, 44 (9): 2844-2850.
[10]	祁鹏年, 廖雨伦, 覃飙. 基于深度学习的中文命名实体识别研究综述[J]. 小型微型计算机系统, 2023, 44 (9): 1857-1868.
[11]	LI J, FEI H, LIU J, et al. Unified named entity recognition as word-word relation classification[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(10): 10965-10973.
[12]	QI P, QIN B. SSMI: semantic similarity and mutual information maximization based enhancement for Chinese NER[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(11): 13474-13482.
[13]	ZHAO X, GREENBERG J, AN Y, et al. Fine-tuning BERT model for materials named entity recognition[C]// 2021 IEEE International Conference on Big Data (Big Data), IEEE, 2021: 3717-3720.
[14]	刘蓓, 许卓明, 陶皖, 等. 少样本关系抽取研究综述[J]. 计算机工程与应用, 2023, 59 (15): 27-37.
[15]	NAYAK T, NG H T. Effective modeling of encoder-decoder architecture for joint entity and relation extraction[C]// Proceedings of the AAAI conference on artificial intelligence, 2020, 34(05): 8528-8535.
[16]	XUE F, SUN A, ZHANG H, et al. Gdpnet: Refining latent multi-view graph for relation extraction[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(16): 14194-14202.
[17]	卢经纬, 郭超, 戴星原, 等. 问答ChatGPT之后: 超大预训练模型的机遇和挑战[J]. 自动化学报, 2023, 49(4):705-717.
[18]	杨波, 孙晓虎, 党佳怡, 等. 面向医疗问答系统的大语言模型命名实体识别方法[J]. 计算机科学与探索, 2023, 17 (10): 2389-2402.
[19]	王昀, 胡珉, 塔娜, 等. 大语言模型及其在政务领域的应用[J/OL]. 清华大学学报(自然科学版), 1-10 [2023-11-26] https://doi.org/10.16511/j.cnki.qhdxxb.2023.26.042.
[20]	徐月梅, 胡玲, 赵佳艺, 等. 大语言模型的技术应用前景与风险挑战[J]. 计算机应用, 2024, 44(6):1655-1662.
[21]	秦菲. 基于深度学习的自动生成文本摘要及其评估的研究[D]. 桂林: 桂林电子科技大学, 2020.
[22]	LIN C Y. Rouge: A package for automatic evaluation of summaries[C]// Text summarization branches out, 2004: 74-81.
[23]	MENG K, SHARMA A S, ANDONION A, et al. Mass-editing memory in a transformer[J]. arXiv preprint arXiv:2210.07229, 2022.
[24]	SEE A, LIU P J, MANNING C D. Get to the point: Summarization with pointer-generator networks[J]. arXiv preprint arXiv:1704.04368, 2017.
[25]	GU J, LU Z, LI H, et al. Incorporating copying mechanism in sequence-to-sequence learning[J]. arXiv preprint arXiv:1603.06393, 2016.
[26]	BLEI D M, NG A Y, JORDAN M I. Latent dirichlet allocation[J]. Journal of machine Learning research, 2003, 3(1): 993-1022.
[27]	SOUZA F, NOGUEIRA R, LOTUFO R. Portuguese named entity recognition using BERT-CRF[J]. arXiv preprint arXiv:1909.10649, 2019.
[28]	WU S, SONG X, FENG Z. MECT: Multi-metadata embedding based cross-transformer for Chinese na- med entity recognition[J]. arXiv preprint arXiv: 2107.05418, 2021.
[29]	SNELL J, SWERSKY K, ZEMEL R. Prototypical networks for few-shot learning[J]. Advances in neural information processing systems, 2017, 30:4077-4087.
[30]	HE K, HUANG Y, MAO R, et al. Virtual prompt pre-training for prototype-based few-shot relation extraction[J]. Expert Systems with Applications, 2023, 213: 118927.

混淆矩阵		实际表现
混淆矩阵		1	0
预测表现	1	TP	FP
预测表现	0	FN	TN

抽取类型	模型名称	P	R	F1
实体抽取	Bi-LSTM-CRF	82.5	80.3	81.4
	MECT	85.7	82.9	84.3
	MOSS（原始）	86.1	82.1	84.1
	ChatGLM（原始）	87.3	83.3	85.3
	Chinese Llama（原始）	88.6	84.0	86.2
	Chinese Llama（仅KME）	90.2	87.2	88.7
	Chinese Llama（仅LoRA）	89.6	87.4	88.5
	Chinese Llama（KME+LoRA）	91.0	88.2	89.6
关系抽取	Proto-CNN	71.3	68.4	69.8
	VPP	79.4	76.1	77.7
	MOSS（原始）	81.7	78.2	79.9
	ChatGLM（原始）	81.6	77.4	79.4
	Chinese Llama（原始）	82.5	80.2	81.3
	Chinese Llama（仅KME）	84.3	82.9	83.6
	Chinese Llama（仅LoRA）	83.6	82.3	82.9
	Chinese Llama（KEM+LoRA）	85.3	84.0	84.6

基于大语言模型的司法文本摘要生成与评价技术研究

Research on the Generation and Evaluation of Judicial Text Summarization Based on Large Language Models

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 30

相关文章 2

编辑推荐

Metrics

本文评价

[1]	赵小丹, 胡林. 基于深度学习的农业科技政策知识抽取方法研究[J]. 数据与计算发展前沿, 2024, 6(4): 106-115.
[2]	韦一金, 樊景超. 基于ChatGLM2-6B的农业政策问答系统[J]. 数据与计算发展前沿, 2024, 6(4): 116-127.