Research on the Generation and Evaluation of Judicial Text Summarization Based on Large Language Models

doi:10.11871/jfdc.issn.2096-742X.2024.06.007

Abstract

Abstract:

[Objective] With the development of science and technology in modern society, text summarization technology is widely used in various aspects of life and plays a significant role. However, in the judicial field, the use of traditional deep learning models to generate judicial text summaries is facing issues such as redundancy, inconsistent information, and insufficient understanding of text semantics. Moreover, the existing quality evaluation method for generated summaries, ROUGE, is relatively simplistic, focusing only on the overlap between the generated summary and the reference summary without considering deep semantic information. There is a need to construct new multi-dimensional indicators to evaluate the quality of generated summaries. [Methods] This article constructs a vertical large language model in the judicial field with the help of knowledge editing and parameter fine-tuning techniques to generate judicial text summaries, addressing the problem of traditional model’s insufficient understanding of professional judicial texts. It also proposes two evaluation indicators: information missing factor and information density. The large language model is used to extract knowledge from the summaries, measure text knowledge, and calculate the two types of indicators based on the measurement results to further evaluate the quality of the generated summaries at the intrinsic semantic level. [Conclusions] Experiments have proven that the method proposed in this article, which generates judicial text summaries based on a vertical domain large language model, helps to address issues of information redundancy and insufficient understanding of professional texts. Additionally, the two proposed summary evaluation indicators complement the evaluation of summary generation quality from the aspects of information consistency and redundancy, enriching the methods of evaluating summaries.

Key words: large language models, judicial text summary generation, text summary quality evaluation, domain fine-tuning techniques, knowledge extraction

PEI Bingsen,LI Xin,JIANG Zhangtao,LIU Mingshuai. Research on the Generation and Evaluation of Judicial Text Summarization Based on Large Language Models[J]. Frontiers of Data and Computing, 2024, 6(6): 62-73, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.06.007.

Figures/Tables 7

Fig.1

Fig.2

Table 1

Table 2

Table 3

Experimental results in the summary test set"

模型名称	ROUGE-1	ROUGE-2	ROUGE-L	$η I D F$ （信息缺失因子）	$ρ I D$ （信息密度）
CopyNet	36.2	31.4	31.5	0.42	0.47
LDA	37.4	32.0	30.7	0.40	0.52
Pointer-Generator-Trm	40.1	34.8	32.2	0.37	0.56
Chinese Llama（原始）	40.4	33.9	33.5	0.31	0.56
Chinese Llama（仅LoRA）	42.1	35.0	36.2	0.29	0.61
Chinese Llama（仅KME）	42.6	34.6	35.9	0.20	0.73
Chinese Llama（LoRA+KME）	45.5	36.1	37.3	0.18	0.78

Table 3

Fig.3

Fig.4

References 30

[1]	王骞, 雷景生, 唐小岚. 融合多层注意力表示的中文新闻文本摘要生成[J]. 计算机应用与软件, 2023, 40 (10): 191-198.
[2]	张琪, 范永胜, 金独亮. 基于MMR和WordNet的新闻文本摘要生成研究[J]. 西南师范大学学报(自然科学版), 2023, 48 (5): 77-86.
[3]	史继筠, 张驰, 王禹桥, 等. 基于知识辅助的结构化医疗报告生成[J/OL]. 计算机科学, 1-13[2024-05-18]. http://kns.cnki.net/kcms/detail/50.1075.TP.20240321.1439.003.html.
[4]	赵冠博, 张勇入领域知识的跨境民族文化生成式摘要方法[J]. 南京大学学报(自然科学), 2023, 59 (4): 620-628.
[5]	赵伟, 王文娟, 任彦凝, 等. 基于改进Transformer的生成式文本摘要模型[J]. 重庆邮电大学学报(自然科学版), 2023, 35 (1): 185-192.
[6]	施国良, 周抒, 王云峰, 等. 基于改进多头注意力机制的专利文本摘要生成研究[J]. 数据分析与知识发现, 2023, 7 (6): 61-72.
[7]	WANG Q, LIU P, ZHU Z, et al. A text abstraction summary model based on BERT word embedding and reinforcement learning[J]. Applied Sciences, 2019, 9(21): 4701.
[8]	DING J, LI Y, NI H, et al. Generative text summary based on enhanced semantic attention and gain-benefit gate[J]. IEEE Access, 2020, 8: 92659-92668.
[9]	魏鑫炀, 秦永彬, 唐向红, 等. 融合法条的司法裁判文书摘要生成方法[J]. 计算机工程与设计, 2023, 44 (9): 2844-2850.
[10]	祁鹏年, 廖雨伦, 覃飙. 基于深度学习的中文命名实体识别研究综述[J]. 小型微型计算机系统, 2023, 44 (9): 1857-1868.
[11]	LI J, FEI H, LIU J, et al. Unified named entity recognition as word-word relation classification[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(10): 10965-10973.
[12]	QI P, QIN B. SSMI: semantic similarity and mutual information maximization based enhancement for Chinese NER[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(11): 13474-13482.
[13]	ZHAO X, GREENBERG J, AN Y, et al. Fine-tuning BERT model for materials named entity recognition[C]// 2021 IEEE International Conference on Big Data (Big Data), IEEE, 2021: 3717-3720.
[14]	刘蓓, 许卓明, 陶皖, 等. 少样本关系抽取研究综述[J]. 计算机工程与应用, 2023, 59 (15): 27-37.
[15]	NAYAK T, NG H T. Effective modeling of encoder-decoder architecture for joint entity and relation extraction[C]// Proceedings of the AAAI conference on artificial intelligence, 2020, 34(05): 8528-8535.
[16]	XUE F, SUN A, ZHANG H, et al. Gdpnet: Refining latent multi-view graph for relation extraction[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(16): 14194-14202.
[17]	卢经纬, 郭超, 戴星原, 等. 问答ChatGPT之后: 超大预训练模型的机遇和挑战[J]. 自动化学报, 2023, 49(4):705-717.
[18]	杨波, 孙晓虎, 党佳怡, 等. 面向医疗问答系统的大语言模型命名实体识别方法[J]. 计算机科学与探索, 2023, 17 (10): 2389-2402.
[19]	王昀, 胡珉, 塔娜, 等. 大语言模型及其在政务领域的应用[J/OL]. 清华大学学报(自然科学版), 1-10 [2023-11-26] https://doi.org/10.16511/j.cnki.qhdxxb.2023.26.042.
[20]	徐月梅, 胡玲, 赵佳艺, 等. 大语言模型的技术应用前景与风险挑战[J]. 计算机应用, 2024, 44(6):1655-1662.
[21]	秦菲. 基于深度学习的自动生成文本摘要及其评估的研究[D]. 桂林: 桂林电子科技大学, 2020.
[22]	LIN C Y. Rouge: A package for automatic evaluation of summaries[C]// Text summarization branches out, 2004: 74-81.
[23]	MENG K, SHARMA A S, ANDONION A, et al. Mass-editing memory in a transformer[J]. arXiv preprint arXiv:2210.07229, 2022.
[24]	SEE A, LIU P J, MANNING C D. Get to the point: Summarization with pointer-generator networks[J]. arXiv preprint arXiv:1704.04368, 2017.
[25]	GU J, LU Z, LI H, et al. Incorporating copying mechanism in sequence-to-sequence learning[J]. arXiv preprint arXiv:1603.06393, 2016.
[26]	BLEI D M, NG A Y, JORDAN M I. Latent dirichlet allocation[J]. Journal of machine Learning research, 2003, 3(1): 993-1022.
[27]	SOUZA F, NOGUEIRA R, LOTUFO R. Portuguese named entity recognition using BERT-CRF[J]. arXiv preprint arXiv:1909.10649, 2019.
[28]	WU S, SONG X, FENG Z. MECT: Multi-metadata embedding based cross-transformer for Chinese na- med entity recognition[J]. arXiv preprint arXiv: 2107.05418, 2021.
[29]	SNELL J, SWERSKY K, ZEMEL R. Prototypical networks for few-shot learning[J]. Advances in neural information processing systems, 2017, 30:4077-4087.
[30]	HE K, HUANG Y, MAO R, et al. Virtual prompt pre-training for prototype-based few-shot relation extraction[J]. Expert Systems with Applications, 2023, 213: 118927.

混淆矩阵		实际表现
混淆矩阵		1	0
预测表现	1	TP	FP
预测表现	0	FN	TN

抽取类型	模型名称	P	R	F1
实体抽取	Bi-LSTM-CRF	82.5	80.3	81.4
	MECT	85.7	82.9	84.3
	MOSS（原始）	86.1	82.1	84.1
	ChatGLM（原始）	87.3	83.3	85.3
	Chinese Llama（原始）	88.6	84.0	86.2
	Chinese Llama（仅KME）	90.2	87.2	88.7
	Chinese Llama（仅LoRA）	89.6	87.4	88.5
	Chinese Llama（KME+LoRA）	91.0	88.2	89.6
关系抽取	Proto-CNN	71.3	68.4	69.8
	VPP	79.4	76.1	77.7
	MOSS（原始）	81.7	78.2	79.9
	ChatGLM（原始）	81.6	77.4	79.4
	Chinese Llama（原始）	82.5	80.2	81.3
	Chinese Llama（仅KME）	84.3	82.9	83.6
	Chinese Llama（仅LoRA）	83.6	82.3	82.9
	Chinese Llama（KEM+LoRA）	85.3	84.0	84.6