NKCorpus：利用海量网络数据构建大型高质量中文数据集

doi:10.11871/jfdc.issn.2096-742X.2022.03.003

数据与计算发展前沿 ›› 2022, Vol. 4 ›› Issue (3): 30-45.

CSTR: 32002.14.jfdc.CN10-1649/TP.2022.03.003

doi: 10.11871/jfdc.issn.2096-742X.2022.03.003

• 专刊:先进智能计算平台及应用(下) • 上一篇下一篇

NKCorpus：利用海量网络数据构建大型高质量中文数据集

李东闻(),钟震宇(),申峻宇(),王昊天(),孙羽菲(),张玉志()

南开大学,软件学院,天津 300350

收稿日期:2022-02-18 出版日期:2022-06-20 发布日期:2022-06-20
通讯作者: 孙羽菲
作者简介:李东闻,南开大学,博士研究生,主要研究领域为自然语言处理及深度学习。
本文中负责项目整体开发及论文撰写。
LI Dongwen is currently a doctoral stud-ent in the College of Software at Nankai University, Tianjin, China. Her current research interests include anomaly detection, deep learning and natural language processing.
In this paper, she is mainly responsible for the overall project development and paper writing.
E-mail: lidongwen@mail.nankai.edu.cn|钟震宇,南开大学,硕士研究生,主要研究领域为自然语言处理、高性能计算、智能运维。
本文主要负责项目整体开发及论文撰写。
ZHONG Zhenyu is currently a master’s student in the College of Software at Nankai University, Tianjin, China. His current research interests include natural language processing, high performance computing and AIOps.
In this paper, he is mainly responsible for the overall project development and paper writing.
E-mail: zyzhong@mail.nankai.edu.cn|申峻宇,南开大学,本科生,主要研究领域为自然语言处理。
本文主要参与论文撰写及数据去重环节的项目开发。
SHEN Junyu is currently an undergra-duate in the College of Software at Nankai University, Tianjin, China. His current research interest is natural language processing.
In this paper, he is mainly involved in the project development of data deduplication and paper writing.
E-mail: junyu_nk@163.com|王昊天,南开大学,博士研究生,主要研究领域为自然语言处理。
本文主要参与数据去重和高质量数据筛选环节的项目开发。
WANG Haotian is currently a doctoral student in the College of Software at Nankai University, Tianjin, China. His current research interest is natural lang-uage processing.
In this paper, he is mainly involved in the project development of data deduplication and high-quality data filtering.
E-mail: 1392300702@qq.com|孙羽菲,南开大学,软件学院,特聘研究员,博士,主要研究方向为深度学习、异构计算、人工智能等。
本文主要负责项目指导及总体统稿。
SUN Yufei, Ph.D., is a professor at the College of Software, Nankai University. Her research interests include deep learn-ing, heterogeneous computing, artifi-cial intelligence, etc.
In this paper, she is mainly responsible for project guidance and the final compilation, edition of the paper.
E-mail: yufei_sun@sina.com|张玉志,南开大学,软件学院,院长,讲席教授,博士,主要研究方向为深度学习及其他人工智能相关领域。
本文主要负责文献调研及项目指导。
ZHANG Yuzhi, Ph.D., is currently a dis-tinguished professor and the dean of the College of Software at Nankai Univer-sity, Tianjin, China. His research interests include deep learning and other aspects in artificial intelligence.
In this paper, he is mainly responsible for project guidance and the related work investigation.
E-mail: zyz@nankai.edu.cn

NKCorpus: Extracting High Quality Large Chinese Dataset from Web Data

LI Dongwen(),ZHONG Zhenyu(),SHEN Junyu(),WANG Haotian(),SUN Yufei(),ZHANG Yuzhi()

College of software, Nankai University, Tianjin 300350, China

Received:2022-02-18 Online:2022-06-20 Published:2022-06-20
Contact: SUN Yufei

摘要/Abstract

摘要：

【目的】大规模、高质量的中文数据集对于大型中文预训练语言模型及其他自然语言处理模型的训练至关重要,因此需要设计并完善一种可以构建大规模中文数据集的框架。【方法】利用语言提取、文本清洗、数据去重等多种方法对原始数据进行处理获取数据集,并利用并行技术对数据处理框架的效率进行优化。【结果】提出了一个流程完善且高效的可以利用海量网络数据构建大型高质量中文数据集的框架NKCorpus,并且利用NKCorpus构建了约700GB的可直接用于中文预训练语言模型的训练工作的高质量中文数据集。【结论】NKCorpus已能够基本满足当前对于大规模、高质量中文数据集的高效构建需求。

关键词: 自然语言处理, 中文数据集, 数据集构建

Abstract:

[Objective] For large-scale Chinese pre-trained language models or other natural language processing models, it is very important to collect and process large-scale high-quality Chinese data for model training. Therefore, a comprehensive large-scale dataset construction framework is required. [Methods] We use pipeline preprocessing procedures such as language extraction, text cleaning, and deduplication to process data. The performance of our framework is also optimized by parallel computing techniques. [Results] A comprehensive and efficient dataset construction framework NKCorpus is proposed to construct large-scale high-quality Chinese corpus datasets from massive web data, and a high-quality Chinese dataset of about 700GB is constructed using NKCorpus. [Conclusions] NKCorpus can meet the current needs for the efficient construction of large-scale, high-quality Chinese datasets.

Key words: natural language processing, Chinese dataset, dataset construction

李东闻,钟震宇,申峻宇,王昊天,孙羽菲,张玉志. NKCorpus：利用海量网络数据构建大型高质量中文数据集[J]. 数据与计算发展前沿, 2022, 4(3): 30-45.

LI Dongwen,ZHONG Zhenyu,SHEN Junyu,WANG Haotian,SUN Yufei,ZHANG Yuzhi. NKCorpus: Extracting High Quality Large Chinese Dataset from Web Data[J]. Frontiers of Data and Computing, 2022, 4(3): 30-45, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2022.03.003.

图/表 17

表1

大型预训练语言模型及对应训练数据规模"

模型	参数量	训练数据量
GPT-2	15亿	40GB
CPM	26亿	100GB
GPT-3	1750亿	1TB
盘古 $α$	2000亿	500GB
源1.0	2457亿	5TB
悟道2.0	17500亿	4.9TB

表1

表2

表3

图1

表4

表5

表6

表7

图2

图3

图4

图5

图6

表8

表9

表10

表11

参考文献 28

[1]	BROWN T B, MANN B, RYDER N, et al. Language Mo-dels areFew-Shot Learners[C/OL]. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, Neur-IPS 2020. https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.
[2]	Yang Z, Dai Z, Yang Y, et al. XLNet: generalized autor-egressive pretraining for language understanding[C]// Proceedings of the 33rd International Conference on Ne-ural Information Processing Systems, 2019: 5753-5763.
[3]	Raffel C, Shazeer N, Roberts A, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer[J]. J. Mach. Learn. Res, 2019, 21(140): 1-67.
[4]	Wenzek G, Lachaux M A, Conneau A, et al. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data[C]// Proceedings of the 12th Language Res-ources and Evaluation Conference, 2020: 4003-4012.
[5]	Zhu Y, Kiros R, Zemel R, et al. Aligning books and movies: Towards story-like visual explanations by watch-ing movies and reading books[C]// Proceedings of the IEEE international conference on computer vision, 2015: 19-27.
[6]	Radford A, Wu J, Child R, et al. Language models are unsupervised multitask learners[J]. OpenAI blog, 2019, 1(8): 9.
[7]	Xiao C, Zhong H, Guo Z, et al. Cail2018: A large-scale legal dataset for judgment prediction[J]. arXiv preprint arXiv:1807.02478, 2018.
[8]	Chen S, Ju Z, Dong X, et al. MedDialog: a large-scale medical dialogue dataset[C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020: 9241-9250.
[9]	Xu L, Zhang X, Dong Q. CLUECorpus2020: A large-scale Chinese corpus for pre-training language model[J]. arXiv preprint arXiv:2003.01355, 2020.
[10]	Grave É, Bojanowski P, Gupta P, et al. Learning Word Vectors for 157 Languages[C/OL]// Proceedings of the Eleventh International Conference on Language Reso-urces and Evaluation (LREC 2018), 2018, https://aclan-thology.org/L18-1550.pdf.
[11]	Qiu X, Sun T, Xu Y, et al. Pre-trained models for natural language processing: A survey[J]. Science China Techno-logical Sciences, 2020, 63(10): 1872-1897.
[12]	Sun M, Li J, Guo Z, et al. Thuctc: an efficient chinese text classifier[J/OL]. GitHub Repository, 2016, https://github.com/thunlp/THUCTC.
[13]	Yuan S, Zhao H, Du Z, et al. Wudaocorpora: A super large-scale chinese corpora for pre-training language models[J]. AI Open, 2021, 2: 65-68. doi: 10.1016/j.aiopen.2021.06.001
[14]	Lin J, Men R, Yang A, et al. M6: A chinese multimodal pretrainer[J]. arXiv preprint arXiv:2103.00823, 2021.
[15]	Zeng W, Ren X, Su T, et al. PanGu-$\alpha $: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation[J]. arXiv preprint arXiv: 2104.12369, 2021.
[16]	Kudo T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018: 66-75.
[17]	Heafield K. KenLM: Faster and smaller language model queries[C]// Proceedings of the sixth workshop on statist-ical machine translation, 2011: 187-197.
[18]	Lauriola I, Lavelli A, Aiolli F. An introduction to deep learning in natural language processing: models, techni-ques, and tools[J]. Neurocomputing, 2022, 470: 443-456. doi: 10.1016/j.neucom.2021.05.103
[19]	Zhang Z, Han X, Zhou H, et al. CPM: A large-scale generative Chinese pre-trained language model[J]. AI Open, 2021, 2: 93-99. doi: 10.1016/j.aiopen.2021.07.001
[20]	Lui M, Baldwin T. langidpy: An off-the-shelf language identification tool[C]// Proceedings of the ACL 2012 syst-em demonstrations, 2012: 25-30.
[21]	Buck C, Heafield K, Van Ooyen B. N-gram counts and language models from the common crawl[C]// Proceed-ings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), 2014: 3579-3584.
[22]	汤佳杰, 曹永忠, 顾浩. 基于文本标点密度连续和的网页正文抽取[J]. 计算机时代, 2020, 1:69-72.
[23]	Rivest R, Dusse S. The MD5 message-digest algorithm[J]. RFC, 1992, 1321:1-21.
[24]	Jaccard P. The distribution of the flora in the alpine zone. 1[J]. New phytologist, 1912, 11(2): 37-50. doi: 10.1111/j.1469-8137.1912.tb05611.x
[25]	Gionis A, Indyk P, Motwani R. Similarity search in high dimensions via hashing[C]// Vldb. 1999, 99(6): 518-529.
[26]	Broder A Z. On the resemblance and containment of documents[C]// Proceedings. Compression and Comp-lexity of SEQUENCES 1997 (Cat. No. 97TB100171), IEEE, 1997: 21-29.
[27]	Kenton J D M W C, Toutanova L K. BERT: Pre-training of Deep Bidirectional Transformers for Language Under-standing[C]// Proceedings of NAACL-HLT, 2019: 4171-4186.
[28]	Li X, Meng Y, Sun X, et al. Is Word Segmentation Necessary for Deep Learning of Chinese Representat-ions?[C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019: 3242-3252.

年份	压缩后大小（TB）
2017	109.66
2018	101.02
2019	96.61
2020	79.19
2021	77.34
总计	463.82

时间	英文	俄文	德文	中文
9月	45.45%	6.53%	5.61%	4.99%
10月	45.40%	6.80%	5.68%	4.83%
11/12月	46.25%	5.99%	5.41%	5.30%

文本长度	阈值
1至70	80%
71至230	70%
大于230	60%

类别	数量
色情	1690
反动	792
暴恐	254
民生	734
贪腐	97
其他	2559

类别	条目	版本
开发语言	Python	3.7.0
数据库	MySQL	8.0.27
	MongoDB	4.4.9

NKCorpus：利用海量网络数据构建大型高质量中文数据集

NKCorpus: Extracting High Quality Large Chinese Dataset from Web Data

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 28

相关文章 1

编辑推荐

Metrics

本文评价

类别	型号	数量
CPU	16C32T Intel(R) Xeon(R) Gold 5218 CPU @2.30GHz	2
RAM	192G	1
离线存储磁盘	4TB	100

累计处理文件数	单次处理一个文件方案累计用时（s）	先局部再整体文件方案累计用时（s）
200	1685.2	1154.92
400	3674.55	2430.93
600	5866.27	3643.01
800	8497.65	4826.43
1000	11378.84	6020.76

处理模块	单机单进程处理数据量（TB/天）	单机多进程处理数据量（TB/天）/ 对应进程数	双机多进程处理数据量（TB/天）
数据下载	2.2	2.2/1	4.3
中文提取	0.4	2.4/6	4.8
文本过滤及清洗	0.5	3.2/6	6.5
数据去重	0.7	1.5/2	3.0
高质量数据筛选	0.2	0.6/3	1.2
文本格式处理	2	10.6/5	22

处理步骤	已获取数据量（TB）
数据下载	195
中文提取	19
文本过滤&清洗	6.6
数据去重	1.5
高质量数据筛选	0.7
文本格式处理	0.7

	NKCorpus构建出的数据集	CLUECorpus
小型GPT模型	15.47	68.76
KenLM （分字）模型	15844.36	11309.31
KenLM （分词）模型	118246.69	133829.06