A Survey on Short-text Generation Technology

doi:10.11871/jfdc.issn.2096-742X.2021.03.010

Abstract

Abstract:

[Context] Research on short-text generation is of great importance in improving the efficiency of reading and writing, the impact of communication and guidance, the satisfaction of intelligent human-computer interaction and the capacity for machine semantic comprehension. However, short-text generation technology faces many difficulties and challenges due to the weakness of generation technology and the increasing complexity of realistic implementation requirements.[Methods] The neural network based generation method is a key artificial intelligence technology that has accomplished several pioneering achievements in short-text summary, dialogue generation, text generation of comments, poetry creation, and other linguistic tasks. [Results] This paper presents the research status and development of neural network based short-text generation technology in the aspects of the generation model, goal, and evaluation metric, providing a guide for further research into short-text generation technology. [Conclusions] we summarized the difficulties, challenges and trends of neural network based short-text generation technology.

Key words: Natural Language Generation (NLG), neural network model, short-text, evaluation method

ZHANG Chenyang,DU Yihua. A Survey on Short-text Generation Technology[J]. Frontiers of Data and Computing, 2021, 3(3): 111-125.

Figures/Tables 6

Table 1

Fig.1

Fig.2

Fig.3

Fig.4

Table 2

The latest research literature on four types of generation requirements"

类别	文献作者	生成模型/方法	描述
语句连贯表达	Li等（2020）	Transformer	论文采用预训练模型和微调的方法提出了硬格式诗歌生成模型SongNet,结合模板方法能够生成流畅连贯的诗句^[61]。
	Zhang等（2020）	Transformer	论文提出了基于预训练和微调方式的天马模型（PEGASUS）,在文本摘要生成中取得了显著成效^[62]。
	Peng等（2020）	Transformer	论文针对少样本场景下的任务导向型对话,采用预训练的方法提高了生成回复的流畅度^[63]。
语句多样表达	Su等(2020)	Seq2Seq	论文提出了基于统计风格信息指导的强化学习策略,在保证生成质量的同时提升生成对话的多样性^[64]。
	Su等（2020）	Seq2Seq	论文将贴吧评论、俗语和书籍内容等非对话文本引入到对话生成中,用以提升对话生成的多样性^[65]。
	Duan等（2020）	Transformer	论文针对查询式广告生成任务,在给定关键词的前提下通过引入外部知识来生成多样性的广告文案^[66]。
语境关联表达	倪海清等（2020）	Seq2Seq	论文针对短文本摘要任务,将短文本的整体语义信息引入生成模型,以确保生成摘要的语义完整性和关联性^[67]。
	Wang等（2020）	Seq2seq	论文中引入了外部知识用于捕获对话间的关联,同时提出了回复指导注意力机制,引导模型生成一致性回复^[68]。
	Byeongchang Kim等（2020）	Transformer	论文针对基于知识的对话生成任务提出了序列知识转换模型,能在生成时选择更适合的知识以提升对话的语境关联^[69]。
个性化生成	Zheng等（2020）	Transformer	论文中基于预训练设计了个性化对话模型,通过用户角色和对话历史构建丰富的对话回复文本^[70]。
	Yang等（2020）	Seq2Seq	论文通过多任务学习和强化学习策略,设计了从输入句子中识别用户特征的作者分析模块用于生成个性化对话^[71]。
	Chen等（2020）	Seq2Seq	论文中为了提升广告邮件的关注度,通过软模板方法结合用户偏好和产品描述来获得个性化主题^[72]。

Table 2

References 87

[1]	Perera R, Nand P. Recent Advances in Natural Language Generation: A Survey and Classification of the Empi- rical Literature[J]. Computing and Informatics, 2017, 36(1):1-32. doi: 10.4149/cai_2017_1_1
[2]	ALSMADI I, GAN K H. Review of short-text classifica-tion[J]. International Journal of Web Information Sys-tems, 2019, 15(2):155-182.
[3]	Song G, Ye Y, Du X, et al. Short text classification: a survey[J]. Journal of Multimedia, 2014, 9(5):635-643.
[4]	Li Shushen. Data and Computing Drive Science and Te- chnology Innovation[J]. Frontiers of Data&Computing, 2019, 1(1):1.DOI: 10.11871/jfdc.issn.2096-742X.2019.01.001.PID:21.86101.2/jfdc.2096-742X.2019.01.001.
[5]	贾熹滨, 李让, 胡长建等. 智能对话系统研究综述[J]. 北京工业大学学报, 2017, 43(009):1344-1356.
[6]	Tai Y, He H, Zhang W Z, et al. Automatic Generation of Review Content in Specific Domain of Social Network Based on RNN[C]. 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC).IEEE, 2018: 601-608.
[7]	CCF中文信息技术专委会. 文本自动生成研究进展与趋势[C]. 中国计算机学会, 2015: 298-323.
[8]	司畅, 张铁峰. 关于自然语言生成技术的研究[J]. 信息技术, 2010, 34(09):108-110.
[9]	Bex F J, Reed C. Dialogue Templates for Automatic Argu-ment Processing[C]. Computational Models of Argument Comma. 2012, 245:366-377. DOI: 10.3233/978-1-61499-111-3-366.
[10]	Krepych S, Spivak I. Algorithm of Automatic Generation of Hotel Descriptions Using Templates Based on Markov Chains[C]. 2018 International Scientific-Practical Conference Problems of Infocommunications. Science and Technology (PIC S&T), 2019: 257-260. DOI: 10.1109/INFOCOMMST.2018.8632149.
[11]	姜倩盼. 自然语言处理的挑战与未来[J]. 信息与电脑(理论版), 2013, 14:219-221.
[12]	蒋锐滢, 崔磊等. 基于主题模型和统计机器翻译方法的中文格律诗自动生成[J]. 计算机学报, 2015, 38(12):2426-2436.
[13]	Kyunghyun Cho, Bart van Merrienboer, et al. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation[DB/OL]. arXiv preprint arXiv:1406.1078,2014(2014-9-3) [2020-12-15]. https://arxiv.org/abs/1406.1078.
[14]	Sutskever I, Vinyals, Le Q V. Sequence to Sequence Lear-ning with Neural Networks[C]. NIPS MITPress, 2014, 2:3104-3112.
[15]	Bahdanau D, Cho K, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate[DB/OL]. arXiv preprint arXiv; 1409.0473,2014(2016-5-19)[2020-12-15]. https://arxiv.org/abs/1409.0473.
[16]	Vinyals O, Fortunato M, et al. Pointer Networks[C]. 12th International Conferenceon Neural Information Proce-ssing Systems.Montreal:MITPress, 2015:2692-2700.
[17]	Gu J, Lu Z, Li H, et al. Incorporating Copying Mechanism in Sequence-to-Sequence Learning[C]. Proceedings of the 54th Annual Meeting of the Association for Com-putational Linguistics, 2016, 1. DOI: 10.18653/v1/P16-1154.
[18]	Ranzato M, Chopra S, et al. Sequence level training with recurrent neural networks[DB/OL]. arXiv preprint arXiv: 1511.06732, 2015(2016-5-6)[2020-12-15]. https://arxiv.org/ abs/1511.06732
[19]	Kingma, Diederik P, et al. Auto-Encoding Variational Bayes[DB/OL]. arXiv preprint arXiv: 1312.6114,2013(2014-5-1)[2020-12-15].https://arxiv.org/abs/1312.6114.
[20]	Kingma D P, Salimans T, Jozefowicz R, et al. Improving Variational Inference with Inverse Autoregressive Flow[C]. Proc of the 30th Annual Conf on Neural Information Processing Systems. North Miami Beach, Florida: Curran Associates,Inc, 2016:4736-4744.
[21]	CHEN X, KINGMA D, SALIMANS T, et al. Variational lossy autoencoder[DB/OL]. arXiv preprint arXiv:1611. 02731,2016(2017-3-4)[2020-12-15]. https://arxiv.org/abs/1611.02731.
[22]	Shen Xiaoyu, Su Hui, et al. Improving Variational Encoder-Decoders in Dialogue Generation[C]. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018, 32(1).
[23]	Yang Zichao, Hu Zhiting, et al. Improved variational auto-encoders for text modeling using dilated convolutions[C]. The International Conference on Machine Learning, 2017, 70:3881-3890.
[24]	Deng Y, Kim Y, Chiu J, et al. Latent Alignment and Variational Attention[C]. In Advances in Neural Information Proces-sing Systems, 2018: 9712-9724.
[25]	Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Gene-rative Adversarial Networks[J]. Advances in Neural Infor-mation Processing Systems, 2014, 3:2672-2680.
[26]	Zhang Y, GAN Z, CAEINL.Generating Text via Adversarial Training[C]. In NIPS workshop on Adversarial Training, 2016, 21:238-265.
[27]	Yu L, Zhang W, Wang J, et al. SeqGAN: Sequence Gene-rative Adversarial Nets with Policy Gradient[C]. In Proce-edings of AAAI, 2017, 31(1).
[28]	Fedus W, Goodfellow I, Dai A M. MaskGAN: Better Text Generation via Filling in the ______[DB/OL]. arXiv preprint arXiv: 1801.07736,2018(2018-3-1)[2020-12-15]. https://arxiv.org/abs/1801.07736.
[29]	Pei Ke, Fei Huang, Minlie. et al. ARAML: A Stable Adver-sarial Training Framework for Text Generation[C]. Pro-ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019: 4262-4272.
[30]	Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need[C]. Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
[31]	Dai Z, Yang Z, Yang Y, et al. Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context[C]. Proceedings of the 57th Annual Meeting of the Associa-tion for Computational Linguistics. 2019: 2978-2988.
[32]	Kitaev N, Kaiser U, Levskaya A. Reformer: The Efficient Transformer[DB/OL]. arXiv preprint arXiv：2001.04451,2020(2020-2-18)[2020-12-15]. https://arxiv.org/abs/2001.04451.
[33]	Cao Z, Li W, Li S, et al. Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization[C]. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018(1):152-161.
[34]	Yang J, Hu M, Qiu C. A Hybrid Retrieval-Generation Neural Conversation Model[C]. The 28th ACM Interna-tional Conference, 2019: 1341-1350.
[35]	Cai D, Wang Y, Bi W, et al. Retrieval-guided Dialogue Response Generation via a Matching-to-Generation Framework[C]. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019: 1866-1875. DOI: 10.18653/v1/D19-1195.
[36]	Wang Q, Zhou Z, Huang L, et al. Paper Abstract Writing through Editing Mechanism[C]. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018(2):260-265.
[37]	Hancock B, Bordes A, Mazare P E, et al. Learning from Dialogue after Deployment: Feed Yourself, Chatbot![C]. Proceedings of the 57th Annual Meeting of the Asso-ciation for Computational Linguistics. 2019: 3667-3684.
[38]	Koncel-Kedziorski R, Bekal D, Luan Y, et al. Text Generation from Knowledge Graphs with Graph Transfor-merss[C]. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa-tional Linguistics:Human Language Technologies. Stro-udsburg:ACL, 2019: 2284-2293.
[39]	CHEN Tao, AN Junxiu. Sentiment Classification of Microblog Short Text Based on Feature Fusion[J]. Frontiers of Data & Computing, 2020, 2(6):21-29.DOI: 10.11871/jfdc.issn.2096-742X.2020.06.003.PID:21.86101.2/jfdc.2096-742X.2020.06.003.
[40]	Vig J, Ramea K. Comparison of Transfer-Learning Approaches for Response Selection in Multi-Turn Conver-sations[C]. Association for the Advancement of Artificial Intelligence. 2019.
[41]	Bao S, He H, Wang F, et al. PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable[C]. Proceedings of the 58th Annual Meeting of the Associa-tion for Computational Linguistics, 2020:85-96.
[42]	C Qu, L Yang, M Qiu, et al. BERT with History Answer Embedding for Conversational Question Answering[C]. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019: 1133-1136.
[43]	Z Wang, P Ng, X Ma, et al. Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering[C]. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019: 5878-5882.
[44]	J Li, M Galley, C Brockett, et al. A Diversity-Promoting Objective Function for Neural Conversation Models[C]. In NAACL, 2016:110-119.
[45]	J Li, W Monroe, J Dan.A Simple, Fast Diverse Decoding Algorithm for Neural Generation[DB/OL].arXiv preprint arXiv:1611.08562,2016(2016-12-22) [2020-12-15]. https://arxiv.org/abs/1611.08562.
[46]	L Shao, S Gouws, D Britz, et al. Generating Long and Diverse Responses with Neural Conversation Models[DB/OL]. arXiv preprint arXiv:1701.03185,2017(2017-7-31) [2020-12-15]. https://arxiv.org/abs/1701.03185v1.
[47]	Wang K, Wan X. SentiGAN: Generating Sentimental Texts via Mixture Adversarial Networks[C]. Twenty-Seventh International Joint Conference on Artificial Intelligence IJCAI-18, 2018: 4446-4452.
[48]	Liu Z, Fu Z, Cao J, et al. Rhetorically Controlled Encoder-Decoder for Modern Chinese Poetry Generation[C]. THE 57TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2019), 2019: 1992-2001.
[49]	Gao J, Bi W, Liu X, et al. A Discrete CVAE for Response Generation on Short-Text Conversation[C]. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019: 435-444.
[50]	Yao L, Zhang Y, Feng Y, et al. Towards Implicit Content-Introducing for Generative Short-Text Conversation Systems[C]. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017:2190-2199.
[51]	Dziri N, Kamalloo E, Mathewson K W, et al. Augmenting Neural Response Generation with Context-Aware Topical Attention[C]. Proceedings of the First Workshop on NLP for Conversational AI, 2018: 18-31.
[52]	Zheng H T, Wang W, Chen W, et al. Automatic Generation of News Comments Based on Gated Attention Neural Networks[C]. In Proceedings of IEEE Access, 2018, 6:702-710.
[53]	Asghar N, Poupart P, Hoey J, et al. Affective Neural Response Generation[C]. In European Conference on InformationRetrieval, 2017: 154-166.
[54]	Zhong P, Wang D, Miao C. An Affect-Rich Neural Conversational Model with Biased Attention and Weighted Cross-Entropy Loss[C]. Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence. 2019, 33(01):7492-7500.
[55]	Zhou H, Young T, Huang M, et al. Commonsense Knowle-dge Aware Conversation Generation with Graph Attention[C]. Twenty-Seventh International Joint Conference on Artificial Intelligence IJCAI-18, 2018: 4623-4629.
[56]	Lian R, Xie M, Wang F, et al. Learning to Select Knowledge for Response Generation in Dialog Systems[C]. Twenty-Eighth International Joint Conference on Artificial Inte-lligence IJCAI-19. 2019: 5081-5087.
[57]	Wang Z, Wang J, Gu H, et al. Automatic Conditional Generation of Personalized Social Media Short Texts[C]. Proceedings of the Pacific Rim International Conference on Artificial Intelligence, 2019:56-63.
[58]	Liu B, Xu Z, Sun C, et al. Content-Oriented User Modeling for Personalized Response Ranking in Chatbots[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2017, 26(1):122-133. doi: 10.1109/TASLP.2017.2763243
[59]	Ni J, Mcauley J. Personalized Review Generation by Expanding Phrases and Attending on Aspect-Aware Representations[C]. Proceedings of the 56th Annual Meet-ing of the Association for Computational Linguistics. 2018: 706-711.
[60]	Luo L, Huang W, Zeng Q, et al. Learning Personalized End-to-End Goal-Oriented Dialog[C]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33:6794-6801
[61]	P Li, H Zhang, X Liu, S Shi. Rigid Formats Controlled Text Generation[C]. In ACL, 2020: 742-751.
[62]	J Zhang, Y Zhao, et al. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization[C]. In Proceedings of the 37th International Conference on Machine Learning, 2020c: 2021-2032.
[63]	Baolin Peng, Chenguang Zhu, et al. Few-shot natural language generation for task-oriented dialog[C]. the Association for Computational Linguistics: EMNLP 2020, 2020:172-182.
[64]	Su Y, Cai D, Wang Y, et al. Stylistic Dialogue Generation via Information-Guided Reinforcement Learning Strategy[DB/OL]. arXiv preprint arXiv:2004.02202, 2020(2020-4-5)[2020-12-15]. https://arxiv.org/abs/2004.02202.
[65]	Su H, Shen X, Zhao S, et al. Diversifying Dialogue Generation with Non-Conversational Text[C]. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 7087-7097.
[66]	Siyu Duan, Wei Li, Cai Jing, et al. Query-Variant Adver-tisement Text Generation with Association Knowledge[DB/OL].arXiv preprint arXiv: 2004. 06438. 2020(2020-4-14)[2020-12-15].https://arxiv.org/abs/2004.06438.
[67]	倪海清, 刘丹, 史梦雨. 基于语义感知的中文短文本摘要生成模型[J]. 计算机科学, 2020, 047(006):74-78.
[68]	Wang J, Liu J, Bi W, et al. Improving Knowledge-Aware Dialogue Generation via Knowledge Base Question Answering[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(5):9169-9176. doi: 10.1609/aaai.v34i05.6453
[69]	Byeongchang Kim, Jaewoo Ahn, Gunhee Kim. Sequen-tial latent knowledge selection for knowledge-grounded dialogue[DB/OL]. arXiv preprint arXiv: 2002.07510. 2020(2020-6-16)[2020-12-15].https://arxiv.org/abs/2002.07510.
[70]	Zheng Y, Zhang R, Huang M, et al. A Pre-Training Based Personalized Dialogue Generation Model with Persona-Sparse Data[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(5):9693-9700. doi: 10.1609/aaai.v34i05.6518
[71]	Yang M, Huang W, Tu W, et al. Multitask Learning and Reinforcement Learning for Personalized Dialog Genera-tion: An Empirical Study[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, (99):1-14.
[72]	Chen Y H, Chen P Y, Shuai H H, et al. TemPEST: Soft Template-Based Personalized EDM Subject Generation through Collaborative Summarization[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(5):7538-7545. doi: 10.1609/aaai.v34i05.6252
[73]	Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, et al. How2: a large-scale dataset for multimodal language understanding[DB/OL].arXiv preprint arXiv: 1811.00347, 2018(2018-12-7)[2020-12-15].https://arxiv.org/abs/1811.00347.
[74]	Junnan Zhu, Yu Zhou, Jiajun Zhang, et al. Multimodal sum-marization with guidance of multimodal reference[C]. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(5):9749-9756.
[75]	Chen Cui, Wenjie Wang, Xuemeng Song, et al. User Attention-guided Multimodal Dialog Systems[C]. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2018:445-454.
[76]	K Papineni, S Roukos, T Ward. BLEU: a method for automatic evaluation of machine translation[C]. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, 2002: 311-318.
[77]	C Lin. Rouge: a package for automatic evaluation of summaries[C]. Association for Computational Lingui-stics, 2004: 74-81.
[78]	Blei D M, Ng A, Jordan M I. Latent dirichlet allocation[J]. The Journal of Machine Learning Research, 2003, 3: 993-1022.
[79]	Li J, Galley M, Brockett C, et al. A Diversity-Promoting Objective Function for Neural Conversation Models[C]. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2015: 110-119.
[80]	Zhu Y, Lu S, Zheng L, et al. Texygen: A Benchmarking Platform for Text Generation Models[C]. The 41st Inter-national ACM SIGIR Conference ACM, 2018: 1097-1100.
[81]	A Celikyilmaz, E Clark, J Gao. Evaluation of Text Generation:A Survey[DB/OL].arXiv preprint arXiv: 2006.14799,2020(2020-6-26)[2020-12-15].https://arxiv.org/abs/2006.14799.
[82]	A Kannan, O Vinyals. Adversarial Evaluation of Dia-logue Models[DB/OL].arXiv preprint arXiv: 1701.08198,2017 (2017-1-27)[2020-12-15].https://arxiv.org/abs/1701.08198.
[83]	R Lowe, M Noseworthy, IV Serban, et al. Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses[C]. Pro of the 55th Annual Meeting of the Association for Computational Linguistics, 2017: 1116-1126.
[84]	Hassan Kane, Yusuf Kocyigit, Pelkins Ajanoh, et al. Towards neural language evaluators[DB/OL]. arXiv preprint arXiv: 1909.09268,2019(2019-10-30) [2020-12-15]. https://arxiv.org/abs/1909.09268.
[85]	Tianyi Zhang, Varsha Kishore, Felix Wu, et al. Bertscore: Evaluating text generation with bert[DB/OL]. arXiv preprint arXiv: 1904.09675,2020(2020-2-24) [2020-12-15]. https://arxiv.org/abs/1904.09675.
[86]	Thibault Sellam, Dipanjan Das, Ankur Parikh. BLEURT: Learning Robust Metrics for Text Generation[C]. The International association of computational linguistics, 2020: 7881-7892.
[87]	Liao Fangyu, Hong Xuehai, Wang Yang, Chu Dawei. The Data and Computing Platform is An Important Infrastructure Which Drives Modern Scientific Research Deve-lopment[J]. Frontiers of Data & Computing, 2019, 1(1):2-10.DOI: 10.11871/jfdc.issn.2096.742X.2019.01.002. PID:21.86101.2/jfdc.2096-742X.2019.01.002.