数据与计算发展前沿 ›› 2022, Vol. 4 ›› Issue (3): 131-140.

CSTR: 32002.14.jfdc.CN10-1649/TP.2022.03.010

doi: 10.11871/jfdc.issn.2096-742X.2022.03.010

• 技术与应用 • 上一篇    下一篇

TSAIE：图像增强文本的多模态情感分析模型

刘琦玮1,2(),李俊1,*(),顾蓓蓓1(),赵泽方1,2()   

  1. 1.中国科学院计算机网络信息中心,北京 100083
    2.中国科学院大学,北京 100049
  • 收稿日期:2021-08-10 出版日期:2022-06-20 发布日期:2022-06-20
  • 通讯作者: 李俊
  • 作者简介:刘琦玮,中国科学院计算机网络信息中心,硕士研究生,主要研究领域为自然语言处理、情感分析等。
    本文中承担的任务是模型设计、实验设计与文献撰写。
    LIU Qiwei is a graduate student in the Computer Network Information Center of Chinese Academy of Sciences. Her main research areas are natural language processing and sentiment analysis.
    In this paper, she is responsible for the model design, experi-ments design, and paper writing.
    E-mail: liuqiwei@cnic.cn|李俊,中国科学院计算机网络信息中心,研究员,博士生导师,中国科学院特聘研究员。主要研究领域为人工智能和大数据应用、互联网体系结构等。
    本文中负责研究指导,论文结构组织。
    LI Jun is a research fellow and Ph.D. supervisor at the Comp-uter Network Information Center of Chinese Academy of Sciences, specially appointed researcher of Chinese Academy of Sciences. His main research areas are artificial intelligence and big data technical applications and future Internet architecture.
    In this paper, he is responsible for the research guidance and paper structure organization.
    E-mail: lijun@cnic.cn|顾蓓蓓,中国科学院计算机网络信息中心,高级工程师,硕士,主要研究方向为高性能计算和人工智能战略研究。
    本文中承担的任务是研究指导。
    GU Beibei, master’s degree, is a senior engineer at the Comp-uter Network Information Center of Chinese Academy of Sci-ences. Her main research areas are High Performance Computing and Artificial Intelligence Strategy Research.
    In this paper, she is responsible for the research guidance.
    E-mail: gbb@cnic.cn|赵泽方,中国科学院计算机网络信息中心,博士研究生,主要研究领域为自然语言处理、情感分析等。
    本文中承担的任务为实验设计。
    ZHAO Zefang is a Ph.D. student in Com-puter Network Information Center of Chinese Academy of Sciences. His main research areas are natural language processing and sentiment analysis.
    In this paper,he is responsible for the experiments design.
    E-mail: zhaozefang@cnic.cn

TSAIE: Text Sentiment Analysis Model Based on Image Enhancement

LIU Qiwei1,2(),LI Jun1,*(),GU Beibei1(),ZHAO Zefang1,2()   

  1. 1. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100083, China
    2. University of Chinese Academy of Sciences, Beijing 100049, China
  • Received:2021-08-10 Online:2022-06-20 Published:2022-06-20
  • Contact: LI Jun

RichHTML

44

PDF

311

摘要：

【目的】近年来,以图文结合的多模态数据分析模型已经逐渐成为社交网络中情感分析的重要途径。【方法】本文针对多模态情感分析中存在的图文特征融合问题,提出了一种基于图像增强文本的多模态情感分析模型TSAIE。该模型分别提取文本特征和图片特征,然后设计了基于Transformer编码器与注意力机制的组合注意力图文特征融合模块,通过该模块计算出文本中的每一个词和图片的信息相关度,提升文本特征的情感表征能力,最后将经过组合注意力计算之后的文本特征与图片特征拼接后输入全连接层。【结论】实验结果表明,在MVSA-Single数据集上,情感分类的准确率和F1值分别提高了3.11%和2.53%,在MVSA-Multi数据集上,情感分类的准确率和F1值分别提高了1.33%和0.74%,从而验证了TSAIE模型的有效性。

关键词: 多模态, 情感分析, Transformer, 注意力机制, 图文特征融合

Abstract:

[Objective] In recent years, the multimodal data analysis model combined with text and image has gradually become an important approach for sentiment analysis in social networks. [Methods] Aiming at the problem of image and text feature fusion in multimodal sentiment analysis, a multimodal sentiment analysis model TSAIE based on image enhancement is proposed. The model extracts text features and image features respectively. A combined attention graphic feature fusion module based on Transformer Encoder and attention mechanism is designed. Through this module, the information relevance of each word and picture in the text is calculated to improve the emotional representation ability of text features. Finally, the text feature and image feature after combined attention calculation are concatenated and input into the full connection layer. [Conclusions] The experimental results show that the accuracy and F1 value of sentiment classification are increased by 3.11% and 2.53% respectively on the MVSA-single data set, and 1.33% and 0.74% respectively on the MVSA-Multi data set, thus verifying the effectiveness of the TSAIE model.

Key words: multimodal, sentiment analysis, Transformer, attention mechanism, image add text feature fusion