Abstract:

[Objective] In recent years, the multimodal data analysis model combined with text and image has gradually become an important approach for sentiment analysis in social networks. [Methods] Aiming at the problem of image and text feature fusion in multimodal sentiment analysis, a multimodal sentiment analysis model TSAIE based on image enhancement is proposed. The model extracts text features and image features respectively. A combined attention graphic feature fusion module based on Transformer Encoder and attention mechanism is designed. Through this module, the information relevance of each word and picture in the text is calculated to improve the emotional representation ability of text features. Finally, the text feature and image feature after combined attention calculation are concatenated and input into the full connection layer. [Conclusions] The experimental results show that the accuracy and F1 value of sentiment classification are increased by 3.11% and 2.53% respectively on the MVSA-single data set, and 1.33% and 0.74% respectively on the MVSA-Multi data set, thus verifying the effectiveness of the TSAIE model.

Key words: multimodal, sentiment analysis, Transformer, attention mechanism, image add text feature fusion