Text-image semantic relevance identification for aspect-based multimodal sentiment analysis

Multimodal sentiment analysis (MSA)

In recent years, MSA task has attracted widespread attention in academic community (Cambria et al., 2017; Poria et al., 2020), which aims to model text and other non-text modalities (e.g., visual and auditory modalities), and generally involves two subtasks: MSA for conversation and MSA for social media. In MSA for conversation, existing methods mainly focus on adopting different deep learning models (e.g., long short-term memory network (Hochreiter & Schmidhuber, 1997), gate recurrent unit (Chung et al., 2014), and Transformer (Vaswani et al., 2017)) to interact the information between different modalities, which have demonstrated better performance in various MSA tasks (e.g., sentiment analysis (Zadeh et al., 2017; Poria, Cambria & Gelbukh, 2015; Poria et al., 2017; Liang et al., 2018), emotion analysis (Busso et al., 2004; Lee et al., 2011), and sarcasm detection (Castro et al., 2019; Cai, Cai & Wan, 2019)). In MSA for social media, it mainly includes sentiment analysis of social media image (Chen et al., 2014b; You et al., 2015; Yang et al., 2018a; Yang et al., 2018b) and multimodal sentiment analysis of text-image integration (You et al., 2016; Kumar & Garg, 2019; Kumar et al., 2020; Xu, Mao & Chen, 2018). However, the above research methods mainly focus on coarse-grained sentiment analysis (i.e., identifying the global sentiment reflected by each sample) and cannot be directly employed for fine-grained ABMSA tasks.

Aspect-based sentiment analysis (ABSA)

As an important fine-grained sentiment analysis task, ABSA has been widely researched and applied in Natural Language Processing (NLP) field over the past decade (Cambria et al., 2017), and its current methods can be broadly divided into two categories: discrete feature-based method and deep learning-based method. Discrete feature-based method focuses on designing multi-specific features to train learning classifiers for sentiment analysis (Vo & Zhang, 2015; Pontiki et al., 2016). Deep learning-based method mainly adopts various neural network models to encode aspects and corresponding context information, including the method based on recursive neural network (Dong et al., 2014), convolutional neural network (Xue & Li, 2018), recurrent neural network (Ma, Peng & Cambria, 2018; Chen et al., 2017), attention mechanism (Wang et al., 2018; Yang et al., 2019; Meškele & Frasincar, 2020; Zhao et al., 2021), graph convolutional network (Wang et al., 2020; Zhang & Qian, 2020), and pre-trained BERT model that has achieved great success in the past few years (Xu et al., 2019; Sun, Huang & Qiu, 2019). However, the above research methods mainly focus on text-based unimodal information, and do not take into account the fact that relevant information from other modalities (e.g., visual modality) can also contribute to sentiment analysis.

Aspect-Based Multimodal Sentiment Analysis (ABMSA)

To conduct research on ABSA utilizing information from different modalities, researchers have developed numerous models for ABMSA in recent years by employing various effective methods in different tasks. Xu et al. (2019) first explored the ABMSA task and proposed a multi-interactive memory network model MIMN based on bidirectional long short-term memory (BiLSTM) network, which extracts specific aspect sentiment information from text and image, while also constructed an e-commerce comment dataset for the task. Yu & Jiang (2019) proposed an ABMSA model TomBERT based on the BERT (Devlin et al., 2018) architecture that is the basis for most extended models at present, and manually constructed two ABMSA Twitter datasets. Yu & Jiang (2019) proposed an ABMSA model ESAFN based on entity-sensitive attention and fusion network, which captures the relationship between aspect, text, and image information effectively. Lu et al. (2019) proposed a pre-trained visual language model ViLBERT that takes aspect-text pairs as input text, and the model can also be employed for the ABMSA task. Khan & Fu (2021) proposed an innovative model CapBERT that employs cross-modal transformation to convert the image content into text caption, and performs final sentiment analysis based on text modality only. Wang et al. (2021) proposed a recurrent attention network model SaliencyBERT also based on BERT, which effectively captures both intra-modal and inter-modal dynamics by designing a recurrent attention mechanism. Zhao et al. (2022) proposed a knowledge enhancement framework KEF based on adjective-noun pairs (ANPs) to incorporate into various models, which improves the visual attention and sentiment prediction capabilities in the ABMSA task. Although the above ABMSA methods have been validated to be effective, they often neglect to identify whether the semantics between modalities are relevant. Inspired by the above KEF framework, our model introduce ANPs as image auxiliary information, but capture the semantic relevance between modalities by calculating the similarity between text and image features before multimodal interaction, which facilitates the effective development of subsequent work.

Overview

Figure 1 illustrates the overall architecture of TISRI, which contains the following modules: (1) Unimodal Feature Extraction Module. (2) Multimodal Feature Relevance Identification Module. (3) Aspect-Multimodal Feature Interaction Module. (4) Multimodal Feature Fusion Module.

As shown at the bottom of Fig. 1, for a given multimodal sample, we first extract word feature representations from the input text and aspect, respectively, and visual feature representation from the input image, then interact aspect representation with text and image representation to generate aspect-aware text representation and aspect-aware image representation.

Next, we obtain the semantic similarity between text and image by constructing a multimodal feature relevance identification module. The overall method is shown in Fig. 2, where the fusion representations of text and image are obtained through cross-modal interaction, and then an image gate is constructed in a specific way to dynamically control the input image information.

To enable better semantic expression of image features, we propose an image feature auxiliary reconstruction layer. As shown in Fig. 3, the image visual representation is fine-tuned by introducing ANPs extracted from each image in the adopted datasets as image auxiliary information to minimize their representation differences.

Finally, to prevent the model being influenced by irrelevant image region content, we interact aspect-aware text representation with aspect-aware image representation, and then generate the final image representation. As shown at the top of Fig. 1, we further concatenate aspect-aware text representation and final image representation, and obtain the final sentiment label through a sentiment analysis linear layer.

Unimodal feature extraction module

In this module, we adopt two pre-trained models to extract unimodal feature representations from aspect, text and image, respectively.

Aspect and text representation

Given an input text, we divide it into two parts: aspect T and its corresponding context C, and replace the aspect position in C with a special character “$T$”. For text encoding, we employ pre-trained language model RoBERTa (Liu et al., 2019) as the text encoder of our model, which has been proven to achieve more competitive performance in various NLP tasks including ABSA as an enhancement to the BERT architecture (Dai et al., 2021). For T and C, we follow the implementation mechanism of RoBERTa by inserting two special tokens into each input (i.e., “<s>” at the beginning and “</s>” at the end), and then feeding them into text encoder to obtain the hidden representations of aspect and context: (1)${\displaystyle H_T=\mathrm{RoBERTa}\left(T\right)}$ (2)${\displaystyle H_C=\mathrm{RoBERTa}\left(C\right)}$ where H_T ∈ ℝ^d×t and H_C ∈ ℝ^d×c, d is the hidden dimension, t is the length of aspect, and c is the length of context.

Next, we concatenate C with T as sentence S. For S, we use the token “</s>” to separate C from T, and then obtain the hidden representation of sentence through RoBERTa as follows: (3)${\displaystyle H_S=\mathrm{RoBERTa}\left(S\right)}$ where H_S ∈ ℝ^d×s, s = c + t is the length of sentence. The implementations of aspect, context, and sentence encoding are shown at the bottom of Figs. 1 and 2.

Image representation

For image encoding, we employ residual network (ResNet) (He et al., 2016) as the image encoder of our model. Compared to the previous VGG network (Simonyan & Zisserman, 2014), ResNet adopts residual connections to avoid gradient vanishing problem as the number of layers increases, which allows for deeper extraction of semantic information in image recognition tasks. Specifically, given an input image I, we first resize it to I′ with 224 × 224 pixels, and then take the output of the last convolutional layer in pre-trained 152-layer ResNet as the image visual representation: (4)${\displaystyle H_I=\mathrm{ResNet}\left(I^{{}^{\prime }}\right)}$ where H_I ∈ ℝ^2,048×49, 49 is the number of visual blocks with the same size by dividing I′ into 7 ×7, and 2,048 is the vector dimension of each visual block.

Since we will subsequently conduct cross-modal interaction with text and image, it is necessary to project image representation to the same semantic space as text representation. We employ a linear transformation function for H_I to obtain the projected image representation: (5)${\displaystyle H_V=W_I^T{H}_I+b_I}$ where $W_I^T\in {\mathbb{R}}^{d\times 2,048}$ and b_I ∈ ℝ^d is the learnable parameters. The implementation of image encoding is shown at the bottom of Fig. 1.

Multimodal feature relevance identification module

For images in multimodal samples, while they can provide information beyond text, our purpose is to use image to assist in analyzing the sentiment polarity of aspect in text, and images that are irrelevant to text semantics may lead to misalignment of aspects and introduce additional interference to the model. Therefore, we propose a Multimodal Feature Relevance Identification (MFRI) module, which provides an image gate when integrating text and image, and dynamically controls the input image information based on its relevance to text semantics. The module is divided into two layers: (1) Text-Image Cross-Modal Interaction layer. (2) Image Gate Construction layer. As shown in Fig. 2, we provide a detailed introduction to the implementation methods of these two layers in the following sections.

Text-image cross-modal interaction layer

To better learn sentence feature representation in image, we introduce a Multi-head Cross-modal Attention (MC-ATT) mechanism (Tsai et al., 2019) layer, which treats image representation H_V as query, and sentence representation H_S as key and value, then involves two layer normalization (LN) (Ba, Kiros & Hinton, 2016) and a feedforward network (FFN) (Vaswani et al., 2017) as follows: (6)${\displaystyle Z_{V\to S}=\mathrm{LN}\left(H_V+\text{MC-ATT}\left(H_V,H_S\right)\right)}$ (7)${\displaystyle H_{V\to S}=\mathrm{LN}\left(Z_{V\to S}+\mathrm{FFN}\left({{Z{}_V}_{\to }}_S\right)\right)}$ where H_V→S ∈ ℝ^d×49 is the image-aware sentence representation generated by MC-ATT layer. However, image representation is treated as query in the above MC-ATT layer, and each vector in H_V→S represents a visual block rather than a word representation in sentence. We expect that image-aware sentence representation can reflect on each word in sentence. Given this problem, we introduce another MC-ATT layer that treats H_S as query, and H_V→S as key and value, then generates the final image-aware sentence representation $H_{V\to S}^{{}^{\prime }}$, where $H_{V\to S}^{{}^{\prime }}\in {\mathbb{R}}^{d\times s}$.

To obtain the image representation for each word in sentence, we adopt the same method as above for cross-modal interaction, treating H_S as query, and H_V as key and value, then generating the sentence-aware image representation H_S→V, where H_S→V ∈ ℝ^d×s.

Image gate construction layer

Yu et al. (2020) introduced visual gate to dynamically control the contribution of image visual features to each word in text in the multimodal named entity recognition task, and achieved effective experimental results. Inspired by this work, we construct a gate for the input image in our model, which is responsible for dynamically controlling the contribution of image information by assigning a weight in [0, 1] to each image based on its relevance to corresponding sentence, (i.e., preserving the higher relevance image by assigning a higher weight, and filtering the lower relevance image by assigning a lower weight) Specifically, we first concatenate $H_{V\to S}^{\prime }$ and H_S→V, and then construct the gate based on text-image relevance weight through linear transformation and nonlinear activation function: (8)${\displaystyle g=\sigma \left(W_{S\to V}^T\left[H_{V\to S}^{{}^{\prime }};H_{S\to V}\right]+{b_S}_{\to V}\right)}$ where $W_{S\to V}^T\in {\mathbb{R}}^{d\times 2d}$ and b_S→V ∈ ℝ^d is the learnable parameters, σ is the element-wise nonlinear activation function, which is used to control the output of g in [0, 1].

Based on the above image gate g, we can obtain the image representation that assigns relevance weight: (9)${\displaystyle H_V^{{}^{\prime }}=g\cdot H_{S\to V}}$

Aspect-multimodal feature interaction module

After obtaining the feature representations of aspect, context, and image, we analyze the relationships between aspect and image as well as context, respectively. Furthermore, we design an Image Feature Auxiliary Reconstruction (IFAR) layer, which serves as an auxiliary supervision for visual representation. The specific technical scheme of this module is shown at the middle part of Fig. 1, and the internal architecture of IFAR layer is shown in Fig. 3. We provide detailed implementation methods for them in the following sections.

Aspect interaction layer

The main purpose of this layer is to obtain aspect-aware image representation and aspect-aware context representation, so we introduce MC-ATT layer to interact with aspect and image as well as context, respectively, to promote information integration between modalities. Specifically, we first conduct cross-modal feature interaction between aspect and image, treating aspect representation H_T as query, and image representation H_V as key and value: (10)${\displaystyle Z_{T\to V}=\mathrm{LN}\left(H_T+\mathrm{MC}-\mathrm{ATT}\left(H_T,H_V\right)\right)}$ (11)${\displaystyle H_{T\to V}=\mathrm{LN}\left(Z_{T\to V}+\mathrm{FFN}\left({{Z{}_T}_{\to }}_V\right)\right)}$ where H_T→V ∈ ℝ^d×t is the aspect-aware image representation generated by MC-ATT layer.

Similarly, we can also obtain the aspect-aware context representation H_T→C, where H_T→C ∈ ℝ^d×t.

Image feature auxiliary reconstruction layer

To improve the effectiveness of visual feature representation, we introduce ANPs extracted from image in each sample. Since the nouns and adjectives in ANPs can reflect real content and sentiment in image to some extent, we employ them as auxiliary supervision for visual representation to obtain a more intuitive image semantic expression. Specifically, we adopt DeepSentiBank (Chen et al., 2014a) to generate 2,089 ANPs for each image and select the top k ANPs as image auxiliary information. As the example in ‘Introduction’, the top five ANPs are “hot blonde”, “sexy blonde”, “great smile”, “pretty dress” and “sexy model”.

However, the extraction of image ANPs is essentially a coarse-grained extraction method, so ANPs may contain the content of image regions that are irrelevant to aspect or semantic information that is incorrectly recognized for image, and directly using these ANPs can significantly introduce additional interference to the model due to their inaccuracy. Zhao et al. (2022) obtained nouns relevant to aspect by calculating semantic similarity between aspect representation and ANPs noun representation in the construction of ABMSA knowledge enhancement framework, and achieved excellent alignment effect in their experiment. Inspired by this work, we concatenate the above k ANPs and their corresponding nouns, respectively, and feed them into text encoder to obtain the ANPs representation H_ANPs and the noun representation H_N, and then we employ cosine similarity to calculate the semantic similarity between H_T and H_N to achieve the aspect alignment: (12)${\displaystyle \alpha =\frac{H_T^T\cdot H_N}{∥H_T∥\cdot ∥H_N∥}}$ where α is the similarity score between H_T and H_N, which we use as a weight vector representing the semantic relevance of ANPs to the aspect in image. Next, we assign each individual in ANPs representation with its corresponding relevance weight to obtain the image auxiliary information representation: (13)${\displaystyle H_{ANPs}^{{}^{\prime }}=\alpha \cdot H_{ANPs}}$

Furthermore, based on the construction of image gate g in MFRI module, we also treat g as image auxiliary information gate to dynamically control the contribution of ANPs to the model, and then obtain the final image auxiliary information representation: (14)${\displaystyle H_{ANPs}^{{}^{\prime \prime }}=g\cdot H_{ANPs}^{{}^{\prime }}}$

To enable visual attention to be more intuitive and accurate in representing the visual features of aspect in image, we introduce a reconstruction loss function based on mean square error (MSE) to minimize the difference between aspect-aware image representation H_T→V and final image auxiliary information representation $H_{ANPs}^{{}^{\prime \prime }}$: (15)${\displaystyle {\mathcal{L}}_R=\frac{1}{\left|D\right|}\sum _{i=1}^{\left|D\right|}{\left(H_{ANPs}^{{}^{\prime \prime }}-H_{T\to V}\right)}^2}$

Multimodal feature fusion module

In this module, we fuse aspect-aware context representation H_T→C and aspect-aware image representation H_T→V with our Image Feature Filtering (IFF) method to obtain the final aspect output representation. The implementation is shown at the top of Fig. 1. First, we introduce MC-ATT layer to implement the interaction between H_T→C and H_T→V to obtain the visual feature representation corresponding to aspect-aware context in aspect-aware image as the final aspect-aware image representation, and then filter the irrelevant region content in image: (16)${\displaystyle Z_{T\to C\to V}=\mathrm{LN}\left(H_{T\to C}+\mathrm{MC}-\mathrm{ATT}\left(H_{T\to C},H_{T\to V}\right)\right)}$ (17)${\displaystyle H_{T\to C\to V}=\mathrm{LN}\left(Z_{T\to C\to V}+\mathrm{FFN}\left({{Z{}_{T\to C}}_{\to }}_V\right)\right)}$

Next, we concatenate H_T→C and H_T→C→V, and then feed them into a multimodal self-attention layer based on Transformer for feature fusion between modalities: (18)${\displaystyle H=\mathrm{Transformer}\left(H_{T\to C};H_{T\to C\to V}\right)}$

Finally, we feed the first token representation H⁰ into Softmax layer to obtain the final sentiment label: (19)${\displaystyle P\left(y\left|H\right.\right)=\mathrm{Softmax}\left(W^T{H}^0+b\right)}$

We adopt the cross entropy loss constructed by predicted values of aspect-based sentiment labels and their true values as the training loss function for the model sentiment analysis task: (20)${\displaystyle {\mathcal{L}}_S=\frac{1}{\left|D\right|}\sum _{j=1}^{\left|D\right|}logP\left(y^j\left|H^0\right.\right)}$

To further optimize all parameters of our model, we train the loss function for sentiment analysis jointly with image auxiliary reconstruction, and then construct the final training loss function combining the two tasks: (21)${\displaystyle \mathcal{L}={\mathcal{L}}_S+\lambda {\mathcal{L}}_R}$ where λ is the tradeoff hyper-parameter used to control the contribution of reconstruction loss.

Experiment

In this chapter, we conduct extensive experiments on two ABMSA datasets to validate the effectiveness of our TISRA model.

Experimental settings

Compared baselines

In this section, we evaluate the performance of TISRI by comparing it to various existing methods. Specifically, we consider comparing the following unimodal and multimodal methods with our model:

• Res-Target: a baseline method for obtaining the visual feature representation of input image directly from the ResNet model.

• AE-LSTM (Wang et al., 2016): an attention-based LSTM model for obtaining important context relevant to aspect.

• MGAN (Fan, Feng & Zhao, 2018): a multi-grained attention network that fuses aspect and context at different granularity.

• BERT (Devlin et al., 2018): a pre-trained language model with stacked Transformer encoder layers for the interaction between aspect and text.

• RoBERTa (Liu et al., 2019): a further improved BERT model by adopting better training strategies and larger corpus.

• MIMN (Xu et al. 2019): a multi-interactive memory network for the interaction between aspect, text, and image.

• ESAFN (Yu & Jiang, 2019): an entity-sensitive attention and fusion network for obtaining inter-modal dynamics of aspect, text, and image.

• ViLBERT (Lu et al., 2019): a pre-trained visual language model that takes aspect-text pairs as input text.

• TomBERT (Yu & Jiang 2019): an aspect-aware ABMSA method based on multimodal BERT model architecture.

• SaliencyBERT (Wang et al., 2021): a recursive attention network based on multimodal BERT model architecture for ABMSA.

• CapBERT (Khan & Fu 2021): a method of converting image into text caption and feeding it with the input text to a pre-processed BERT model.

• KEF-TomBERT (Zhao et al., 2022): an extended baseline to apply a proposed knowledge enhancement framework KEF to TomBERT.

• KEF-SaliencyBERT (Zhao et al., 2022): an extended baseline to apply a proposed knowledge enhancement framework KEF to SaliencyBERT.

• CapRoBERTa: an extended baseline that replaces BERT with RoBERTa in CapBERT.

• KEF-TomRoBERTa: an extended baseline that replaces BERT with RoBERTa in KEF-TomBERT.

Ablation study

To further investigate the impact of individual unit in TISRI on model performance, we perform ablation analysis on TWITTER-2015 and TWITTER-2017 datasets for the main designed units in our model: (1) Image Feature Filtering (IFF) method. (2) Image Feature Auxiliary Reconstruction (IFAR) layer. (3) Multimodal Feature Relevance Identification (MFRI) module. We first remove the above three units respectively, and then remove these units at the same time leaving only the bone framework, so we can have a clearer and more comprehensive understanding of the contribution of individual unit to the model performance improvement. The experimental results are shown in Table 6, where w/o represents the removal of corresponding unit.

First, we can learn that removing IFF unit decreases accuracy by about 1.9% and 1.5% on the two datasets, respectively, which validates that retaining useful information in image and implementing filtering on text-irrelevant image region content helps reduce the impact of interference on model performance. Next, removing IFAR unit decreases accuracy by about 2.1% and 2.5% on the two datasets, respectively, which proves that the unit has a large contribution to model performance improvement and validates that adopting ANPs as image auxiliary information can be more intuitive for semantic expression of visual feature representation. Then, removing the MFCR unit decreases accuracy by about 0.7% and 1.8% on the two datasets, respectively, which validates that assigning image to an inter-modal relevance weight can prevent additional interference to the model from text-irrelevant images. We can also learn that there are more images with higher text-image relevance weights in TWITTER-2015 dataset than in TWITTER-2017 dataset, which validates the reason we inferred in ‘Values of epoch and batch size’. Finally, we remove all above units and observe that accuracy decreases by about 2.2% and 4.5% on the two datasets, respectively, which validates the effectiveness of our designed units in the model and also validates that these units contribute to model performance improvement to some extent from another perspective.

Parameter analysis

In this section, we provide a detailed introduction and analysis of the process of evaluating optimal hyper-parameters. All the above experiments are set based on optimized model hyper-parameters.

Values of epoch and batch size

To analyze the impact of different epoch and batch size on model performance, we determine the final values of epoch and batch size through experiments in this subsection. Figures 4 and 5 demonstrate the model performance of different epoch and batch size values on the two datasets, respectively, and we can draw the following inferences. First, we experiment with the value of epoch. We find that as the value of epoch increases, the model performance shows an upward trend and then gradually stabilizes. The model performance is optimal when epoch equals 8, and then accuracy and macro-F1 start to gradually decrease when epoch equals 9. Thus, we set the value of epoch to 9 in our experiments. Accuracy and macro-F1 corresponding to the epoch setting of TISRI on TWITTER-2015 and TWITTER-2017 datasets are shown in Figs. 4A and 4B.

Then, we analyze the value of batch size using 8, 16 and 32, respectively, and experimental results on the two datasets are shown in Table 7. We can clearly find that the model achieves the best performance on both datasets when batch size equals 16. The possible reasons are speculated as follows: When batch size equals 8, it is small for the number of samples in the two datasets, and the model training is not only time-consuming but also difficult to converge, which leads to the underfitting of model. In a certain range, the increase of batch size is conducive to the stability of model convergence. However, when batch size equals 32, the model may fall into local minimum because it is too large, which leads to the deterioration of model generalization performance. Thus, we set the value of batch size to 16 in our experiments.

Value of k

To explore the impact of ANPs on model performance, we extract the top k ANPs for each image where k is set as each integer in [1, 10], and take values for them respectively to experiment. Figures 5A and 5B demonstrate the model performance of k value on the two datasets, respectively, and we can draw the following inferences. First, the model performance is poor without ANPs as image auxiliary information, which indicates that combining ANPs can improve the performance of TISRI. Second, the model performance shows a fluctuating upward trend as the number of ANPs increases and reaches the best state when k equals 5. However, the model performance no longer improves but shows a trend of decline when k is greater than 5. The possible reason is speculated as follows: The number of aspects involved in each text in the two datasets may not exceed 5, and when k is greater than it, image auxiliary information may introduce additional interference to the model. Therefore, we set the value of k to 5 in our experiments.

Value of λ

To investigate the effect of trade-off hyper-parameter λ that controls the auxiliary reconstruction loss contribution of IFAR layer on model performance, we set λ to a decimal number with an interval of 0.1 in the range of [0, 1] to experiment. Figures 6A and 6B demonstrate the model performance of λ value on the two datasets, respectively. The model performance shows a fluctuating upward trend as λ increases, which has a more obvious effect on the TWITTER-2017 dataset. When λ equals 0.8, the model performance reaches the best state, and then decreases gradually as λ increases. We speculate that the possible reason is that ANPs as image auxiliary information only serve to improve the semantic expression of image visual features. When the trade-off hyper-parameter λ exceeds a certain value, image auxiliary information plays a dominant role in image representation, but these ANPs may have semantic information of image recognition error, so the model will largely introduce additional interference when λ is too large and produce negative effect. Thus, we set the trade-off hyper-parameter λ to 0.8 in the error back propagation process.

Error analysis

On the basis of the above experiments, we further perform error analysis on TISRI to deepen our understanding of model performance. The error prediction reasons for TISRI are broadly divided into the following categories: (1) The ANPs are incorrect in image semantic recognition. For example, given a text “Petition to have Jessica Lange come back for American Horror Story season 6” with “Jessica Lange” as the aspect and an image of Jessica Lange with a cigarette in her mouth, but the ANPs produce words such as “laughing” and “funny” that completely hinder correct sentiment recognition, so TISRI introduces incorrect word information into the image representation adjustments during model training, which affects the model performance to some extent. (2) The aspect in text cannot find nouns with high similarity in ANPs. For example, given a text “This morning @ SheilaGCraft hosted a brunch amp poured into our WILD Women to honor them for their leadership in 2014 !” with “WILD Women” as the aspect and an image of a group of women dining together, but “WILD Women” is actually an organization name that cannot be represented intuitively in image, which causes ANPs to recognize some aspect-irrelevant nouns, so the image is also affected by these ANPs and cannot accurately express the sentiment semantics of aspect. (3) The model cannot recognize deeper sentiment in text and image. For example, given a text “RT @ NYRangers: OFFICIAL: Martin St . Louis announces retirement from the National Hockey League . # NYR” with “Martin” as the aspect and an image of Martin waving on the field with the crowd cheering, but our model can only identify semantic features on the surface of text and image combined with ANPs, and cannot feel the deeper sentiment of Martin’s unwillingness to leave the stadium, so this problem is also the difficulty and challenge for existing models to further intelligently identify sentiment in the future.