Zhao, X; Chen, Y; Li, W; Gao, L; Tang, B; (2022) MAG+: An Extended Multimodal Adaptation Gate for Multimodal Sentiment Analysis. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. (pp. pp. 4753-4757). Institute of Electrical and Electronics Engineers (IEEE)

Preview

Text MAG___An_EXTENDED_Multimodal_Adaptation_Gate_for_Multimodal_Sentiment_Analysis.pdf - Accepted Version Download (1MB) | Preview

Abstract

Human multimodal sentiment analysis is a challenging task that devotes to extract and integrate information from multiple resources, such as language, acoustic and visual information. Recently, multimodal adaptation gate (MAG), an attachment to transformer-based pre-trained language representation models, such as BERT and XLNet, has shown state-of-the-art performance on multimodal sentiment analysis. MAG only uses a 1-layer network to fuse multimodal information directly, and does not pay attention to relationships among different modalities. In this paper, we propose an extended MAG, called MAG+, to reinforce multimodal fusion. MAG+ contains two modules: multi-layer MAGs with modality reinforcement (M3R) and Adaptive Layer Aggregation (ALA). In the MAG with modality reinforcement of M3R, each modality is reinforced by all other modalities via crossmodal attention at first, and then all modalities are fused via MAG. The ALA module leverages the multimodal representations at low and high levels as the final multimodal representation. Similar to MAG, MAG+ is also attached to BERT and XLNet. Experimental results on two widely used datasets demonstrate the efficacy of our proposed MAG+.

Download activity - last month

Export as XMLJSONCSV

Download activity - last 12 months

Export as JSONXMLCSV

Downloads by country - last 12 months

Export as JSONXMLCSV

Archive Staff Only

View Item