Abstract

Abstract

Multimodal Emotion Recognition in Conversations (MERC) aims to identify the emotional states expressed in utterances by modeling the complex interactions of textual, acoustic, and visual modalities within dynamic dialogue contexts. While recent advances leverage graph neural networks and contrastive learning to improve semantic alignment and representation robustness, they often suffer from rigid cross-modal alignment, which suppresses modality-specific nuances and degrades performance in dynamic or semantically ambiguous settings with noise. To address these limitations, we propose TriEMO, a novel Triple Semantic Alignment framework based on Modality Pair Contrastive Learning Graph Network for MERC. TriEMO introduces three key modules to enhance cross-modal representation and alignment: Modality Pair Data Augmentation (MPDA), which constructs contrastive learning units based on modality pairs to enrich pairwise semantic interactions; Rigid Modality Pair-based Contrastive Learning (RMPCL) combined with Modality Pair Hypergraph Learning (MPHL) and a Virtual Emotional Modality (VEM), which serves as a semantic anchor to coordinate information transmission both within temporal segments and across modality pairs, ensuring robust semantic alignment; and Triple Adversarial Contrastive Learning (TACL), which mitigates over-alignment in deep graph propagation by pulling modality pairs closer while pushing single-instance modality pairs away, thereby preserving discriminative features. Finally, a lightweight linear classifier predicts the emotion for each utterance. Extensive experiments on the IEMOCAP and MELD benchmarks demonstrate the effectiveness of TriEMO. On IEMOCAP, TriEMO outperforms state-of-the-art graph learning-based methods by 2.21% weighted F1 and contrastive learning-based methods by 0.69%, with a notable improvement of 2.13% on the Neutral category. On MELD, it achieves 3.18% weighted F1 improvement over the strongest graph learning baselines and 0.97% over contrastive learning approaches, with a 2.45% gain on the challenging Disgust emotion. These consistent improvements highlight TriEMO’s superior robustness and generalization for multimodal emotion recognition in conversational settings.

Fig. 1. (a) Modality-pair-based balanced alignment aims to fuse every modality pair and mitigate excessive modality specificity while enhancing cross-modal feature interaction during the semantic alignment process, as formulated in Eqs. (10) and (14). (b) The virtual modality semantic anchor reinforces a unified cross-modal semantic center during contrastive learning and coordinates information transmission in the graph learning process, as described in Eqs. (20) and (22). (c) Network-depth adversarial contrastive learning promotes semantic discriminability across different stages of the network, preventing over-alignment as depth increases, as defined in Eq. (23).

Fig. 2. The overview of our proposed TriEMO framework can be divided into seven components. First, initial features are extracted using a pre-trained backbone. The Modality Pair Interaction-based Data Augmentation (MPDA) module coarsely enriches the semantic representations of paired modalities. The Rigid Modality Pair-based Contrastive Learning (RMPCL) Alignment module further refines this process by imposing strict modality balance constraints. Subsequently, a Virtual Emotional Modality (VEM) is introduced to serve as a semantic anchor, while performing Modality Pair based Hypergraph Learning (MPHL) both within temporal segments and between modality pairs to ensure modality-level consistency. Next, a VEM-centered Triple Adversarial Contrastive Learning (TACL) Modulation is employed to adversarially align semantic representations across modalities and mitigate the loss of modality-specific distinctiveness caused by deep graph convolution. Finally, a lightweight linear classification layer is used to predict the emotion for each utterance.

https://doi.org/10.1016/j.inffus.2026.104289