TriEMO:基于模态对对比学习图网络的三重语义对齐的多模态情感识别方法
TriEMO: Triple semantic alignment based on modality pair contrastive learning graph network for multimodal emotion recognition
Multimodal Emotion Recognition in Conversations (MERC) aims to identify the emotional states expressed in utterances by modeling the complex interactions of textual, acoustic, and visual modalities within dynamic dialogue contexts. While recent advances leverage graph neural networks and contrastive learning to improve semantic alignment and representation robustness, they often suffer from rigid cross-modal alignment, which suppresses modality-specific nuances and degrades performance in dynamic or semantically ambiguous settings with noise. To address these limitations, we propose TriEMO, a novel Triple Semantic Alignment framework based on Modality Pair Contrastive Learning Graph Network for MERC. TriEMO introduces three key modules to enhance cross-modal representation and alignment: Modality Pair Data Augmentation (MPDA), which constructs contrastive learning units based on modality pairs to enrich pairwise semantic interactions; Rigid Modality Pair-based Contrastive Learning (RMPCL) combined with Modality Pair Hypergraph Learning (MPHL) and a Virtual Emotional Modality (VEM), which serves as a semantic anchor to coordinate information transmission both within temporal segments and across modality pairs, ensuring robust semantic alignment; and Triple Adversarial Contrastive Learning (TACL), which mitigates over-alignment in deep graph propagation by pulling modality pairs closer while pushing single-instance modality pairs away, thereby preserving discriminative features. Finally, a lightweight linear classifier predicts the emotion for each utterance. Extensive experiments on the IEMOCAP and MELD benchmarks demonstrate the effectiveness of TriEMO. On IEMOCAP, TriEMO outperforms state-of-the-art graph learning-based methods by 2.21% weighted F1 and contrastive learning-based methods by 0.69%, with a notable improvement of 2.13% on the Neutral category. On MELD, it achieves 3.18% weighted F1 improvement over the strongest graph learning baselines and 0.97% over contrastive learning approaches, with a 2.45% gain on the challenging Disgust emotion. These consistent improvements highlight TriEMO’s superior robustness and generalization for multimodal emotion recognition in conversational settings.
2026.07.21
详情