Speaker-Aware Interactive Graph Attention Network for Emotion Recognition in Conversation
Article No.: 256, Pages 1 - 18
Abstract
Recently, Emotion Recognition in Conversation (ERC) has attracted much attention and has become a hot topic in the field of natural language processing. Conversation is conducted in chronological order; current utterance is more likely influenced by nearby utterances. At the same time, speaker dependency also plays a core role in the conversation dynamic. The combined effect of the sequence-aware information and the speaker-aware information makes the emotion’s dynamic change. However, past works used simple information fusion methods to model the two kinds of information but ignored their interactive influence. Thus, we propose a novel method entitled SIGAT (Speaker-aware Interactive Graph Attention Network) to solve the problem. The core module is a mutual interactive module in which a dual-connection (self-connection and interact-connection) graph attention network is constructed. The advantage of SIGAT is modeling the speaker-aware and sequence-aware information in a unified graph and updating them simultaneously. In this way, we model the interactive influence of them and obtain the final representations, which have richer contextual clues. Experimental results on the four public datasets demonstrate that SIGAT outperforms the state-of-the-art models.
References
[1]
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation 42, 4 (2008), 335.
[2]
Zi Chai and Xiaojun Wan. 2020. Learning to ask more: Semi-autoregressive sequential question generation under dual-graph interaction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 225–237.
[3]
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. 2020. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, ICML, Vol. 119. 1725–1735.
[4]
Deepanway Ghosal, Navonil Majumder, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020. COSMIC: COmmonSense knowledge for eMotion Identification in Conversations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings. 2470–2481.
[5]
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 154–164.
[6]
Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann. 2018a. Icon: Interactive conversational memory network for multimodal emotion detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2594–2604.
[7]
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018b. Conversational memory network for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Vol. 1. 2122–2132.
[8]
Jingwen Hu, Yuchen Liu, Jinming Zhao, and Qin Jin. 2021. MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP (Volume 1: Long Papers). 5666–5675.
[9]
Taichi Ishiwatari, Yuki Yasuda, Taro Miyazaki, and Jun Goto. 2020. Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 7360–7370.
[10]
Yanbin Jiang, Huifang Ma, Yuhang Liu, Zhixin Li, and Liang Chang. 2021. Enhancing social recommendation via two-level graph attentional networks. Neurocomputing 449 (2021), 71–84.
[11]
Wenxiang Jiao, Michael R. Lyu, and Irwin King. 2020. Real-time emotion recognition via attention gated hierarchical memory network. In The 34th AAAI Conference on Artificial Intelligence, AAAI 2020. 8002–8009.
[12]
Wenxiang Jiao, Haiqin Yang, Irwin King, and Michael R. Lyu. 2019. HiGRU: Hierarchical gated recurrent units for utterance-level emotion recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 397–406.
[13]
Y. Kim. 2014. Convolutional neural networks for sentence classification. arXiv:1408.5882.
[14]
Thomas N. Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).
[15]
Joosung Lee and Wooin Lee. 2022. CoMPM: Context modeling with speaker’s pre-trained memory tracking for emotion recognition in conversation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022. Association for Computational Linguistics, 5669–5679.
[16]
Jingye Li, Donghong Ji, Fei Li, Meishan Zhang, and Yijiang Liu. 2020. HiTrans: A transformer-based context- and speaker-sensitive model for emotion detection in conversations. In Proceedings of the 28th International Conference on Computational Linguistics. 4190–4200.
[17]
Jiangnan Li, Zheng Lin, Peng Fu, Qingyi Si, and Weiping Wang. 2020. A hierarchical transformer with speaker modeling for emotion recognition in conversation. arXiv preprint arXiv:2012.14781 (2020).
[18]
Wei Li, Wei Shao, Shaoxiong Ji, and Erik Cambria. 2022. BiERU: Bidirectional emotional recurrent unit for conversational sentiment analysis. Neurocomputing 467 (2022), 73–82.
[19]
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Vol. 1. 986–995.
[20]
Zaijing Li, Fengxiao Tang, Ming Zhao, and Yusen Zhu. 2022. EmoCaps: Emotion capsule based model for conversational emotion recognition. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). 1610–1618.
[21]
Zheng Lian, Bin Liu, and Jianhua Tao. 2021. CTNet: Conversational transformer network for emotion recognition. IEEE/ACM Trans. Audio, Speech and Lang. Proc. (2021), 985–1000.
[22]
Hongmin Liu, Zhenzhen Xiao, Bin Fan, Hui Zeng, Yifan Zhang, and Guoquan Jiang. 2021. PrGCN: Probability prediction with graph convolutional network for person re-identification. Neurocomputing 423 (2021), 57–70.
[23]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
[24]
Yi-Ju Lu and Cheng-Te Li. 2020. GCAN: Graph-aware co-attention networks for explainable fake news detection on social media. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 505–514.
[25]
H. Ma, J. Wang, L. Qian, and H. Lin. 2021. HAN-ReGRU: Hierarchical attention network with residual gated recurrent unit for emotion recognition in conversation. Neural Computing and Applications 33, 3 (2021).
[26]
Yukun Ma, Khanh Linh Nguyen, Frank Z. Xing, and Erik Cambria. 2020. A survey on empathetic dialogue systems. Inf. Fusion 64 (2020), 50–70.
[27]
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. DialogueRNN: An attentive RNN for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 6818–6825.
[28]
Weizhi Nie, Rihao Chang, Minjie Ren, Yuting Su, and Anan Liu. 2021. I-GCN: Incremental graph convolution network for conversation emotion detection. IEEE Transactions on Multimedia (2021).
[29]
Soujanya Poria, Erik Cambria, and Alexander F. Gelbukh. 2015. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP, Lluís Màrquez, Chris Callison-Burch, Jian Su, Daniele Pighin, and Yuval Marton (Eds.). 2539–2544.
[30]
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long papers). 873–883.
[31]
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Rada Mihalcea, Gautam Naik, and Erik Cambria. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In ACL 2019: The 57th Annual Meeting of the Association for Computational Linguistics. 527–536.
[32]
Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019a. Emotion recognition in conversation: Research challenges, datasets, and recent advances. IEEE Access 7 (2019a), 100943–100953.
[33]
Libo Qin, Zhouyang Li, Wanxiang Che, Minheng Ni, and Ting Liu. 2021. Co-GAT: A co-interactive graph attention network for joint dialog act recognition and sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 13709–13717.
[34]
Minjie Ren, Xiangdong Huang, Wenhui Li, Dan Song, and Weizhi Nie. 2021. LR-GCN: Latent relation-aware graph convolutional network for conversational emotion recognition. IEEE Transactions on Multimedia (2021).
[35]
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In The Semantic Web — 15th International Conference, ESWC 2018, Vol. 10843. 593–607.
[36]
Weizhou Shen, Junqing Chen, Xiaojun Quan, and Zhixian Xie. 2021. DialogXL: All-in-one XLNet for multi-party conversation emotion recognition. In Proceedings of the AAAI Conference on Artificial Intelligence. 13789–13797.
[37]
Weizhou Shen, Siyue Wu, Yunyi Yang, and Xiaojun Quan. 2021. Directed acyclic graph network for conversational emotion recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 1551–1560.
[38]
Xiaohui Song, Liangjun Zang, Rong Zhang, Songlin Hu, and Longtao Huang. 2022. EmotionFlow: Capture the dialogue level emotion transitions. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022. IEEE, 8542–8546.
[39]
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In Proceedings of the 6th International Conference on Learning Representations, ICLR.
[40]
Songlong Xing, Sijie Mai, and Haifeng Hu. 2020. Adapted dynamic memory network for emotion recognition in conversation. IEEE Transactions on Affective Computing (2020).
[41]
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems. 5753–5763.
[42]
Sayyed M. Zahiri and Jinho D. Choi. 2017. Emotion detection on TV show transcripts with sequence-based convolutional neural networks. In AAAI Workshops. 44–52.
[43]
Peixiang Zhong, Di Wang, and Chunyan Miao. 2019. Knowledge-enriched transformer for emotion detection in textual conversations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 165–176.
[44]
Lixing Zhu, Gabriele Pergola, Lin Gui, Deyu Zhou, and Yulan He. 2021. Topic-driven and knowledge-aware transformer for dialogue emotion detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021 (Volume 1: Long Papers), 2021, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). 1571–1582.
Index Terms
- Speaker-Aware Interactive Graph Attention Network for Emotion Recognition in Conversation
Recommendations
Speaker-aware cognitive network with cross-modal attention for multimodal emotion recognition in conversation
AbstractEmotion recognition in conversation (ERC) has gained considerable attention owing to its extensive applications in the field of human-computer interaction. However, previous models have had certain limitations in exploring the potential emotional ...
Bi-stream graph learning based multimodal fusion for emotion recognition in conversation
AbstractEmotion Recognition in Conversation (ERC) is the process of automatically detecting and understanding emotions expressed in a conversation, which plays an important role in human–computer interaction. A conversation generates different modality ...
Highlights- A novel bi-stream graph learning framework is proposed.
- Capturing intra-modal contextual information by unimodal stream graph learning.
- Capturing inter -modal interaction information by cross-modal stream graph learning.
- ...
Dynamic interactive multiview memory network for emotion recognition in conversation
AbstractWhen available, multimodal data is key for enhanced emotion recognition in conversation. Text, audio, and video in dialogues can facilitate and complement each other in analyzing speakers’ emotions. However, it is very challenging to ...
Highlights- This work focuses on the dynamic interactions during information fusion process.
Comments
Information & Contributors
Information
Published In
December 2023
194 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 December 2023
Online AM: 07 November 2023
Accepted: 28 September 2023
Revised: 22 May 2023
Received: 29 March 2022
Published in TALLIP Volume 22, Issue 12
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Key Research Development Program of China
- Major Project of Anhui Province
- Anhui Province Key Research and Development Program
- General Programmer of the National Natural Science Foundation of China
- National Natural Science Foundation of China
- Major Projects of Science and Technology in Anhui Province
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 334Total Downloads
- Downloads (Last 12 months)334
- Downloads (Last 6 weeks)31
Reflects downloads up to 06 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in