research-article

MEDIC: A Multimodal Empathy Dataset in Counseling

Authors:

Shangfei WangAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 6054 - 6062

https://doi.org/10.1145/3581783.3612346

Published: 27 October 2023 Publication History

Abstract

Although empathic interaction between counselor and client is fundamental to success in the psychotherapeutic process, there are currently few datasets to aid a computational approach to empathy understanding. In this paper, we construct a multimodal empathy dataset collected from face-to-face psychological counseling sessions. The dataset consists of 771 video clips. We also propose three labels (i.e., expression of experience, emotional reaction, and cognitive reaction) to describe the degree of empathy between counselors and their clients. Expression of experience describes whether the client has expressed experiences that can trigger empathy, and emotional and cognitive reactions indicate the counselor's empathic reactions. As an elementary assessment of the usability of the constructed multimodal empathy dataset, an interrater reliability analysis of annotators' subjective evaluations for video clips is conducted using the intraclass correlation coefficient and Fleiss' Kappa. Results prove that our data annotation is reliable. Furthermore, we conduct empathy prediction using three typical methods, including the tensor fusion network, the sentimental words aware fusion network, and a simple concatenation model. The experimental results show that empathy can be well predicted on our dataset. Our dataset is available for research purposes.

References

[1]

Firoj Alam, Morena Danieli, and Giuseppe Riccardi. 2018. Annotating and modeling empathy in spoken conversations. Computer Speech & Language 50 (2018), 40--61.

Digital Library

[2]

Paul C. Amrhein, William R. Miller, Theresa B. Moyers, and Denise Ernst. 2008. Manual for the Motivational Interviewing Skill Code (MISC). Motivational Interviewing Skill Code v. 2.1 (2008).

[3]

Godfrey T Barrett-Lennard. 1981. The empathy cycle: Refinement of a nuclear concept. Journal of counseling psychology 28, 2 (1981), 91.

[4]

Godfrey T. Barrett-Lennard. 2011. The phases and focus of empathy. Barrett-Lennard, G.T(2011) The phases and focus of empathy. British Journal of Medical Psychology, 66 (1). pp. 3--14. (2011).

[5]

Pablo Barros, Nikhil Churamani, Angelica Lim, and Stefan Wermter. 2019. The omg-empathy dataset: Evaluating the impact of affective behavior in storytelling. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1--7.

[6]

Arthur C Bohart and Leslie S Greenberg. 1997. Empathy: Where are we and where do we go from here? (1997).

[7]

Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and Joao Sedoc. 2018. Modeling empathy and distress in reaction to news stories. arXiv preprint arXiv:1808.10399 (2018).

[8]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7291--7299.

[9]

Minping Chen and Xia Li. 2020. SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis. international conference on computational linguistics (2020).

[10]

Matthew Davis, Monique G. Davis, M. Edward Davis, Mark E. Davis, Mark Mitchell Davis, MM Davis, Mdp Davis, Frederick B. Davis, H. Davis, and I. W. Davis. 1980. A multidimensional approach to individual differences in empathy. JSAS Catalog of Selected Documents in Psychology (1980).

[11]

Frans B. M. de Waal. 2008. Putting the Altruism Back into Altruism: The Evolution of Empathy. Annual Review of Psychology (2008).

[12]

Robert Elliott, Arthur C Bohart, Jeanne C Watson, and Leslie S Greenberg. 2011. Empathy. Psychotherapy 48, 1 (2011), 43.

[13]

Robert Elliott, Leslie S Greenberg, Jeanne C Watson, Ladislav Timulak, and Elizabeth Freire. 2013. Research on humanistic-experiential psychotherapies. (2013).

[14]

Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76, 5 (1971), 378.

[15]

James Gibson, Dogan Can, Bo Xiao, Zac E. Imel, David C. Atkins, Panayiotis G. Georgiou, and Shrikanth S. Narayanan. 2016. A Deep Learning Approach to Modeling Empathy in Addiction Counseling. conference of the international speech communication association (2016).

[16]

Emily A. Holmes, Ata Ghaderi, Catherine J. Harmer, Paul Ramchandani, Pim Cuijpers, Anthony P. Morrison, Jonathan P. Roiser, Claudi L H Bockting, Rory C. O'Connor, Roz Shafran, Michelle L. Moulds, and Michelle G. Craske. 2018. The Lancet Psychiatry Commission on psychological treatments research in tomorrow's science. The Lancet Psychiatry (2018).

[17]

Terry K Koo and Mae Y Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine 15, 2 (2016), 155--163.

[18]

Shiro Kumano, Kazuhiro Otsuka, Dan Mikami, and Junji Yamato. 2011. Analyzing empathetic interactions based on the probabilistic modeling of the co-occurrence patterns of facial expressions in group meetings. In 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG). IEEE, 43--50.

[19]

Michael J Lambert, Allen E Bergin, and SL Garfield. 1994. The effectiveness of psychotherapy. Encyclopedia of psychotherapy 1 (1994), 709--714.

[20]

Louise Liddon, Roger Kingerlee, and John A Barry. 2018. Gender differences in preferences for psychological treatment, coping strategies, and triggers to help-seeking. British Journal of Clinical Psychology 57, 1 (2018), 42--58.

[21]

Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, Vol. 8. Citeseer, 18--25.

[22]

John McLeod. 2003. Doing counselling research. Sage.

[23]

William R Miller, Theresa B Moyers, Denise Ernst, and Paul Amrhein. 2003. Manual for the motivational interviewing skill code (MISC). Unpublished manuscript. Albuquerque: Center on Alcoholism, Substance Abuse and Addictions, University of New Mexico (2003).

[24]

TB Moyers, T Martin, JK Manuel, WR Miller, and D Ernst. 2007. Revised global scales: motivational interviewing treatment integrity 3.0 (MITI 3.0). University of New Mexico, Center on Alcoholism, Substance Abuse and Addictions (CASAA) 28 (2007).

[25]

T B Moyers, J K Manuel, & D Ernst, Lisa Hagen Glynn, Christiana Fortini, Ellis Baron, Hans Bertens, Saskia Boom, Angelita Casanovas, Jos Dobber, Jannet De Jonge, Maarten Merkx, Janet Murris, Hetty De Laat, and Riëtta Oberink. 2022. Motivational Interviewing Treatment Integrity Coding Manual 4.2.1 (MITI 4.2.1).

[26]

Theresa B Moyers, Tim Martin, Jennifer K Manuel, William R Miller, and D Ernst. 2003. The motivational interviewing treatment integrity (MITI) code. Unpublished manuscript (2003).

[27]

Verónica Pérez-Rosas, Rada Mihalcea, Ken Resnicow, Satinder Singh, and Lawrence C. An. 2017. Understanding and Predicting Empathic Behavior in Counseling Therapy. meeting of the association for computational linguistics (2017).

[28]

Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset. meeting of the association for computational linguistics (2019).

[29]

Catherine M Reich, Jeffrey S Berman, Rick Dale, and Heidi M Levitt. 2014. Vocal synchrony in psychotherapy. Journal of Social and Clinical Psychology 33, 5 (2014), 481.

[30]

Robert L Selman. 1980. The growth of interpersonal understanding: Developmental and clinical analyses. Academy Press.

[31]

Ashish Sharma, Adam S. Miner, David C. Atkins, and Tim Althoff. 2020. A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support. empirical methods in natural language processing (2020).

[32]

Tania Singer and Claus Lamm. 2009. The social neuroscience of empathy. Annals of the New York Academy of Sciences (2009).

[33]

Bo Xiao, Dogan Can, Panayiotis G. Georgiou, David C. Atkins, and Shrikanth S. Narayanan. 2012. Analyzing the language of therapist empathy in Motivational Interview based psychotherapy. asia pacific signal and information processing association annual summit and conference (2012).

[34]

Bo Xiao, Zac E Imel, Panayiotis G Georgiou, David C Atkins, and Shrikanth S Narayanan. 2015. "Rate my therapist": automated detection of empathy in drug and alcohol counseling via speech and language processing. PloS one 10, 12 (2015), e0143055.

[35]

Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor Fusion Network for Multimodal Sentiment Analysis. empirical methods in natural language processing (2017).

Cited By

Chen TShen YChen XZhang L(2024)PsyChatbot: A Psychological Counseling Agent Towards Depressed Chinese Population Based on Cognitive Behavioural TherapyACM Transactions on Asian and Low-Resource Language Information Processing10.1145/3676962Online publication date: 5-Jul-2024
https://dl.acm.org/doi/10.1145/3676962
Feng TWang XHan FZhang LZhu WCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)U2UData: A Large-scale Cooperative Perception Dataset for Swarm UAVs Autonomous FlightProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681151(7600-7608)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681151
Montiel-Vázquez ECruz CUresti JGomez R(2024)EmpatheticExchanges: Toward Understanding the Cues for Empathy in Dyadic ConversationsIEEE Access10.1109/ACCESS.2024.352000012(195097-195110)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3520000

Index Terms

MEDIC: A Multimodal Empathy Dataset in Counseling
1. Computing methodologies
  1. Artificial intelligence

Recommendations

Effects of (in)accurate empathy and situational valence on attitudes towards robots
HRI '10: Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction

Empathy has great potential in human-robot interaction. However, the challenging nature of assessing the user's emotional state points to the importance of also understanding the effects of empathic behaviours incongruent with users' affective ...
Empathy for Everyone?: The Effect of Age When Evaluating a Virtual Agent
HAI '18: Proceedings of the 6th International Conference on Human-Agent Interaction

The present study investigated the role of age in the perception of emotional nonverbal behaviors of a virtual assistant in a 2 (seniors vs young participants) x 2 (happy vs sad situations) x 4 (three emotional nonverbal behaviors related to each ...
Modeling empathy for a virtual human: how, when and to what extent?
AAMAS '11: The 10th International Conference on Autonomous Agents and Multiagent Systems - Volume 3

Going along the questions of how, when and to what extent does empathy arise in humans, we propose an approach to model empathy for EMMA -- an Empathic MultiModal Agent -- based on three processing steps: First, the Empathy Mechanism by which an ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

project from Anhui Science Technology Agency
National Natural Science Foundation of China

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
174
Total Downloads

Downloads (Last 12 months)121
Downloads (Last 6 weeks)11

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen TShen YChen XZhang L(2024)PsyChatbot: A Psychological Counseling Agent Towards Depressed Chinese Population Based on Cognitive Behavioural TherapyACM Transactions on Asian and Low-Resource Language Information Processing10.1145/3676962Online publication date: 5-Jul-2024
https://dl.acm.org/doi/10.1145/3676962
Feng TWang XHan FZhang LZhu WCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)U2UData: A Large-scale Cooperative Perception Dataset for Swarm UAVs Autonomous FlightProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681151(7600-7608)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681151
Montiel-Vázquez ECruz CUresti JGomez R(2024)EmpatheticExchanges: Toward Understanding the Cues for Empathy in Dyadic ConversationsIEEE Access10.1109/ACCESS.2024.352000012(195097-195110)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3520000

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents