research-article

Speech-based Continuous Emotion Prediction by Learning Perception Responses related to Salient Events: A Study based on Vocal Affect Bursts and Cross-Cultural Affect in AVEC 2018

Authors:

Kalani Wataraka Gamage,

Vidhyasaharan Sethu,

Eliathamby AmbikairajahAuthors Info & Claims

AVEC'18: Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop

Pages 47 - 55

https://doi.org/10.1145/3266302.3266314

Published: 15 October 2018 Publication History

Abstract

This paper presents a novel framework for speech-based continuous emotion prediction. The proposed model characterises the perceived emotion estimation as time-invariant responses to salient events. Then arousal and valence variation over time is modelded as the ouput of a parallel array of time-invariant filters where each filter represents a salient event in this context, and the impulse response of the filter represents the learned perception emotion response. The proposed model is evaluted by considering vocal affect bursts/non-verbal vocal gestures as salient event candidates. The proposed model is validated based on the development dataset of AVEC 2018 challenge development dataset and achieves the highest accuracy of valence prediction among single modal methods based on speech or speech-transcript. We tested this model on cross-cultural settings provided by AVEC 2018 challenge test set, and the model performs reasonably well for an unseen culture as well and outperform speech-based baselines. Further we explore inclusion of interlocutor related cues to the proposed model and decision level fusion with existing features. Since the proposed model was evaluated solely based on laughter and slight laughter affect bursts which were nominated as salient by proposed saliency constrains of the model, the results presented highlight the significance of aforementioned gestures in human emotion expression and perception

References

[1]

Schuller, B., Batliner, A.: Computational paralinguistics: emotion, affect and personality in speech and language processing' (John Wiley & Sons, 2013)

Digital Library

[2]

Ringeval, F.: AVEC 2017 Real-life Depression, and Affect Recognition Workshop and Challenge' Proceedings of the 7th International Workshop on Audio/Visual Emotion Challenge, Mountain View, CA), USA, October 2017, pp.3--9

Digital Library

[3]

Chen, S., Jin, Q., Zhao, J., Wang, S.: 'Multimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition', Proceedings of the 7th International Workshop on Audio/Visual Emotion Challenge, Mountain View, CA), USA, October 2017, pp.19--26

Digital Library

[4]

Scherer K. R., "Affect bursts," in Emotions?: essays on emotion theory, S. H. M. van. Goozen, N. E. van de. Poll, and J. A. Sergeant, Eds. L. Erlbaum, 1994, pp. 161--193.

[5]

C. N. Anagnostopoulos, T. Iliou, and I. Giannoukos, "Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011," Artif. Intell. Rev., vol. 43, no. 2, pp. 155--177, 2012.

Digital Library

[6]

K. W. Gamage, V. Sethu and E. Ambikairajah, "Modeling variable length phoneme sequences - A step towards linguistic information for speech emotion recognition in wider world," 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), San Antonio, TX, 2017, pp. 518--523.

[7]

The SEWA project: "SEWA database - Home', https://db.sewaproject.eu/, accessed December 2017

[8]

P. C. Ellsworth and K. R. Scherer, "APPRAISAL PROCESSES IN EMOTION," Handb. Affect. Sci., pp. 572--595, 2003.

[9]

A. Moors, P. C. Ellsworth, K. R. Scherer, and N. H. Frijda, "Appraisal Theories of Emotion: State of the Art and Future Development," Emot. Rev., vol. 5, no. 2, pp. 119--124, Apr. 2013.

[10]

Scherer, K. R.: "Vocal markers of emotion: Comparing induction and acting elicitation,' Comput. Speech Lang., 2013, 27, (1), pp. 40--58.

Digital Library

[11]

M. Schröder, "Experimental study of affect bursts," Speech Commun., vol. 40, no. 1--2, pp. 99--116, 2003.

Digital Library

[12]

K. R. Scherer, "On the symbolic functions of vocal affect expression," J. Lang. Soc. Psychol., vol. 7, no. 2, pp. 79--100, 1988.

[13]

J. D. Moore, L. Tian, and C. Lai, "Word-Level Emotion Recognition Using High-Level Features," Comput. Linguist. Intell. Text Process. Lect. Notes Comput. Sci., vol. 8404, pp. 17--31, 2014.

Digital Library

[14]

L. Tian, J. D. Moore, and C. Lai, "Emotion recognition in spontaneous and acted dialogues," 2015 Int. Conf. Affect. Comput. Intell. Interact. ACII 2015, pp. 698--704, 2015.

Digital Library

[15]

K. R. Scherer, "Affect Bursts as Evolutionary Precursors of Speech and Music," in Stephen J. Gould: The Scientific Legacy, 2013.

[16]

Ringeval, F., Sonderegger, A., Sauer, J., Lalanne, D.: "Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions'. 2013 10th IEEE Int. Conf. Work. Autom. Face Gesture Recognition, FG 2013, Shanghai, China, 2013.

[17]

"Cronbach's alpha", En.wikipedia.org, 2018. {Online}. Available: https://en.wikipedi {1}"Cronbach's alpha", En.wikipedia.org, 2018. {Online}. Available: https://en.wikipedia.org/wiki/Cronbach%27s_alpha. {Accessed: 14- Jun- 2018}.a.org/wiki/Cronbach%27s_alpha. {Accessed: 14- Jun- 2018}.

[18]

Ritter, N. (2010). "Understanding a widely misunderstood statistic: Cronbach's alpha". Paper presented at Southwestern Educational Research Association (SERA) Conference 2010: New Orleans, LA

[19]

F. Ringeval et al., "AVEC 2018 Workshop and Challenge: Bipolar Disorder and Cross-Cultural Affect Recognition," Proc. 8th Annu. Work. Audio/Visual Emot. Chall. - AVEC '18, 2018.

Digital Library

[20]

F. Ringeval et al., "Avec 2017," Proc. 7th Annu. Work. Audio/Visual Emot. Chall. - AVEC '17, pp. 3--9, 2017.

Digital Library

[21]

C. C. Lee, C. Busso, S. Lee, and S. Narayanan, "Modeling mutual influence of interlocutor emotion states in dyadic spoken interactions," Proc. Annu. Conf. Int. Speech Commun. Assoc. INTERSPEECH, pp. 1983--1986, 2009.

[22]

A. Metallinou, A. Katsamanis, and S. Narayanan, "A hierarchical framework for modeling multimodality and emotional evolution in affective dialogs," in 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2012, pp. 2401--2404.

[23]

D. A. Sauter, F. Eisner, P. Ekman, and S. K. Scott, "Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations," Proc. Natl. Acad. Sci., vol. 107, no. 6, pp. 2408--2412, 2010.

Cited By

Abu Salih MAbargil MBadarneh Sklein Selle NIrani MAtzil S(2023)Evidence for cultural differences in affect during mother–infant interactionsScientific Reports10.1038/s41598-023-31907-y13:1Online publication date: 24-Mar-2023
https://doi.org/10.1038/s41598-023-31907-y
Ong DWu ZZhi-Xuan TReddan MKahhale IMattek AZaki J(2021)Modeling Emotion in Complex Stories: The Stanford Emotional Narratives DatasetIEEE Transactions on Affective Computing10.1109/TAFFC.2019.295594912:3(579-594)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TAFFC.2019.2955949
Khorram SMcInnis MProvost E(2021)Jointly Aligning and Predicting Continuous Emotion AnnotationsIEEE Transactions on Affective Computing10.1109/TAFFC.2019.291704712:4(1069-1083)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TAFFC.2019.2917047
Show More Cited By

Index Terms

Speech-based Continuous Emotion Prediction by Learning Perception Responses related to Salient Events: A Study based on Vocal Affect Bursts and Cross-Cultural Affect in AVEC 2018
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning

Recommendations

Joint Discrete and Continuous Emotion Prediction Using Ensemble and End-to-End Approaches
ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

This paper presents a novel approach in continuous emotion prediction that characterizes dimensional emotion labels jointly with continuous and discretized representations. Continuous emotion labels can capture subtle emotion variations, but their ...
Emotion Recognition in Continuous Mandarin Chinese Speech: Visualizing Emotional Expression from Continuous Speech in a 2D Emotional Space
Speech Emotion Perception by Human and Machine
Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction

The human speech contains and reflects information about the emotional state of the speaker. The importance of research of emotions is increasing in telematics, information technologies and even in health services. The research of the mean acoustical ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AVEC'18: Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop

October 2018

113 pages

ISBN:9781450359832

DOI:10.1145/3266302

General Chairs:
Fabien Ringeval
Grenoble Alps University, France
,
Björn Schuller
University of Augsburg/Imperial College London, Germany/UK
,
Michel Valstar
University of Nottingham, UK
,
Roddy Cowie
Queen's University Belfast, UK
,
Maja Pantic
Imperial College London/Twente University, UK/The Netherlands

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '18

Sponsor:

SIGMM

MM '18: ACM Multimedia Conference

October 22, 2018

Seoul, Republic of Korea

Acceptance Rates

AVEC'18 Paper Acceptance Rate 11 of 23 submissions, 48%;

Overall Acceptance Rate 52 of 98 submissions, 53%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
337
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Abu Salih MAbargil MBadarneh Sklein Selle NIrani MAtzil S(2023)Evidence for cultural differences in affect during mother–infant interactionsScientific Reports10.1038/s41598-023-31907-y13:1Online publication date: 24-Mar-2023
https://doi.org/10.1038/s41598-023-31907-y
Ong DWu ZZhi-Xuan TReddan MKahhale IMattek AZaki J(2021)Modeling Emotion in Complex Stories: The Stanford Emotional Narratives DatasetIEEE Transactions on Affective Computing10.1109/TAFFC.2019.295594912:3(579-594)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TAFFC.2019.2955949
Khorram SMcInnis MProvost E(2021)Jointly Aligning and Predicting Continuous Emotion AnnotationsIEEE Transactions on Affective Computing10.1109/TAFFC.2019.291704712:4(1069-1083)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TAFFC.2019.2917047
Dong YYang X(2021)Affect-salient event sequence modelling for continuous speech emotion recognitionNeurocomputing10.1016/j.neucom.2021.06.036458:C(246-258)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1016/j.neucom.2021.06.036
Dong YYang X(2020)Affect-Salient Event Sequences Modelling for Continuous Speech Emotion Recognition Using Connectionist Temporal Classification2020 IEEE 5th International Conference on Signal and Image Processing (ICSIP)10.1109/ICSIP49896.2020.9339383(773-778)Online publication date: 23-Oct-2020
https://doi.org/10.1109/ICSIP49896.2020.9339383
Kaya HFedotov DDresvyanskiy DDoyran MMamontov DMarkitantov MAkdag Salah AKavcar EKarpov ASalah ARingeval FSchuller BValstar MCummins NCowie RPantic M(2019)Predicting Depression and Emotions in the Cross-roads of Cultures, Para-linguistics, and Non-linguisticsProceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop10.1145/3347320.3357691(27-35)Online publication date: 15-Oct-2019
https://dl.acm.org/doi/10.1145/3347320.3357691

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents