poster

Timing and entrainment of multimodal backchanneling behavior for an embodied conversational agent

Authors:

Benjamin Inden,

Ipke WachsmuthAuthors Info & Claims

ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

Pages 181 - 188

https://doi.org/10.1145/2522848.2522890

Published: 09 December 2013 Publication History

Abstract

We report on an analysis of feedback behavior in an Active Listening Corpus as produced verbally, visually (head movement) and bimodally. The behavior is modeled in an embodied conversational agent and displayed in a conversation with a real human to human participants for perceptual evaluation. Five strategies for the timing of backchannels are compared: copying the timing of the original human listener, producing backchannels at randomly selected times, producing backchannels according to high level timing distributions relative to the interlocutor's utterance and pauses, or according to local entrainment to the interlocutors' vowels, or according to both. Human observers judge that models with global timing distributions miss less opportunities for backchanneling than random timing.

References

[1]

P. A. Barbosa. Incursões em torno do ritmo da fala. Pontes, Campinas, 2006.

[2]

S. Benus. Adaptation in turn-initiation. In A. E. et al., editor, Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces: Theoretical and Practical Issues., Lecture Notes in Computer Science {LNCS 6456}, pages 72--80. Springer Verlag, Berlin, Heidelberg, 2011.

Digital Library

[3]

S. Benus, A. Gravano, and J. Hirschberg. The prosody of backchannels in American English. In Proceedings of the 16th International Congress of Phonetic Sciences, pages 1065--1068, Saarbrücken, Germany, 2007.

[4]

P. Boersma and D. Weenink. Praat: Doing phonetics by computer. version 5.3.04, 2012.

[5]

M. Breen, L. Dilley, J. Kraemer, and E. Gibson. Inter-transcriber reliability for two systems of prosodic annotation: Tobi (Tones and Break Indices) and RaP (Rhythm and Pitch). Corpus Linguistics and Linguistic Theory, 8:277--312, 2010.

[6]

M. Bull and M. Aylett. An analysis of the timing of turn-taking in a corpus of goal-orientated dialogue. In Proceedings of ICSLP, pages 1775--1778, Sydney, 1998.

[7]

H. Buschmeier, Z. Malisz, M. Włodarczak, S. Kopp, and P. Wagner. 'Are you sure you're paying attention?' -- 'Uh-huh'. Communicating understanding as a marker of attentiveness. In Proceedings of Interspeech 2011, pages 2057--2060, Florence, Italy, 2011.

[8]

E. Couper-Kuhlen. English Speech Rhythm: Form and function in everyday verbal interaction. Benjamins, Amsterdam, 1993.

[9]

F. Cummins. Practice and performance in speech produced synchronously. Journal of Phonetics, 31:139--148, 2003.

[10]

F. Cummins and R. Port. Rhythmic constraints on stress timing in english. Journal of Phonetics, 26:145--171, 1998.

[11]

I. de Kok, R. Poppe, and D. Heylen. Iterative perceptual learning for social behavior synthesis. 2013.

[12]

D. Gibbon and F. R. Fernandes. Annotation-mining for rhythm model comparison in brazilian portuguese. In Proceedings of INTERSPEECH, 2005.

[13]

C. Heinrich and F. Schiel. Estimating speaking rate by means of rhythmicity parameters. In Proceedings of Interspeech, 2011.

[14]

L. Huang, L.-P. Morency, and J. Gratch. Learning backchannel prediction model from parasocial consensus sampling: A subjective evaluation. In Proceedings of the International Conference on Intelligent Virtual Agents, 2010.

Digital Library

[15]

L. Huang, L.-P. Morency, and J. Gratch. Parasocial consensus sampling: Combining multiple perspectives to learn virtual human behavior. In Proceeedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, 2010.

Digital Library

[16]

B. Inden, Z. Malisz, P. Wagner, and I. Wachsmuth. Rapid entrainment to spontaneous speech: A comparison of oscillator models. In Proceedings of the Cognitive Science Conference, 2012.

[17]

I. Jauk, P. Wagner, and I. Wachsmuth. Dynamic perception-production oscillation model in human-machine communication. In Proceedings of the International Conference on Multimodal Interaction, 2011.

Digital Library

[18]

G. Jonsdottir, J. Gratch, E. Fast, and K. Thórisson. Fluid semantic back-channel feedback in dialogue: Challenges and progress. In Proceedings of the International Conference on Interactive Virtual Agents, 2007.

Digital Library

[19]

M. Kawasaki, Y. Yamada, Y. Ushiku, E. Miyauchi, and Y. Yamaguchi. Inter-brain synchronization during coordination of speech rhythm in human-to-human social interaction. Nature, 3, 2013.

[20]

S. Kopp, J. Allwood, K. Grammer, E. Ahlsen, and T. Stockmeier. Modeling embodied feedback with virtual humans. In I. Wachsmuth and G. Knoblich, editors, Modeling communication with robots and virtual humans. Springer-Verlag Berlin Heidelberg, 2008.

Digital Library

[21]

S. Kopp, B. Jung, N. Leßmann, and I. Wachsmuth. Max -- a multimodal assistant in virtual reality construction. Künstliche Intelligenz, 4:11--17, 2003.

[22]

S. Kousidis, Z. Malisz, P. Wagner, and D. Schlangen. Exploring annotation of head gesture forms in spontaneous human interaction. In TiGeR 2013, Tilburg Gesture Research Meeting, 2013 (submitted).

[23]

R. M. Maatman, J. Gratch, and S. Marsella. Natural behavior of a listening agent. In Proceedings of the International Conference on Interactive Virtual Agents, pages 25--36, 2005.

Digital Library

[24]

L.-P. Morency, I. de Kok, and J. Gratch. Predicting listener backchannels: A probabilistic multimodal approach. In Proceedings of the 8th International Conference on Intelligent Virtual Agents, 2008.

Digital Library

[25]

Z. Néda, E. Ravasz, T. Vicsek, Y. Brechet, and A. L. Barabasi. The physics of rhythmic applause. Physical Review E, 61:6987, 2000.

[26]

R. Poppe, K. P. Truong, and D. Heylen. Bbackchannel: Quantity, type and timing matters. In Proccedings of Intelligent Virtual Agents, 2011.

Digital Library

[27]

R. Poppe, K. P. Truong, and D. Heylen. Perceptual evaluation of backchannel strategies for artificial listeners. Autonomous Agents and Multi-Agent Systems, 2013.

[28]

R. Poppe, K. P. Truong, D. Reidsma, and D. Heylen. Backchannel strategies for artificial listeners. In Proceedings of the Intelligent Virtual Agents Conference, 2010.

Digital Library

[29]

C. Richardson, R. Dale, and K. Schockley. Synchrony and swing in conversation: Coordination, temporal dynamics, and communication. In I. Wachsmuth, M. Lenzen, and G. Knoblich, editors, Embodied Communication in Humans and Machines. Oxford University Press, 2007.

[30]

F. Tamburini and P. Wagner. On automatic prominence detection for german. In Proceedings of INTERSPEECH, 2007.

[31]

K. P. Truong, R. Poppe, I. de Kok, and D. Heylen. A multimodal analysis of vocal and visual backchannels in spontaneous dialogs. In Proceedings of INTERSPEECH, 2011.

[32]

I. Wachsmuth. Communicative rhythms in gesture and speech. In Proceedings of the International Gesture Workshop on Gesture-Based Communication in Human-Computer Interaction, 1999.

Digital Library

[33]

P. Wagner, Z. Malisz, B. Inden, and I. Wachsmuth. Interaction phonology -- a temporal co-ordination component enabling communicative alignment. In Towards a New Theory of Communication. John Benjamins, to appear.

[34]

M. Wilson and T. P. Wilson. An oscillator model of the timing of turn-taking. Psychonomic Bulletin and Review, 12:957--968, 2005.

[35]

P. Wittenburg, H. Brugman, A. Russel, A. Klassmann, and H. Sloetjes. Elan: A professional framework for mulmultimodal research. In Proceedings of the fifth International Conference on Language Resources and Evaluation, 2006.

[36]

M. Włodarczak, H. Buschmeier, Z. Malisz, S. Kopp, and P. Wagner. Listener head gestures and verbal feedback expressions in a distraction task. In Proc. of the Interdisciplinary Workshop on Feedback Behaviours in Dialogue, 2012.

[37]

M. Włodarczak, J. Simko, and P. Wagner. Temporal entrainment in overlapped speech. In Proceedings of Interspeech, 2012.

[38]

Y. Zhang and J. Glass. Speech rhythm guided syllable nuclei detection. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2009.

Digital Library

Cited By

Onishi KTanaka HNakamura S(2023)Multimodal Voice Activity Prediction: Turn-taking Events Detection in Expert-Novice ConversationProceedings of the 11th International Conference on Human-Agent Interaction10.1145/3623809.3623837(13-21)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3623809.3623837
Seok SJeon TChae YKim CLim Y(2023)Explorative Study on the Non-verbal Backchannel Prediction Model for Human-Robot InteractionSocial Robotics10.1007/978-981-99-8715-3_23(264-275)Online publication date: 3-Dec-2023
https://doi.org/10.1007/978-981-99-8715-3_23
Andriella ATorras CAlenyà G(2023)Implications of Robot Backchannelling in Cognitive TherapySocial Robotics10.1007/978-3-031-24667-8_48(546-557)Online publication date: 1-Feb-2023
https://doi.org/10.1007/978-3-031-24667-8_48
Show More Cited By

Index Terms

Timing and entrainment of multimodal backchanneling behavior for an embodied conversational agent
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces

Recommendations

Entrainment in Human-Agent Text Communication
Agent Computing and Multi-Agent Systems

Non-verbal information such as utterance speed and switching pause create an impression of the speaker. If intelligent agents could handle such non-verbal information properly, the quality of interactions between agents and human users would improve. ...
Acoustic-Prosodic Entrainment and Rapport in Collaborative Learning Dialogues
MLA '14: Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge

In spoken dialogue analysis, the speech signal is a rich source of information. We explore in this paper how low level features of the speech signal, such as pitch, loudness, and speaking rate, can inform a model of student interaction in collaborative ...
Embodied conversational agents: computing and rendering realistic gaze patterns
PCM'06: Proceedings of the 7th Pacific Rim conference on Advances in Multimedia Information Processing

We describe here our efforts for modeling multimodal signals exchanged by interlocutors when interacting face-to-face. This data is then used to control embodied conversational agents able to engage into a realistic face-to-face interaction with human ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

December 2013

630 pages

ISBN:9781450321297

DOI:10.1145/2522848

General Chairs:
Julien Epps
The University of New South Wales, Australia
,
Fang Chen
National ICT Australia, Australia
,
Sharon Oviatt
Incaa Designs, USA
,
Kenji Mase
Nagoya University, Japan
,
Program Chairs:
Andrew Sears
Rochester Institute of Technology, USA
,
Kristiina Jokinen
University of Helsinki, Finland
,
Björn Schuller
Technische Universität München, Germany

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 December 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

ICMI '13

Sponsor:

SIGCHI

ICMI '13: 2013 International Conference on Multimodal Interaction

December 9 - 13, 2013

Sydney, Australia

Acceptance Rates

ICMI '13 Paper Acceptance Rate 49 of 133 submissions, 37%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
378
Total Downloads

Downloads (Last 12 months)40
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Onishi KTanaka HNakamura S(2023)Multimodal Voice Activity Prediction: Turn-taking Events Detection in Expert-Novice ConversationProceedings of the 11th International Conference on Human-Agent Interaction10.1145/3623809.3623837(13-21)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3623809.3623837
Seok SJeon TChae YKim CLim Y(2023)Explorative Study on the Non-verbal Backchannel Prediction Model for Human-Robot InteractionSocial Robotics10.1007/978-981-99-8715-3_23(264-275)Online publication date: 3-Dec-2023
https://doi.org/10.1007/978-981-99-8715-3_23
Andriella ATorras CAlenyà G(2023)Implications of Robot Backchannelling in Cognitive TherapySocial Robotics10.1007/978-3-031-24667-8_48(546-557)Online publication date: 1-Feb-2023
https://doi.org/10.1007/978-3-031-24667-8_48
Cumbal RKazzi DWinberg VEngwall OMartinho CDias JCampos JHeylen D(2022)Shaping unbalanced multi-party interactions through adaptive robot backchannelsProceedings of the 22nd ACM International Conference on Intelligent Virtual Agents10.1145/3514197.3549680(1-7)Online publication date: 6-Sep-2022
https://dl.acm.org/doi/10.1145/3514197.3549680
Hoegen RAneja DMcDuff DCzerwinski MPelachaud CMartin JBuschmeier HLucas GKopp S(2019)An End-to-End Conversational Style Matching AgentProceedings of the 19th ACM International Conference on Intelligent Virtual Agents10.1145/3308532.3329473(111-118)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3308532.3329473
Hussain NErzin ESezgin TYemez Y(2019)Batch Recurrent Q-Learning for Backchannel Generation Towards Engaging Agents2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)10.1109/ACII.2019.8925443(1-7)Online publication date: Sep-2019
https://doi.org/10.1109/ACII.2019.8925443
Lala DNishida T(2017)A data-driven passing interaction model for embodied basketball agentsJournal of Intelligent Information Systems10.1007/s10844-015-0386-z48:1(27-60)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1007/s10844-015-0386-z
Oertel CGustafson JBlack ABöck RBonin FCampbell NPoppe R(2016)On data driven parametric backchannel synthesis for expressing attentiveness in conversational agentsProceedings of the Workshop on Multimodal Analyses enabling Artificial Agents in Human-Machine Interaction10.1145/3011263.3011272(43-47)Online publication date: 12-Nov-2016
https://dl.acm.org/doi/10.1145/3011263.3011272
Malisz ZWłodarczak MBuschmeier HSkubisz JKopp SWagner P(2016)The ALICO corpusLanguage Resources and Evaluation10.1007/s10579-016-9355-650:2(411-442)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1007/s10579-016-9355-6
Lubold NPon-Barry HWalker E(2015)Naturalness and rapport in a pitch adaptive learning companion2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)10.1109/ASRU.2015.7404781(103-110)Online publication date: Dec-2015
https://doi.org/10.1109/ASRU.2015.7404781

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten