poster

Multimodal end-of-turn prediction in multi-party meetings

Authors:

Dirk HeylenAuthors Info & Claims

ICMI-MLMI '09: Proceedings of the 2009 international conference on Multimodal interfaces

Pages 91 - 98

https://doi.org/10.1145/1647314.1647332

Published: 02 November 2009 Publication History

Abstract

One of many skills required to engage properly in a conversation is to know the appropiate use of the rules of engagement. In order to engage properly in a conversation, a virtual human or robot should, for instance, be able to know when it is being addressed or when the speaker is about to hand over the turn. The paper presents a multimodal approach to end-of-speaker-turn prediction using sequential probabilistic models (Conditional Random Fields) to learn a model from observations of real-life multi-party meetings. Although the results are not as good as expected, we provide insight into which modalities are important when taking a multimodal approach to the problem based on literature and our own results.

References

[1]

M. Argyle and M. Cook. Gaze and mutual gaze. Cambridge University Press, London, United Kingdom, 1976.

[2]

M. Atterer, T. Baumann, and D. Schlangen. Towards incremental end-of-utterance detection in dialogue systems. In Proceedings of International Conference on Computational Linguistics, 2008.

[3]

P. Barkhuysen, E. Krahmer, and M. Swerts. The interplay between auditory and visual cues for end-of-utterance detection. Journal of Acoustical Society of America, 123(1):354 -- 365, 2008.

[4]

P. Boersma and V. van Heuven. Speak and unspeak with praat. Glot International, 5(9-10):341--347, November 2001.

[5]

J. Cassell, Y. I. Nakano, T. W. Bickmore, C. L. Sidner, and C. Rich. Non-verbal cues for discourse structure. In ACL '01: Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, pages 114--123, Morristown, NJ, USA, 2001. Association for Computational Linguistics.

Digital Library

[6]

J. Cassell, J. Sullivan, S. Prevost, and E. F. Churchill. Embodied Conversational Agents. MIT Press, Cambridge Massachusetts, London England, 2000.

[7]

J. Cassell, O. E. Torres, and S. Prevost. Turn taking vs. discourse structure: How best to model multimodal conversation. In Machine Conversations, pages 143--154. Kluwer, 1998.

[8]

J. de Ruiter, H. Mitterer, and N. Enfield. Projecting the end of a speaker's turn: A cognitive cornerstone of conversation. Language, 82(3):515 -- 535, 2006.

[9]

S. Duncan. Some signals and rules for taking speaking turns in conversations. Journal of Personality and Social Psychology, 23(2):283 -- 292, 1972.

[10]

S. Duncan and G. Niederehe. On signalling that it's your turn to speak. Journal of Experimental Social Psychology, 10:234--247, 1974.

[11]

O. Fuentes, D. Vera, and T. Solorio. A filter-based approach to detect end-of-utterances from prosody in dialog systems. In HLT-NAACL (Short Papers), pages 45--48. The Association for Computational Linguistics, 2007.

Digital Library

[12]

J. Fung, D. Hakkani-Tur, M. Magimai-Doss, E. Shriberg, S. Cuendet, and N. Mirghafori. Prosodic features and feature selection for multi-lingual sentence segmentation. In Proceedings of Interspeech 2007, pages 2585--2588, 2007.

[13]

C. Goodwin. Conversational Organization: interaction between speakers and hearers. Academic Press, 1981.

[14]

D. Heylen. Head gestures, gaze and the principles of conversational structure. International Journal of Humanoid Robotics, 3(3):241--267, 2006.

[15]

D. Heylen. Listening heads. In I. Wachsmuth and G. Knoblich, editors, Modeling Communication with robots and virtual humans, volume 4930 of Lecture Notes in Artificial Intelligence, pages 241--259. Springer Verlag, Berlin, 2008.

Digital Library

[16]

http://corpus.amiproject.org. The AMI Meeting Corpus, May 2009.

[17]

A. Kendon. Some functions of gaze direction in social interaction. Acta Psychologica, 26:22--63, 1967.

[18]

J. Laerty, A. McCallum, and F. Pereira. Conditional random fields: probabilistic models for segmenting and labelling sequence data. In ICML, 2001.

[19]

T. Minato, Y. Yoshikawa, T. Noda, S. Ikemoto, H. Ishiguro, and M. Asada. CB2: A child robot with biomimetic body for cognitive developmental robotics. In IROS 2008: Proceedings of the IEEE/RSJ 2008 International Conference on Intelligent RObots and Systems, pages 193--200, 2008.

[20]

L.-P. Morency, I. de Kok, and J. Gratch. Context-based recognition during human interactions: Automatic feature selection and encoding dictionary. In ICMI '08: Proceedings of the 10th International Conference on Multimodal Interfaces, pages 181--188, New York, NY, USA, 2008. ACM.

Digital Library

[21]

L.-P. Morency, I. de Kok, and J. Gratch. Predicting listener backchannels: A probabilistic multimodal approach. In Intelligent Virtual Agents (IVA '08), pages 176--190, 2008.

Digital Library

[22]

D. C. O'Connell, S. Kowal, and E. Kaltenbacher. Turn-taking: A critical analysis of the research tradition. Journal of Psycholinguistic Research, 19(6):345 -- 373, 1990.

[23]

L. R. Rabiner. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257--286, 1989.

[24]

R. J. Rienks, R. Poppe, and D. Heylen. Di erences in head orientation behavior for speakers and listeners: an experiment in a virtual environment. Transactions on Applied Perception, 7(1):accepted for publication, 2010.

Digital Library

[25]

H. Sacks, E. A. Scheglo, and G. Je erson. A simplest systematics for the organization of turn-taking for conversation. Language, 50(4):696 -- 735, 1974.

[26]

D. Sakamoto, T. Kanda, T. Ono, H. Ishiguro, and N. Hagita. Android as a telecommunication medium with a human-like presence. In HRI '07: Proceedings of the ACM/IEEE international conference on Human-robot interaction, pages 193--200, New York, NY, USA, 2007. ACM.

Digital Library

[27]

D. Schlangen. From reaction to prediction: Experiments with computational models of turn-taking. In Proceedings of Interspeech 2006, 2006.

[28]

T. Sikorski and J. F. Allen. A task-based evaluation of the trains-95 dialogue system. In ECAI '96: Workshop on Dialogue Processing in Spoken Language Systems, pages 207--220, London, UK, 1997. Springer-Verlag.

Digital Library

[29]

R. Vertegaal, R. Slagter, G. van der Veer, and A. Nijholt. Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes. In Proceedings of CHI'01, pages 301 -- 308. ACM, 2001.

Digital Library

[30]

R. Vertegaal, G. van der Veer, and H. Vons. E ects of gaze on multiparty mediated communication. In Proceedings of Graphics Interface, pages 95 -- 102, Montreal, Canada, 2000. Morgan Kaufmann Publishers.

[31]

N. Ward and W. Tsukahara. Prosodic features which cue back-channel responses in english and japanese. Journal of Pragmatics, 32(8):1177--1207, 2000.

Cited By

Lee MDeng Z(2024)Online Multimodal End-of-Turn Prediction for Three-party ConversationsProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685742(57-65)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685742
D’Mello SDuran NMichaels AStewart A(2024)Improving collaborative problem-solving skills via automated feedback and scaffolding: a quasi-experimental study with CPSCoach 2.0User Modeling and User-Adapted Interaction10.1007/s11257-023-09387-634:4(1087-1125)Online publication date: 14-Feb-2024
https://doi.org/10.1007/s11257-023-09387-6
Lee MTrinh MDeng Z(2023)Multimodal Turn Analysis and Prediction for Multi-party ConversationsProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614139(436-444)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614139
Show More Cited By

Index Terms

Multimodal end-of-turn prediction in multi-party meetings
1. Computing methodologies
  1. Artificial intelligence
    1. Distributed artificial intelligence
      1. Intelligent agents
    2. Natural language processing
      1. Discourse, dialogue and pragmatics

Recommendations

Multimodal Fusion using Respiration and Gaze for Predicting Next Speaker in Multi-Party Meetings
ICMI '15: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction

Techniques that use nonverbal behaviors to predict turn-taking situations, such as who will be the next speaker and the next utterance timing in multi-party meetings are receiving a lot of attention recently. It has long been known that gaze is a ...
Multimodal Continuous Turn-Taking Prediction Using Multiscale RNNs
ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

In human conversational interactions, turn-taking exchanges can be coordinated using cues from multiple modalities. To design spoken dialog systems that can conduct fluid interactions it is desirable to incorporate cues from separate modalities into ...
A conversation analytical study on multimodal turn-giving cues: end-of-turn prediction
COST'11: Proceedings of the 2011 international conference on Cognitive Behavioural Systems

The present paper focuses on the systematic study of the sequential organization of verbal as well as nonverbal behavior in spontaneous interaction. The study concerns one of the most universal structural features of conversation, the phenomenon of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI-MLMI '09: Proceedings of the 2009 international conference on Multimodal interfaces

November 2009

374 pages

ISBN:9781605587721

DOI:10.1145/1647314

General Chairs:
James L. Crowley
INRIA Grenoble Rhône-Alpes Research Centre, France
,
Yuri Ivanov
MERL, USA
,
Christopher Wren
Google, USA
,
Program Chairs:
Daniel Gatica-Perez
Idiap Research Institute, Switzerland
,
Michael Johnston
AT&T Research, USA
,
Rainer Stiefelhagen
University of Karlsruhe, Germany

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

ICMI-MLMI '09

Sponsor:

SIGCHI

ICMI-MLMI '09: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERFACES/WORKSHOP ON MACHINE LEARNING FOR MULTIMODAL INTERFACES

November 2 - 4, 2009

Massachusetts, Cambridge, USA

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

33
Total Citations
View Citations
372
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)3

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lee MDeng Z(2024)Online Multimodal End-of-Turn Prediction for Three-party ConversationsProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685742(57-65)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685742
D’Mello SDuran NMichaels AStewart A(2024)Improving collaborative problem-solving skills via automated feedback and scaffolding: a quasi-experimental study with CPSCoach 2.0User Modeling and User-Adapted Interaction10.1007/s11257-023-09387-634:4(1087-1125)Online publication date: 14-Feb-2024
https://doi.org/10.1007/s11257-023-09387-6
Lee MTrinh MDeng Z(2023)Multimodal Turn Analysis and Prediction for Multi-party ConversationsProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614139(436-444)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614139
Woo JYang LAchard CPelachaud C(2023)Are we in sync during turn switch?2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG57933.2023.10042799(1-4)Online publication date: 5-Jan-2023
https://dl.acm.org/doi/10.1109/FG57933.2023.10042799
Woo JYang LPelachaud CAchard C(2023)Is Turn-Shift Distinguishable with Synchrony?Artificial Intelligence in HCI10.1007/978-3-031-35894-4_32(419-432)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-35894-4_32
Ishii RRen XMuszynski MMorency L(2022)Trimodal prediction of speaking and listening willingness to help improve turn-changing modelingFrontiers in Psychology10.3389/fpsyg.2022.77454713Online publication date: 18-Oct-2022
https://doi.org/10.3389/fpsyg.2022.774547
Peled ALeinonen THasler B(2022)The Telerobot Contact HypothesisComputer-Human Interaction Research and Applications10.1007/978-3-031-22015-9_5(74-99)Online publication date: 13-Dec-2022
https://doi.org/10.1007/978-3-031-22015-9_5
Yang LAchard CPelachaud C(2022)Multimodal Analysis of InterruptionsDigital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Anthropometry, Human Behavior, and Communication10.1007/978-3-031-05890-5_24(306-325)Online publication date: 16-Jun-2022
https://doi.org/10.1007/978-3-031-05890-5_24
Pelachaud CBusso CHeylen D(2021)Multimodal Behavior Modeling for Socially Interactive AgentsThe Handbook on Socially Interactive Agents10.1145/3477322.3477331(259-310)Online publication date: 10-Sep-2021
https://dl.acm.org/doi/10.1145/3477322.3477331
Stewart AKeirn ZD’Mello S(2021)Multimodal modeling of collaborative problem-solving facets in triadsUser Modeling and User-Adapted Interaction10.1007/s11257-021-09290-yOnline publication date: 2-Feb-2021
https://doi.org/10.1007/s11257-021-09290-y
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents