poster

Timing multimodal turn-taking for human-robot cooperation

Author:

Crystal ChaoAuthors Info & Claims

ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

Pages 309 - 312

https://doi.org/10.1145/2388676.2388744

Published: 22 October 2012 Publication History

Abstract

In human cooperation, the concurrent usage of multiple social modalities such as speech, gesture, and gaze results in robust and efficient communicative acts. Such multimodality in combination with reciprocal intentions supports fluent turn-taking. I hypothesize that human-robot turn-taking can be made more fluent through appropriate timing of multimodal actions. Managing timing includes understanding the impact that timing can have on interactions as well as having a control system that supports the manipulation of such timing. To this end, I propose to develop a computational turn-taking model of the timing and information flow of reciprocal interactions. I also propose to develop an architecture based on the timed Petri net (TPN) for the generation of coordinated multimodal behavior, inside of which the turn-taking model will regulate turn timing and action initiation and interruption in order to seize and yield control. Through user studies in multiple domains, I intend to demonstrate the system's generality and evaluate the system on balance of control, fluency, and task effectiveness.

References

[1]

B. Blumberg, M. Downie, Y. Ivanov, M. Berlin, M. Johnson, and B. Tomlinson. Integrated learning for interactive synthetic characters. In Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pages 417--426, 2002.

Digital Library

[2]

D. Bohus and E. Horvitz. Facilitating multiparty dialog with gaze, gesture, and speech. In Proceedings of the 12th International Conference on Multimodal Interfaces (ICMI), 2010.

Digital Library

[3]

M. Cakmak, C. Chao, and A. L. Thomaz. Designing interactions for robot active learners. IEEE Transactions on Autonomous Mental Development, 2(2):108--118, June 2010.

Digital Library

[4]

J. Cassell, T. Bickmore, L. Campbell, K. Chang, H. Vilhjálmsson, and H. Yan. Requirements for an architecture for embodied conversational characters. In Computer Animation and Simulation, pages 109--120. Springer Verlag, 1999.

[5]

C. Chao, J. H. Lee, M. Begum, and A. Thomaz. Simon plays Simon says: The timing of turn-taking in an imitation game. In IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2011.

[6]

C. Chao and A. Thomaz. Timing in multimodal reciprocal interactions: Control and analysis using timed Petri nets. Journal of Human-Robot Interaction, 1(1):46--67, 2012.

[7]

H. H. Clark and M. A. Krych. Speaking while monitoring addressees for understanding. Journal of Memory and Language, 50(1):62--81, 2004.

[8]

S. Duncan. On the structure of speaker-auditor interaction during speaking turns. Language in Society, 3(2):161--180, 1974.

[9]

G. Hoffman and C. Breazeal. Effects of anticipatory action on human-robot teamwork: Efficiency, fluency, and perception of team. In Proceedings of the 3rd ACM/IEEE Conference on Human-Robot Interaction (HRI), 2007.

Digital Library

[10]

A. G. Holroyd. Generating engagement behaviors in human-robot interaction. Master's thesis, Worcester Polytechnic Institute, 2011.

[11]

H. Kose-Bagci, K. Dautenhan, and C. L. Nehaniv. Emergent dynamics of turn-taking interaction in drumming games with a humanoid robot. In Proceedings of the 17th IEEE International Symposium on Robot and Human Interactive Communication, pages 346--353, 2008.

[12]

B. Mutlu, T. Shiwa, T. K. H. Ishiguro, and N. Hagita. Footing in human-robot conversations: How robots might shape participant roles using gaze cues. In Proceedings of the 4th ACM/IEEE Conference on Human-Robot Interaction (HRI), 2009.

Digital Library

[13]

B. Orestrom. Turn-taking in English conversation. CWK Gleerup, 1983.

[14]

A. Raux and M. Eskenazi. A finite-state turn-taking model for spoken dialog systems. In Proceedings of the Human Language Technologies (HLT), 2009.

Digital Library

[15]

H. Sacks, E. Schegloff, and G. Jefferson. A simplest systematics for the organization of turn-taking for conversation. Language, 50:696--735, 1974.

[16]

E. Schegloff. Overlapping talk and the organization of turn-taking for conversation. Language in Society, 29(1):1--63, 2000.

[17]

E. Tronick, H. Als, and L. Adamson. Structure of early face-to-face communicative interactions. In M. Bullowa, editor, Before Speech: The Beginning of Interpersonal Communication, pages 349--374. Cambridge University Press, Cambridge, 1979.

Cited By

Udeh CChen LDu SLi MWu M(2023)Multimodal Facial Emotion Recognition Using Improved Convolution Neural Networks ModelJournal of Advanced Computational Intelligence and Intelligent Informatics10.20965/jaciii.2023.p071027:4(710-719)Online publication date: 20-Jul-2023
https://doi.org/10.20965/jaciii.2023.p0710
Gebhard PSchneeberger TMehlmann GBaur TAndré EPelachaud CMartin JBuschmeier HLucas GKopp S(2019)Designing the Impression of Social Agents' Real-time Interruption HandlingProceedings of the 19th ACM International Conference on Intelligent Virtual Agents10.1145/3308532.3329435(19-21)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3308532.3329435
Lee NLee S(2018)The Effect of Back-Channeling Cues on Motivation to Continue Human-Machine Textual InteractionProceedings of the 12th International Conference on Ubiquitous Information Management and Communication10.1145/3164541.3164626(1-4)Online publication date: 5-Jan-2018
https://dl.acm.org/doi/10.1145/3164541.3164626
Show More Cited By

Index Terms

Timing multimodal turn-taking for human-robot cooperation
1. Computer systems organization
  1. Embedded and cyber-physical systems
    1. Robotics
      1. External interfaces for robotics
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces

Recommendations

Smooth Turn-taking by a Robot Using an Online Continuous Model to Generate Turn-taking Cues
ICMI '19: 2019 International Conference on Multimodal Interaction

Turn-taking in human-robot interaction is a crucial part of spoken dialogue systems, but current models do not allow for human-like turn-taking speed seen in natural conversation. In this work we propose combining two independent prediction models. A ...
Timing in multimodal turn-taking interactions: control and analysis using timed Petri nets
Inaugural Special Issue: Intersection of Systems Sciences and Human Sciences

Turn-taking interactions with humans are multimodal and reciprocal in nature. In addition, the timing of actions is of great importance, as it influences both social and task strategies. To enable the precise control and analysis of timed discrete ...
Timing in human-robot interaction
HRI '14: Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction

Timing plays a role in a range of human-robot interaction scenarios, as humans are highly sensitive to timing and interaction fluency. It is central to spoken dialogue, with turn-taking, interruptions, and hesitation influencing both task efficiency and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

October 2012

636 pages

ISBN:9781450314671

DOI:10.1145/2388676

General Chairs:
Louis-Philippe Morency
University of Southern California, USA
,
Dan Bohus
Microsoft Research, USA
,
Hamid Aghajan
Stanford University, USA
,
Program Chairs:
Justine Cassell
Carnegie Mellon University, USA
,
Anton Nijholt
University of Twente, Netherlands
,
Julien Epps
The University of New South Wales, Australia

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

ICMI '12

Sponsor:

SIGCHI

ICMI '12: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 22 - 26, 2012

California, Santa Monica, USA

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
255
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)3

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Udeh CChen LDu SLi MWu M(2023)Multimodal Facial Emotion Recognition Using Improved Convolution Neural Networks ModelJournal of Advanced Computational Intelligence and Intelligent Informatics10.20965/jaciii.2023.p071027:4(710-719)Online publication date: 20-Jul-2023
https://doi.org/10.20965/jaciii.2023.p0710
Gebhard PSchneeberger TMehlmann GBaur TAndré EPelachaud CMartin JBuschmeier HLucas GKopp S(2019)Designing the Impression of Social Agents' Real-time Interruption HandlingProceedings of the 19th ACM International Conference on Intelligent Virtual Agents10.1145/3308532.3329435(19-21)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3308532.3329435
Lee NLee S(2018)The Effect of Back-Channeling Cues on Motivation to Continue Human-Machine Textual InteractionProceedings of the 12th International Conference on Ubiquitous Information Management and Communication10.1145/3164541.3164626(1-4)Online publication date: 5-Jan-2018
https://dl.acm.org/doi/10.1145/3164541.3164626
Maniadakis MHourdakis ETrahanias P(2017)Time-informed task planning in multi-agent collaborationCognitive Systems Research10.1016/j.cogsys.2016.09.00443:C(291-300)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1016/j.cogsys.2016.09.004

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents