Article

Identifying the addressee in human-human-robot interactions based on head pose and speech

Authors:

Michael Katzenmaier,

Rainer Stiefelhagen,

Tanja SchultzAuthors Info & Claims

ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces

Pages 144 - 151

https://doi.org/10.1145/1027933.1027959

Published: 13 October 2004 Publication History

Get Access

Abstract

In this work we investigate the power of acoustic and visual cues, and their combination, to identify the addressee in a human-human-robot interaction. Based on eighteen audio-visual recordings of two human beings and a (simulated) robot we discriminate the interaction of the two humans from the interaction of one human with the robot. The paper compares the result of three approaches. The first approach uses purely acoustic cues to find the addressees. Low level, feature based cues as well as higher-level cues are examined. In the second approach we test whether the human's head pose is a suitable cue. Our results show that visually estimated head pose is a more reliable cue for the identification of the addressee in the human-human-robot interaction. In the third approach we combine the acoustic and visual cues which results in significant improvements.

References

[1]

Special Issue on Human-Friendly Robots, volume 16. Journal of the Robotics Society of Japan, 1998.]]

Google Scholar

[2]

Proceedings of the Third IEEE International Conference on Humanoid Robots - Humanoids 2003. IEEE, Karlsruhe, Germany, 2003.]]

Google Scholar

[3]

M. Argyle. Social Interaction. Methuen, London, 1969.]]

Google Scholar

[4]

I. Bakx, K. van Turnhout, and J. Terken. Facial orientation during multi-party interaction with information kiosks. In Proceedings of the Interact 2003, Zurich, Switzerland, 2003.]]

Google Scholar

[5]

J.W. Tankard. Effects of eye position on person perception. Perc. Mot. Skills, (31):883--93, 1970.]]

Google Scholar

[6]

M. Katzenmaier. Determining the adressee in spoken human robot interaction, studienarbeit. Technical report, Fakultat fur Informatik, Universitat Karlsruhe (TH), 2003.]]

Google Scholar

[7]

C. L. Kleinke, A. A. Bustos, F. B. Meeker, and R. A. Staneski. Effects of self-attributed and other-attributed gaze in interpersonal evaluations between males and females. Journal of experimental social Psychology, (9):154--63, 1973.]]

Google Scholar

[8]

P. P. Maglio, T. Matlock, C. S. Campbell, S. Zhai, and B. A. Smith. Gaze and speech in attentive user interfaces. In Proceedings of the International Conference on Multimodal Interfaces, volume 1948 of LNCS. Springer, 2000.]]

Digital Library

Google Scholar

[9]

J. Ruusuvuori. Looking means listening: coordinating displays of engagement in doctor-patient interaction. Social Science & Medicine, 52:1093--1108, 2001.]]

Crossref

Google Scholar

[10]

R. Stiefelhagen. Tracking focus of attention in meetings. In International Conference on Multimodal Interfaces, pages 273--280, Pittsburgh, PA, October 2002. IEEE.]]

Digital Library

Google Scholar

[11]

R. Stiefelhagen and J. Zhu. Head orientation and gaze direction in meetings. In Conference on Human Factors in Computing Systems (CHI2002), Minneapolis, April 2002.]]

Digital Library

Google Scholar

[12]

R. Vertegaal, R. Slagter, G. van der Veer, and A. Nijholt. Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes. In SIGCHI'01, Seattle, March 2001. ACM.]]

Digital Library

Google Scholar

Cited By

View all

Sato HShinohara YOgawa A(2023)Multi-modal modeling for device-directed speech detection using acoustic and linguistic cuesAcoustical Science and Technology10.1250/ast.44.4044:1(40-43)Online publication date: 1-Jan-2023
https://doi.org/10.1250/ast.44.40
Mazzola CRomeo MRea FSciutti ACangelosi A(2023)To Whom are You Talking? A Deep Learning Model to Endow Social Robots with Addressee Estimation Skills2023 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN54540.2023.10191452(1-10)Online publication date: 18-Jun-2023
https://doi.org/10.1109/IJCNN54540.2023.10191452
Jaber RMcMillan D(2022)Cross-Modal Repair: Gaze and Speech Interaction for List AdvancementProceedings of the 4th Conference on Conversational User Interfaces10.1145/3543829.3543833(1-11)Online publication date: 26-Jul-2022
https://dl.acm.org/doi/10.1145/3543829.3543833
Show More Cited By

Index Terms

Identifying the addressee in human-human-robot interactions based on head pose and speech
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Improving automatic speech recognition through head pose driven visual grounding
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

In this paper, we present a multimodal speech recognition system for real world scene description tasks. Given a visual scene, the system dynamically biases its language model based on the content of the visual scene and visual attention of the speaker. ...
Speech-Input Speech-Output Communication for Dysarthric Speakers Using HMM-Based Speech Recognition and Adaptive Synthesis System

Dysarthria is a motor speech disorder that causes inability to control and coordinate one or more articulators. This makes it difficult for a dysarthric speaker to utter certain speech sound units, thereby producing poorly articulated, slurred, and ...
Lithuanian Speech Corpus Liepa for Development of Human-Computer Interfaces Working in Voice Recognition and Synthesis Mode

The problem of speech corpus for design of human-computer interfaces working in voice recognition and synthesis mode is investigated. Specific requirements of speech corpus for speech recognizers and synthesizers were accented. It has been discussed that ...

Comments

Information & Contributors

Information

Published In

ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces

October 2004

368 pages

ISBN:1581139950

DOI:10.1145/1027933

General Chairs:
Rajeev Sharma
Advanced Interfaces
,
Trevor Darrell
Massachusetts Institute of Technology
,
Program Chairs:
Mary Harper
Purdue University, West Lafayette, IN
,
Gianni Lazzari
ITC-IRST
,
Matthew Turk
University of California, Santa Barbara, CA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 October 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICMI04

Sponsor:

ICMI04: Sixth International Conference on Multimodal Interfaces 2004

October 13 - 15, 2004

PA, State College, USA

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

83
Total Citations
View Citations
622
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)6

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Sato HShinohara YOgawa A(2023)Multi-modal modeling for device-directed speech detection using acoustic and linguistic cuesAcoustical Science and Technology10.1250/ast.44.4044:1(40-43)Online publication date: 1-Jan-2023
https://doi.org/10.1250/ast.44.40
Mazzola CRomeo MRea FSciutti ACangelosi A(2023)To Whom are You Talking? A Deep Learning Model to Endow Social Robots with Addressee Estimation Skills2023 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN54540.2023.10191452(1-10)Online publication date: 18-Jun-2023
https://doi.org/10.1109/IJCNN54540.2023.10191452
Jaber RMcMillan D(2022)Cross-Modal Repair: Gaze and Speech Interaction for List AdvancementProceedings of the 4th Conference on Conversational User Interfaces10.1145/3543829.3543833(1-11)Online publication date: 26-Jul-2022
https://dl.acm.org/doi/10.1145/3543829.3543833
Sanghvi NYonetani RKitani KEl Fallah Seghrouchni ASukthankar GAn BYorke-Smith N(2020)MGpiProceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems10.5555/3398761.3398900(1196-1205)Online publication date: 5-May-2020
https://dl.acm.org/doi/10.5555/3398761.3398900
Huang HKimura SKuwabara KNishida T(2020)Generation of Head Movements of a Robot Using Multimodal Features of Peer Participants in Group Discussion ConversationMultimodal Technologies and Interaction10.3390/mti40200154:2(15)Online publication date: 29-Apr-2020
https://doi.org/10.3390/mti4020015
Sang HWang ZHe BZhou YLi G(2020)A Novel Intelligent Robot for Epidemic Identification and Prevention2020 IEEE 16th International Conference on Control & Automation (ICCA)10.1109/ICCA51439.2020.9264540(183-186)Online publication date: 9-Oct-2020
https://doi.org/10.1109/ICCA51439.2020.9264540
Yılmaz EFal MAcartürk C(2020)The Role of Gaze as a Deictic Cue in Human Robot InteractionAugmented Cognition. Human Cognition and Behavior10.1007/978-3-030-50439-7_32(466-478)Online publication date: 10-Jul-2020
https://doi.org/10.1007/978-3-030-50439-7_32
McMillan DBrown BKawaguchi IJaber RSolsona Belenguer JKuzuoka H(2019)Designing with GazeProceedings of the ACM on Human-Computer Interaction10.1145/33592783:CSCW(1-26)Online publication date: 7-Nov-2019
https://dl.acm.org/doi/10.1145/3359278
Jaber RMcMillan DBelenguer JBrown BCowan BClark L(2019)Patterns of gaze in speech agent interactionProceedings of the 1st International Conference on Conversational User Interfaces10.1145/3342775.3342791(1-10)Online publication date: 22-Aug-2019
https://dl.acm.org/doi/10.1145/3342775.3342791
Skantze GGustafson JBeskow J(2019)Multimodal conversational interaction with robotsThe Handbook of Multimodal-Multisensor Interfaces10.1145/3233795.3233799(77-104)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3233795.3233799
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Improving automatic speech recognition through head pose driven visual grounding

Speech-Input Speech-Output Communication for Dysarthric Speakers Using HMM-Based Speech Recognition and Adaptive Synthesis System

Lithuanian Speech Corpus Liepa for Development of Human-Computer Interfaces Working in Voice Recognition and Synthesis Mode

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Improving automatic speech recognition through head pose driven visual grounding

Speech-Input Speech-Output Communication for Dysarthric Speakers Using HMM-Based Speech Recognition and Adaptive Synthesis System

Lithuanian Speech Corpus Liepa for Development of Human-Computer Interfaces Working in Voice Recognition and Synthesis Mode

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations