Article

Elvis: situated speech and gesture understanding for a robotic chandelier

Authors:

Joshua Juster,

Deb RoyAuthors Info & Claims

ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces

Pages 90 - 96

https://doi.org/10.1145/1027933.1027950

Published: 13 October 2004 Publication History

Get Access

Abstract

We describe a home lighting robot that uses directional spotlights to create complex lighting scenes. The robot senses its visual environment using a panoramic camera and attempts to maintain its target goal state by adjusting the positions and intensities of its lights. Users can communicate desired changes in the lighting environment through speech and gesture (e.g., "Make it brighter over there"). Information obtained from these two modalities are combined to form a goal, a desired change in the lighting of the scene. This goal is then incorporated into the system's target goal state. When the target goal state and the world are out of alignment, the system formulates a sensorimotor plan that acts on the world to return the system to homeostasis.

References

[1]

P. Lamere, P. Kwok, et al. Design of the CMU Sphinx-4 Decoder. Eurospeech, September 2003.

Google Scholar

[2]

M. Kuperstein. Neural model of adaptive hand-eye coordination for single postures. Science, 239. 1308--1311, 1988.

Google Scholar

[3]

M. Petrou, L. Shafarenko, and J. Kittler. Histogram-based segmentation in a perceptually uniform color space. IEEE Transactions on Image Processing, vol. 7, pp. 1354--1358, September 1998.

Digital Library

Google Scholar

[4]

A. Kendon. Conducting Interaction. Cambridge: Cambridge University Press 1990.

Google Scholar

[5]

D. McNeill. Hand and Mind. The University of Chicago Press, Chicago, 1992.

Google Scholar

[6]

R. Sharma, J. Cai, S. Chakravarthy, I. Poddar and Y. Sethi. Exploiting Speech/Gesture Cooccurrence for Improving Continuous Gesture Recognition in Weather Narration. In Proc. International Conference on Face and Gesture Recognition, Grenoble, France, 2000.

Digital Library

Google Scholar

[7]

S. Kettebekov and R. Sharma. Understanding Gestures in Multimodal Human-Computer Interaction. International Journal on Artificial Intelligence Tools, vol. 9, no. 2, pp. 205--224, June 2000.

Google Scholar

[8]

P. R. Cohen. Synergic use of direct manipulation and natural language. In Proc. Conference on Human Factors in Computing (CHI), (1989) 227.233. (1997) 415--422.

Digital Library

Google Scholar

[9]

S. Kita, I.V. Gijn, and H.V. Hulst. Movement phases in signs and co-speech gestures, and their transcription by human coders. In Proceedings of Intl. Gesture Workshop, (1997) 23--35.

Digital Library

Google Scholar

[10]

S. Oviatt, A. De Angeli, and K. Kuhn. Integration and synchronization of input modes during multimodal human-computer interaction. In Proceedings of the Conference on Human Factors in Computing Systems (CHI'97), 95--102, ACM Press, New York.

Digital Library

Google Scholar

Cited By

View all

(2013)A Formal Framework for Design and Validation of Multimodal Interactive Systems in Transport DomainHuman‐Computer Interactions in Transport10.1002/9781118601907.ch4(93-128)Online publication date: 24-Jan-2013
https://doi.org/10.1002/9781118601907.ch4
Chen JHaas EBarnes M(2007)Human Performance Issues and User Interface Design for Teleoperated RobotsIEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews10.1109/TSMCC.2007.90581937:6(1231-1245)Online publication date: 1-Nov-2007
https://dl.acm.org/doi/10.1109/TSMCC.2007.905819
Hüwel SWrede B(2006)Spontaneous speech understanding for robust multi-modal human-robot communicationProceedings of the COLING/ACL on Main conference poster sessions10.5555/1273073.1273124(391-398)Online publication date: 17-Jul-2006
https://dl.acm.org/doi/10.5555/1273073.1273124
Show More Cited By

Index Terms

Elvis: situated speech and gesture understanding for a robotic chandelier
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Touch screens
    2. Interaction paradigms

Recommendations

An approach to natural gesture in virtual environments
Special issue on virtual reality software and technology

This article presents research—an experiment and the resulting prototype—on a method for treating gestural input so that it can be used for multimodal applications, such as interacting with virtual environments. This method involves the capture and use ...
Spontaneous spoken dialogues with the furhat human-like robot head
HRI '14: Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction

Furhat [1] is a robot head that deploys a back-projected animated face that is realistic and human-like in anatomy. Furhat relies on a state-of-the-art facial animation architecture allowing accurate synchronized lip movements with speech, and the ...
Multimodal multiparty social interaction with the furhat head
ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

We will show in this demonstrator an advanced multimodal and multiparty spoken conversational system using Furhat, a robot head based on projected facial animation. Furhat is a human-like interface that utilizes facial animation for physical robot heads ...

Comments

Information & Contributors

Information

Published In

ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces

October 2004

368 pages

ISBN:1581139950

DOI:10.1145/1027933

General Chairs:
Rajeev Sharma
Advanced Interfaces
,
Trevor Darrell
Massachusetts Institute of Technology
,
Program Chairs:
Mary Harper
Purdue University, West Lafayette, IN
,
Gianni Lazzari
ITC-IRST
,
Matthew Turk
University of California, Santa Barbara, CA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 October 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICMI04

Sponsor:

ICMI04: Sixth International Conference on Multimodal Interfaces 2004

October 13 - 15, 2004

PA, State College, USA

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
193
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

(2013)A Formal Framework for Design and Validation of Multimodal Interactive Systems in Transport DomainHuman‐Computer Interactions in Transport10.1002/9781118601907.ch4(93-128)Online publication date: 24-Jan-2013
https://doi.org/10.1002/9781118601907.ch4
Chen JHaas EBarnes M(2007)Human Performance Issues and User Interface Design for Teleoperated RobotsIEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews10.1109/TSMCC.2007.90581937:6(1231-1245)Online publication date: 1-Nov-2007
https://dl.acm.org/doi/10.1109/TSMCC.2007.905819
Hüwel SWrede B(2006)Spontaneous speech understanding for robust multi-modal human-robot communicationProceedings of the COLING/ACL on Main conference poster sessions10.5555/1273073.1273124(391-398)Online publication date: 17-Jul-2006
https://dl.acm.org/doi/10.5555/1273073.1273124
Leuski ALavrenko VYu PTsotras VFox ELiu B(2006)Tracking dragon-hunters with language modelsProceedings of the 15th ACM international conference on Information and knowledge management10.1145/1183614.1183714(698-707)Online publication date: 6-Nov-2006
https://dl.acm.org/doi/10.1145/1183614.1183714
Martin JBuisine SPitel GBernsen N(2006)Fusion of children's speech and 2D gestures when conversing with 3D charactersSignal Processing10.1016/j.sigpro.2006.02.04086:12(3596-3624)Online publication date: 1-Dec-2006
https://dl.acm.org/doi/10.1016/j.sigpro.2006.02.040
Hina MRamdane-Cherif ATadj C(2005)A context-sensitive incremental learning paradigm of an ubiquitous multimodal multimedia computing systemWiMob'2005), IEEE International Conference on Wireless And Mobile Computing, Networking And Communications, 2005.10.1109/WIMOB.2005.1512956(104-111)Online publication date: 2005
https://doi.org/10.1109/WIMOB.2005.1512956
Roy DReiter E(2005)Connecting language to the worldArtificial Intelligence10.1016/j.artint.2005.06.002167:1-2(1-12)Online publication date: Sep-2005
https://doi.org/10.1016/j.artint.2005.06.002
Roy D(2005)Semiotic schemas: A framework for grounding language in action and perceptionArtificial Intelligence10.1016/j.artint.2005.04.007167:1-2(170-205)Online publication date: Sep-2005
https://doi.org/10.1016/j.artint.2005.04.007

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

An approach to natural gesture in virtual environments

Spontaneous spoken dialogues with the furhat human-like robot head

Multimodal multiparty social interaction with the furhat head

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

An approach to natural gesture in virtual environments

Spontaneous spoken dialogues with the furhat human-like robot head

Multimodal multiparty social interaction with the furhat head

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations