extended-abstract

Towards Visual Dialogue for Human-Robot Interaction

Authors:

Jose L. Part,

Daniel Hernández García,

Yanchao Yu,

Nancie Gunson,

Christian Dondrup,

Oliver LemonAuthors Info & Claims

HRI '21 Companion: Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction

Pages 670 - 672

https://doi.org/10.1145/3434074.3447278

Published: 08 March 2021 Publication History

Get Access

Abstract

The goal of the EU H2020-ICT funded SPRING project is to develop a socially pertinent robot to carry out tasks in a gerontological healthcare unit. In this context, being able to perceive its environment and have coherent and relevant conversations about the surrounding world is critical. In this paper, we describe current progress towards developing the necessary integrated visual and conversational capabilities for a robot to operate in such environments. Concretely, we introduce an architecture for conversing about objects and other entities present in an environment. The work described in this paper has applications that extend well beyond healthcare and can be used on any robot that requires to interact with its visual and spatial environment in order to be able to perform its duties.

Supplementary Material

MP4 File (hrivd1012vf.mp4)

Supplemental video

Download
24.17 MB

References

[1]

Amanda Cercas Curry, Ioannis Papaioannou, Alessandro Suglia, Shubham Agarwal, Igor Shalyminov, Xu Xinnuo, Ondvrej Duvsek, Arash Eshghi, Ioannis Konstas, Verena Rieser, and Oliver Lemon. 2018. Alana v2: Entertaining and Informative Open-Domain Social Dialogue using Ontologies and Entity Linking. In Proceedings of the 2nd Alexa Prize.

Google Scholar

[2]

Christian Dondrup, Ioannis Papaioannou, and Oliver Lemon. 2019. Petri Net Machines for Human-Agent Interaction. arxiv: 1909.06174

Google Scholar

[3]

Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollá r. 2019. Panoptic Feature Pyramid Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6399--6408.

Crossref

Google Scholar

[4]

Kevin Lai, Liefeng Bo, and Dieter Fox. 2014. Unsupervised Feature Learning for 3D Scene Labeling. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). IEEE, 3050--3057.

Crossref

Google Scholar

[5]

Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Proceedings of the European Conference on Computer Vision (ECCV). Springer Verlag, 740--755.

Google Scholar

[6]

Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. 2019. Detectron2. https://github.com/facebookresearch/detectron2.

Google Scholar

Cited By

View all

Lemon OAlbrecht SWoolridge M(2022)Conversational AI for multi-agent communication in Natural LanguageAI Communications10.3233/AIC-22014735:4(295-308)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.3233/AIC-220147
Gunson NGarcía DSieińska WDondrup CLemon O(2022)Developing a Social Conversational Robot for the Hospital waiting room2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN53752.2022.9900827(1352-1357)Online publication date: 29-Aug-2022
https://dl.acm.org/doi/10.1109/RO-MAN53752.2022.9900827

Index Terms

Towards Visual Dialogue for Human-Robot Interaction
1. Computing methodologies
  1. Artificial intelligence

Recommendations

Combining Visual and Social Dialogue for Human-Robot Interaction
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction

We will demonstrate a prototype multimodal conversational AI system that will act as a receptionist in a hospital waiting room, combining visually-grounded dialogue with social conversation. The system supports visual object conversation in the waiting ...
A dialogue system for multimodal human-robot interaction
ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

This paper presents a POMDP-based dialogue system for multimodal human-robot interaction (HRI). Our aim is to exploit a dialogical paradigm to allow a natural and robust interaction between the human and the robot. The proposed dialogue system should ...
Human-robot collaborative tutoring using multiparty multimodal spoken dialogue
HRI '14: Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction

In this paper, we describe a project that explores a novel experimental setup towards building a spoken, multi-modally rich, and human-like multiparty tutoring robot. A human-robot interaction setup is designed, and a human-human dialogue corpus is ...

Comments

Information & Contributors

Information

Published In

HRI '21 Companion: Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction

March 2021

756 pages

ISBN:9781450382908

DOI:10.1145/3434074

General Chairs:
Cindy Bethel
Mississippi State University, USA
,
Ana Paiva
INESC-ID, IST, University of Lisbon, Portugal & Radcliffe Institute for Advanced Study, Harvard University, USA
,
Program Chairs:
Elizabeth Broadbent
University of Auckland, New Zealand
,
David Feil-Seifer
University of Nevada Reno, USA
,
Daniel Szafir
University of Colorado Boulder, USA

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 March 2021

Check for updates

Author Tags

Qualifiers

Extended-abstract

Funding Sources

European Commission

Conference

HRI '21

Sponsor:

HRI '21: ACM/IEEE International Conference on Human-Robot Interaction

March 8 - 11, 2021

CO, Boulder, USA

Acceptance Rates

Overall Acceptance Rate 192 of 519 submissions, 37%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
233
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Lemon OAlbrecht SWoolridge M(2022)Conversational AI for multi-agent communication in Natural LanguageAI Communications10.3233/AIC-22014735:4(295-308)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.3233/AIC-220147
Gunson NGarcía DSieińska WDondrup CLemon O(2022)Developing a Social Conversational Robot for the Hospital waiting room2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN53752.2022.9900827(1352-1357)Online publication date: 29-Aug-2022
https://dl.acm.org/doi/10.1109/RO-MAN53752.2022.9900827

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

Supplementary Material

References

Cited By

Index Terms

Recommendations

Combining Visual and Social Dialogue for Human-Robot Interaction

A dialogue system for multimodal human-robot interaction

Human-robot collaborative tutoring using multiparty multimodal spoken dialogue

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations