research-article

Public Access

Toward Genuine Robot Teammates: Improving Human-Robot Team Performance Using Robot Shared Mental Models

Authors:

Ravenna Thielstrom,

Matthias ScheutzAuthors Info & Claims

AAMAS '20: Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems

Pages 429 - 437

Published: 13 May 2020 Publication History

Abstract

Effective coordination is a critical requirement for human teaming, and is increasingly needed in teams of humans and robots. Building on decades of work in the behavioral literature, we have implemented a computational framework for coordination based on Shared Mental Models (SMMs) in which robots use a distributed knowledge base to coordinate activity. We also built a novel system connecting the robotic architecture, DIARC, to the 3D simulation environment, Unity, to serve as an evaluation platform for the framework implementation, and also for more general explorations of teaming with autonomous robots. Using this platform, we ran a user study to evaluate the framework by comparing performance of teams in which the robots used SMMs with those that did not. We found that teams in which the robots used SMMs significantly outperformed those without SMMs. This represents the first empirical demonstration that SMMs can be successfully used by fully autonomous robots interacting in natural language to improve team performance, bringing robots a step closer to genuine teammates.

References

[1]

Jenay M Beer, Arthur D Fisk, and Wendy A Rogers. 2014. Toward a framework for levels of robot autonomy in human-robot interaction. Journal of human-robot interaction, Vol. 3, 2 (2014), 74--99.

Digital Library

[2]

Cheryl A Bolstad and Mica R Endsley. 1999. Shared mental models and shared displays: An empirical evaluation of team performance. In proceedings of the human factors and ergonomics society annual meeting, Vol. 43. 213--217.

[3]

Abhizna Butchibabu, Christopher Sparano-Huiban, Liz Sonenberg, and Julie Shah. 2016. Implicit coordination strategies for effective team communication. Human factors, Vol. 58, 4 (2016), 595--610.

[4]

Sharolyn Converse, JA Cannon-Bowers, and E Salas. 1993. Shared mental models in expert team decision making. Individual and group decision making: Current issues, Vol. 221 (1993).

[5]

Myron A Diftler, JS Mehling, Muhammad E Abdallah, Nicolaus A Radford, Lyndon B Bridgwater, Adam M Sanders, Roger Scott Askew, D Marty Linn, John D Yamokoski, FA Permenter, et almbox. 2011. Robonaut 2-the first humanoid robot in space. In Proceedings of the 2011 IEEE international conference on robotics and automation. IEEE, 2178--2183.

[6]

Mica R Endsley. 1988. Situation awareness global assessment technique (SAGAT). In Proceedings of the IEEE 1988 National Aerospace and Electronics Conference. IEEE, 789--795.

[7]

Mica R Endsley. 2000. Situation models: An avenue to the modeling of mental models. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 44. SAGE Publications Sage CA: Los Angeles, CA, 61--64.

[8]

J Alberto Espinosa, Robert E Kraut, Sandra A Slaughter, Javier F Lerch, James D Herbsleb, and Audris Mockus. 2001. Shared Mental Models, Familiarity and Coordination: A Multi-Method Study of Distributed Software Teams. In Proceedings of the International Conference on Information Systems.

[9]

Terrence Fong and Illah Nourbakhsh. 2005. Interaction challenges in human-robot space exploration. Invited article for ACM Interactions special issue on human-robot interaction (2005).

[10]

Terrence Fong, Jennifer Rochlis Zumbado, Nancy Currie, Andrew Mishkin, and David L Akin. 2013. Space telerobotics: unique challenges to human--robot collaboration in space. Reviews of Human Factors and Ergonomics, Vol. 9, 1 (2013), 6--56.

[11]

Felix Gervits, Kathleen Eberhard, and Matthias Scheutz. 2016a. Disfluent but effective? A quantitative study of disfluencies and conversational moves in team discourse. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. Osaka, Japan, 3359--3369.

[12]

Felix Gervits, Kathleen Eberhard, and Matthias Scheutz. 2016b. Team communication as a collaborative process. Frontiers in Robotics and AI, Vol. 3 (2016), 62.

[13]

Felix Gervits, Terry Fong, and Matthias Scheutz. 2018. Shared Mental Models to Support Distributed Human-Robot Teaming in Space. In 2018 AIAA SPACE and Astronautics Forum and Exposition. 5340.

[14]

Felix Gervits, Anton Leuski, Claire Bonial, Carla Gordon, and David Traum. 2019. A Classification-Based Approach to Automating Human-Robot Dialogue. (2019).

[15]

Felix Gervits and Matthias Scheutz. 2018a. Pardon the interruption: Managing turn-taking through overlap resolution in embodied artificial agents. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue. 99--109.

[16]

Felix Gervits and Matthias Scheutz. 2018b. Towards A Conversation-Analytic Taxonomy of Speech Overlap. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).

[17]

Felix Gervits, Charlotte Warne, Harrison Downs, Kathleen Eberhard, and Matthias Scheutz. 2017. Exploring Coordination in Human-Robot Teams in Space. In AIAA SPACE and Astronautics Forum and Exposition. 5309.

[18]

Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. Proceedings of the human factors and ergonomics society annual meeting. 904--908.

[19]

Jamison Heard, Rachel Heald, Caroline E. Harriott, and Julie A. Adams. 2019. A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human--Robot Teams. ACM Trans. Hum.-Robot Interact., Vol. 8, 2, Article 7 (June 2019), 30 pages. https://doi.org/10.1145/3314387

Digital Library

[20]

Catholijn M Jonker, M Birna Van Riemsdijk, and Bas Vermeulen. 2011. Shared mental models. Coordination, organizations, institutions, and norms in agent systems VI. Springer, 132--151.

[21]

M Lee, Tristan Johnson, and Myung H Jin. 2012. Toward understanding the dynamic relationship between team and task shared mental models as determinants of team and individual performances. International journal of information technology and business management, Vol. 8, 1 (2012), 1--14.

[22]

Anton Leuski and David Traum. 2011. NPCEditor: Creating virtual human dialogue using information retrieval techniques. Ai Magazine, Vol. 32, 2 (2011), 42--56.

Digital Library

[23]

Stephanie M Lukin, Felix Gervits, Cory Hayes, Pooja Moolchandani, Anton Leuski, John Rogers, Carlos Sanchez Amaro, Matthew Marge, Clare Voss, and David Traum. 2018. ScoutBot: A Dialogue System for Collaborative Navigation. In Proceedings of ACL 2018, System Demonstrations. 93--98.

[24]

J.E. Mathieu, T.S. Heffner, G.F. Goodwin, E. Salas, and J.A. Cannon-Bowers. 2000. The influence of shared mental models on team process and performance. The Journal of applied psychology, Vol. 85, 2 (2000), 273--83. https://doi.org/10.1037/0021-9010.85.2.273

[25]

Susan Mohammed, Lori Ferzandi, and Katherine Hamilton. 2010. Metaphor no more: A 15-year review of the team mental model construct. Journal of management, Vol. 36, 4 (2010), 876--910.

[26]

Robin R Murphy. 2004. Human-robot interaction in rescue robotics. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), Vol. 34, 2 (2004), 138--153.

Digital Library

[27]

Stefanos Nikolaidis and Julie Shah. 2012. Human-robot teaming using shared mental models. Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (2012).

[28]

Judith Orasanu. 1990. Shared mental models and crew decision making. Technical Report Technical Report #46.

[29]

Scott Ososky, David Schuster, Florian Jentsch, Stephen Fiore, Randall Shumaker, Christian Lebiere, Unmesh Kurup, Jean Oh, and Anthony Stentz. 2012. The importance of shared mental models and shared situation awareness for transforming robots from tools to teammates. In Unmanned Systems Technology XIV, Vol. 8387. International Society for Optics and Photonics, 838710.

[30]

Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et almbox. 2011. The Kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding. IEEE Signal Processing Society.

[31]

Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y Ng. 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software, Vol. 3. Kobe, Japan, 5.

[32]

Eduardo Salas, Carolyn Prince, David P Baker, and Lisa Shrestha. 1995. Situation awareness in team performance: Implications for measurement and training. Human factors, Vol. 37, 1 (1995), 123--136.

[33]

Matthias Scheutz. 2006. ADE: Steps toward a distributed development and runtime environment for complex robotic agent architectures. Applied Artificial Intelligence, Vol. 20, 2--4 (2006), 275--304.

[34]

Matthias Scheutz, Scott A DeLoach, and Julie A Adams. 2017. A framework for developing and using shared mental models in human-agent teams. Journal of Cognitive Engineering and Decision Making, Vol. 11, 3 (2017), 203--224.

[35]

Matthias Scheutz, Thomas Williams, Evan Krause, Bradley Oosterveld, Vasanth Sarathy, and Tyler Frasca. 2019. An overview of the distributed integrated cognition affect and reflection diarc architecture. Cognitive Architectures. Springer, 165--193.

[36]

James Michael Sellers. 2013. Team Workload Questionnaire (TWLQ): Development and Assessment of a Subjective Measure of Team Workload. Ph.D. Dissertation. University of Canterbury. Psychology.

[37]

Daniel Serfaty, Elliot E Entin, and Catherine Volpe. 1993. Adaptation to stress in team decision-making and coordination. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 37. SAGE Publications Sage CA: Los Angeles, CA, 1228--1232.

[38]

Renée J Stout, Janis A Cannon-Bowers, Eduardo Salas, and Dana M Milanovich. 1999. Planning, shared mental models, and coordinated performance: An empirical link is established. Human Factors, Vol. 41, 1 (1999), 61--71.

[39]

K. Sycara and G. Sukthankar. 2006. Literature Review of Teamwork Models. Technical Report Carnegie Mellon University-RI-TR-06-50. Robotics Institute, Carnegie Mellon University, Pittsburgh, PA.

[40]

Kartik Talamadupula, Gordon Briggs, Tathagata Chakraborti, Matthias Scheutz, and Subbarao Kambhampati. 2014. Coordination in human-robot teams using mental modeling and plan recognition. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2957--2962.

[41]

RM Taylor. 1990. Situational Awareness Rating Technique(SART): The development of a tool for aircrew systems design. In Proceedings of Situational Awareness in Aerospace Operations (AGARD) (1990).

[42]

Simon Thiel, Dagmar H"abe, and Micha Block. 2009. Co-operative robot teams in a hospital environment. In 2009 IEEE International Conference on Intelligent Computing and Intelligent Systems, Vol. 2. IEEE, 843--847.

[43]

Willie Walker, Paul Lamere, Philip Kwok, Bhiksha Raj, Rita Singh, Evandro Gouvea, Peter Wolf, and Joe Wölfel. 2004. Sphinx-4: A flexible open source framework for speech recognition. Sun Microsystems (12 2004).

[44]

Tom Williams, Priscilla Briggs, Nathaniel Pelz, and Matthias Scheutz. 2014. Is robot telepathy acceptable? Investigating effects of nonverbal robot-robot communication on human-robot interaction. The 23rd IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 886--891.

[45]

John Yen, Xiaocong Fan, Shuang Sun, Timothy Hanratty, and John Dumer. 2006. Agents with shared mental models for enhancing team decision makings. Decision Support Systems, Vol. 41, 3 (2006), 634--653.

Digital Library

Cited By

He ZLi SSong YCai Z(2024)Towards Building Condition-Based Cross-Modality Intention-Aware Human-AI Cooperation under VR EnvironmentProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642360(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642360
He ZSong YZhou SCai Z(2023)Interaction of Thoughts: Towards Mediating Task Assignment in Human-AI Cooperation with a Capability-Aware Shared Mental ModelProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580983(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580983

Index Terms

Toward Genuine Robot Teammates: Improving Human-Robot Team Performance Using Robot Shared Mental Models

Recommendations

Exploring human-robot trust during teaming in a real-world testbed
HRI '19: Proceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction

Project Aquaticus is a human-robot teaming competition on the water involving autonomous surface vehicles and human operated motorized kayaks. Teams composed of both humans and robots share the same physical environment to play capture the flag. In this ...
Explainability in Human-Robot Teaming
Abstract
In human-robot teaming, one of the crucial keys for the team’s success is that the robot and human teammates can collaborate accordingly in a coordinated manner. Each teammate should be aware of what the other teammate is going to perform and ...
The Cyber-Physical Control Room: A Mixed Reality Interface for Mobile Robot Teleoperation and Human-Robot Teaming
HRI '24: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction

In this work, we present the design and evaluation of an immersive Cyber-Physical Control Room interface for remote mobile robots that provides users with both robot-egocentric and robot-exocentric 3D perspectives. We evaluate the Cyber-Physical Control ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AAMAS '20: Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems

May 2020

2289 pages

ISBN:9781450375184

General Chairs:
Amal El Fallah Seghrouchni
Sorbonne University, France
,
Gita Sukthankar
University of Central Florida, United States
,
Program Chairs:
Bo An
Nanyang Technological University, Singapore
,
Neil Yorke-Smith Yorke-Smith
Delft University of Technology, Netherlands

Sponsors

Publisher

International Foundation for Autonomous Agents and Multiagent Systems

Richland, SC

Publication History

Published: 13 May 2020

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NASA
AFOSR

Conference

AAMAS '19

Sponsor:

SIGAI

AAMAS '19: International Conference on Autonomous Agents and Multiagent Systems

May 9 - 13, 2020

Auckland, New Zealand

Acceptance Rates

Overall Acceptance Rate 1,155 of 5,036 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
470
Total Downloads

Downloads (Last 12 months)176
Downloads (Last 6 weeks)20

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

He ZLi SSong YCai Z(2024)Towards Building Condition-Based Cross-Modality Intention-Aware Human-AI Cooperation under VR EnvironmentProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642360(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642360
He ZSong YZhou SCai Z(2023)Interaction of Thoughts: Towards Mediating Task Assignment in Human-AI Cooperation with a Capability-Aware Shared Mental ModelProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580983(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580983

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents