research-article

Impact of Agent Reliability and Predictability on Trust in Real Time Human-Agent Collaboration

Authors:

Sylvain Daronnat,

Leif Azzopardi,

Mateusz DubielAuthors Info & Claims

HAI '20: Proceedings of the 8th International Conference on Human-Agent Interaction

Pages 131 - 139

https://doi.org/10.1145/3406499.3415063

Published: 10 November 2020 Publication History

Abstract

Trust is a prerequisite for effective human-agent collaboration. While past work has studied how trust relates to an agent's reliability, it has been mainly carried out in turn based scenarios, rather than during real-time ones. Previous research identified the performance of an agent as a key factor influencing trust. In this work, we posit that an agent's predictability also plays an important role in the trust relationship, which may be observed based on users' interactions. We designed a 2x2 within-groups experiment with two baseline conditions: (1) no agent (users' individual performance), and (2) near-flawless agent (upper bound). Participants took part in an interactive aiming task where they had to collaborate with different agents that varied in terms of their predictability, and were controlled in terms of their performance. Our results show that agents whose behaviours are easier to predict have a more positive impact on task performance, reliance and trust while reducing cognitive workload. In addition, we modelled the human-agent trust relationship and demonstrated that it is possible to reliably predict users' trust ratings using real-time interaction data. This work seeks to pave the way for the development of trust-aware agents capable of adapting and responding more appropriately to users.

References

[1]

S. Almajdalawi, V. Pavlinek, M. Mrlik, Q. Cheng, and M. Sedlacik. 2016. The Influence of Virtual Agents on Player Experience and Performance. CHI PLAY (2016). https://doi.org/10.1088/1742--6596/412/1/012003

[2]

Alex Cao, Keshav K. Chintamani, Abhilash K. Pandya, and R. Darin Ellis. 2009. NASA TLX: Software for assessing subjective mental workload. Behavior Research Methods, Vol. 41, 1 (feb 2009), 113--117.

[3]

Alain Chavaillaz, Adrian Schwaninger, Stefan Michel, and Juergen Sauer. 2018. Automation in visual inspection tasks: X-ray luggage screening supported by a system of direct, indirect or adaptable cueing with low and high system reliability. Ergonomics, Vol. 61, 10 (Oct. 2018), 1395--1408.

[4]

Jessie Y. C. Chen, Shan G. Lakhmani, Kimberly Stowers, Anthony R. Selkowitz, Julia L. Wright, and Michael Barnes. 2018. Situation awareness-based agent transparency and human-autonomy teaming effectiveness. Theoretical Issues in Ergonomics Science, Vol. 19, 3 (feb 2018), 259--282.

[5]

Ting Chen, Duncan Campbell, Luis Felipe Gonzalez, and Gilles Coppin. 2015. Increasing Autonomy Transparency through capability communication in multiple heterogeneous UAV management. In IEEE International Conference on Intelligent Robots and Systems. https://doi.org/10.1109/IROS.2015.7353707

[6]

J Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. In Statistical Power Analysis for the Behavioral Sciences, 2nd Edition. Routledge.

[7]

Filipa Correia, Carla Guerra, Samuel Mascarenhas, Francisco S. Melo, and Ana Paiva. 2018. Exploring the Impact of Fault Justification in Human-Robot Trust. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (Stockholm, Sweden) (AAMAS '18). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 507--513.

Digital Library

[8]

S Costo and R Molfino. 2004. A new robotic unit for onboard airplanes bomb disposal. In 35th International symposium on robotics ISR, Vol. 2004. Citeseer, 23--26.

[9]

Ewart J. de Visser, Frank Krueger, Patrick McKnight, Steven Scheid, Melissa Smith, Stephanie Chalk, and Raja Parasuraman. 2012. The World is not Enough: Trust in Cognitive Agents. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 56, 1 (sep 2012), 263--267.

[10]

Mustafa Demir, Nathan J. McNeese, and Nancy J. Cooke. 2017. Team situation awareness within the context of human-autonomy teaming. Cognitive Systems Research, Vol. 46 (2017), 3--12. https://doi.org/10.1016/j.cogsys.2016.11.003

Digital Library

[11]

Katharina Emmerich, Patrizia Ring, and Maic Masuch. 2018. I'm Glad You Are on My Side: How to Design Compelling Game Companions. The Annual Symposium on Computer-Human Interaction in Play Extended Abstracts - CHI PLAY '18 (2018), 141--152.

Digital Library

[12]

Elliot E. Entin and Daniel Serfaty. 2017. Sequential Revision of Belief, Trust Type, and the Order Effect. Human Factors, Vol. 59, 3 (2017), 407--419. https://doi.org/10.1177/0018720816678322

[13]

Xiaocong Fan, Michael McNeese, Bingjun Sun, Timothy Hanratty, Laurel Allender, and John Yen. 2010. Human-Agent Collaboration for Time-Stressed Multicontext Decision Making. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, Vol. 40, 2 (mar 2010), 306--320.

Digital Library

[14]

Xiaocong Fan, Sooyoung Oh, Michael McNeese, John Yen, Haydee Cuevas, Laura Strater, and Mica R. Endsley. 2008. The influence of agent reliability on trust in human-agent collaboration. In Proceedings of the 15th European conference on Cognitive ergonomics: the ergonomics of cool interaction (2008). ACM, 7.

Digital Library

[15]

Amos Freedy, Ewart DeVisser, Gershon Weltman, and Nicole Coeyman. 2007. Measurement of trust in human-robot collaboration. Proceedings of the 2007 International Symposium on Collaborative Technologies and Systems, CTS (2007), 106--114.

[16]

Behnood Gholami, Wassim M. Haddad, and James M. Bailey. 2018. AI in the ICU: In the intensive care unit, artificial intelligence can keep watch. IEEE Spectrum, Vol. 55, 10 (oct 2018), 31--35. https://doi.org/10.1109/MSPEC.2018.8482421

Digital Library

[17]

F. S. Grodzinsky, K. W. Miller, and M. J. Wolf. 2011. Developing artificial agents worthy of trust: "Would you buy a used car from this artificial agent?". Ethics and Information Technology, Vol. 13, 1 (2011), 17--27. https://doi.org/10.1007/s10676-010--9255--1

Digital Library

[18]

Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904--908.

[19]

Jean-Michel Hoc, Mark S. Young, and Jean-Marc Blosseville. 2009. Cooperation between drivers and automation: implications for safety., Vol. 10, 2 (2009), 135--160.

[20]

Kevin Anthony Hoff and Masooda Bashir. 2015. Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, Vol. 57, 3 (2015), 407--434. https://doi.org/10.1177/0018720814547570

[21]

R. R. Hoffman, M. Johnson, J. M. Bradshaw, and A. Underbrink. 2013. Trust in Automation., Vol. 28, 1 (2013), 84--88.

[22]

Indrarnani L. Singh, Robert Molloy and Raja Parasuraman. 1993. Automation- Induced "Complacency": Development of the Complacency-Potential Rating Scale. The International Journal of Aviation Psychology, Vol. 3, October 2013 (1993), 37--41. https://doi.org/10.1207/s15327108ijap0302

[23]

Theodore Jensen, Mohammad Maifi Hasan Khan, Yusuf Albayram, Md Abdullah Al Fahim, Ross Buck, and Emil Coman. 2020. Anticipated Emotions in Initial Trust Evaluations of a Drone System Based on Performance and Process Information. International Journal of Human-Computer Interaction, Vol. 36, 4 (2020), 316--325.

[24]

Jiun-Yin Jian, Ann M. Bisantz, and Colin G. Drury. 2000. Foundations for an Empirically Determined Scale of Trust in Automated Systems. International Journal of Cognitive Ergonomics, Vol. 4, 1 (mar 2000), 53--71. https://doi.org/10.1207/S15327566IJCE0401_04

[25]

Daisuke Karikawa, Hisae Aoyama, Makoto Takahashi, Kazuo Furuta, Toshio Wakabayashi, and Masaharu Kitamura. 2013. A visualization tool of en route air traffic control tasks for describing controller's proactive management of traffic situations., Vol. 15, 2 (2013), 207--218.

[26]

Da-jung Kim and Youn-kyung Lim. 2019. Co-Performing Agent. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. ACM Press, New York, New York, USA, 1--14. https://doi.org/10.1145/3290605.3300714

[27]

G. Klien, David D. Woods, Jeffrey M. Bradshaw, Robert R. Hoffman, and Paul J. Feltovich. 2004. Ten challenges for making automation a "team player" in joint human-agent activity., Vol. 19, 6 (2004), 91--95.

[28]

Alexander Kunze, Stephen J. Summerskill, Russell Marshall, and Ashleigh J. Filtness. 2019. Automation transparency: implications of uncertainty communication for human-automation interaction and interfaces. Ergonomics, Vol. 62, 3 (2019), 345--360. https://doi.org/10.1080/00140139.2018.1547842

[29]

J. D. Lee and K. A. See. 2004. Trust in Automation: Designing for Appropriate Reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society, Vol. 46, 1 (jan 2004), 50--80.

[30]

Stephanie M. Merritt, Heather Heimbaugh, Jennifer LaChapell, and Deborah Lee. 2013. I Trust It, but I Don't Know Why. Human Factors: The Journal of the Human Factors and Ergonomics Society, Vol. 55, 3 (jun 2013), 520--534. https://doi.org/10.1177/0018720812465081

[31]

Stephanie M. Merritt, Deborah Lee, Jennifer L. Unnerstall, and Kelli Huber. 2015. Are well-calibrated users effective users? Associations between calibration of trust and performance on an automation-aided task. Human Factors, Vol. 57, 1 (2015), 34--47. https://doi.org/10.1177/0018720814561675

[32]

NASA. 1986. TASK LOAD INDEX, Nasa TLX, v1.0. NASA (1986).

[33]

Sherry Ogreten, Stephanie Lackey, and Denise Nicholson. 2010. Recommended roles for uninhabited team members within mixed-initiative combat teams. In 2010 International Symposium on Collaborative Technologies and Systems. IEEE.

[34]

Richard Pak, Nicole Fink, Margaux Price, Brock Bass, and Lindsay Sturre. 2012. Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults. Ergonomics, Vol. 55, 9 (July 2012), 1059--1072.

[35]

Kristin E. Schaefer, Jessie Y. C. Chen, James L. Szalma, and P. A. Hancock. 2016. A Meta-Analysis of Factors Influencing the Development of Trust in Automation. Human Factors: The Journal of the Human Factors and Ergonomics Society, Vol. 58, 3 (mar 2016), 377--400.

[36]

Tommaso Sgobba. 2019. B-737 MAX and the crash of the regulatory system. Journal of Space Safety Engineering, Vol. 6, 4 (2019), 299--303.

[37]

TB Sheridan. 1989. Trustworthiness of command and control systems. In Analysis, Design and Evaluation of Man--Machine Systems 1988. Elsevier, 427--431.

[38]

Hirokazu Shirado and Nicholas A. Christakis. 2017. Locally noisy autonomous agents improve global human coordination in network experiments. Nature, Vol. 545, 7654 (2017), 370--374. https://doi.org/10.1038/nature22332

[39]

Indramani L. Singh, Robert Molloy, and Raja Parasuraman. 1993. Automation- Induced "Complacency": Development of the Complacency-Potential Rating Scale. The International Journal of Aviation Psychology, Vol. 3, 2 (apr 1993), 111--122.

[40]

Alessandro Sordoni, Jean-Pierre Briot, Isabelle Alvarez, Eurico Vasconcelos, Marta de Azevedo Irving, and Gustavo Melo. 2010. Design of a Participatory Decision Making Agent Architecture Based on Argumentation and Influence Function - Application to a Serious Game about Biodiversity Conservation. RAIRO - Operations Research, Vol. 44, 4 (2010), 269--283. https://doi.org/10.1051/ro/2010024

[41]

Kimberly Stowers, Nicholas Kasdaglis, Michael Rupp, Jessie Chen, Daniel Barber, and Michael Barnes. 2017. Insights into human-agent teaming: Intelligent agent transparency and uncertainty. Vol. 499. 149--160 pages. https://doi.org/10.1007/978-3-319-41959-6_13

[42]

Trond A. Tjøstheim, Birger Johansson, and Christian Balkenius. 2019. A Computational Model of Trust-, Pupil-, and Motivation Dynamics. In Proceedings of the 7th International Conference on Human-Agent Interaction (Kyoto, Japan) (HAI '19). Association for Computing Machinery, New York, NY, USA, 179--185. https://doi.org/10.1145/3349537.3351896

Digital Library

[43]

Ning Wang, David V Pynadath, and Susan G Hill. 2015. The Impact of POMDP-Generated Explanations on Trust and Performance in Human-Robot Teams. Aamas Aamas (2015), 997--1005. http://delivery.acm.org.prx.library.gatech.edu/10.1145/2940000/2937071/p997-wang.pdf?ip=143.215.137.43{&}id=2937071{&}acc=ACTIVE SERVICE{&}key=A79D83B43E50B5B8.5E2401E94B5C98E0.4D4702B0C3E38B35.4D4702B0C3E38B35{&}CFID=828239253{&}CFTOKEN=84023669&__acm__=1510434143

[44]

Eric N. Wiebe, Allison Lamb, Megan Hardy, and David Sharek. 2014. Measuring engagement in video game-based environments: Investigation of the User Engagement Scale. Computers in Human Behavior, Vol. 32 (2014), 123--132. https://doi.org/10.1016/j.chb.2013.12.001

[45]

EJ Williams. 1949. Experimental Designs Balanced for the Estimation of Residual Effects of Treatments. Australian Journal of Chemistry, Vol. 2, 2 (1949), 149.

Cited By

Chang WEsmaeili BHasanzadeh S(2025)Impacts of Physical and Informational Failures on Worker–Autonomy Trust in Future ConstructionJournal of Construction Engineering and Management10.1061/JCEMD4.COENG-15241151:4Online publication date: Apr-2025
https://doi.org/10.1061/JCEMD4.COENG-15241
Sun NBotev J(2024)Digital Partnerships: Understanding Delegation and Interaction With Virtual AgentsNavigating the Metaverse - A Comprehensive Guide to the Future of Digital Interaction [Working Title]10.5772/intechopen.1006301Online publication date: 13-Aug-2024
https://doi.org/10.5772/intechopen.1006301
Cabanag MStanton CCass J(2024)Fighting Information Overload: Modelling, Predicting and Inducing Cognitive Fit in Human-Autonomy TeamingProceedings of the 12th International Conference on Human-Agent Interaction10.1145/3687272.3688305(204-213)Online publication date: 24-Nov-2024
https://dl.acm.org/doi/10.1145/3687272.3688305
Show More Cited By

Index Terms

Impact of Agent Reliability and Predictability on Trust in Real Time Human-Agent Collaboration
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Human-Agent Trust Relationships in a Real-Time Collaborative Game
CHI PLAY '20: Extended Abstracts of the 2020 Annual Symposium on Computer-Human Interaction in Play

Collaborative virtual agents are often deployed to help users make decisions in real-time. For this collaboration to work, users must adequately trust the agents that they are interacting with. In my research, I use a game where human-agent interactions ...
The influence of agent reliability on trust in human-agent collaboration
ECCE '08: Proceedings of the 15th European conference on Cognitive ergonomics: the ergonomics of cool interaction

Motivation - To investigate ways to support human-automation teams with real-world, imperfect automation where many system failures are the result of systematic failure.

Research approach - An experimental approach was used to investigate how variance ...
Comparing Human Trust Attitudes Towards Human and Agent Teammates
HAI '20: Proceedings of the 8th International Conference on Human-Agent Interaction

Agents' roles in our lives increasingly matter as they engage with people in a variety of important tasks. To achieve successful human-agent teamwork, it is critical to know the differences and similarities in people's attitudes towards human and agent ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HAI '20: Proceedings of the 8th International Conference on Human-Agent Interaction

November 2020

304 pages

ISBN:9781450380546

DOI:10.1145/3406499

General Chairs:
Mohammad Obaid
Chalmers University of Technology, Sweden
,
Omar Mubin
Western Sydney University, Australia
,
Yukie Nagai
The University of Tokyo, Japan
,
Program Chairs:
Hirotaka Osawa
University of Tsukuba, Japan
,
Yomna Abdelrahman
Bundeswehr University Munich, Germany
,
Morten Fjeld
Chalmers University of Technology, Sweden

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 November 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

HAI '20

Sponsor:

SIGCHI

HAI '20: 8th International Conference on Human-Agent Interaction

November 10 - 13, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 121 of 404 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
459
Total Downloads

Downloads (Last 12 months)103
Downloads (Last 6 weeks)15

Reflects downloads up to 02 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chang WEsmaeili BHasanzadeh S(2025)Impacts of Physical and Informational Failures on Worker–Autonomy Trust in Future ConstructionJournal of Construction Engineering and Management10.1061/JCEMD4.COENG-15241151:4Online publication date: Apr-2025
https://doi.org/10.1061/JCEMD4.COENG-15241
Sun NBotev J(2024)Digital Partnerships: Understanding Delegation and Interaction With Virtual AgentsNavigating the Metaverse - A Comprehensive Guide to the Future of Digital Interaction [Working Title]10.5772/intechopen.1006301Online publication date: 13-Aug-2024
https://doi.org/10.5772/intechopen.1006301
Cabanag MStanton CCass J(2024)Fighting Information Overload: Modelling, Predicting and Inducing Cognitive Fit in Human-Autonomy TeamingProceedings of the 12th International Conference on Human-Agent Interaction10.1145/3687272.3688305(204-213)Online publication date: 24-Nov-2024
https://dl.acm.org/doi/10.1145/3687272.3688305
Liu XLv MFan WSun XDong ZXiao F(2024)Reliability of Half Hypercube Networks under Cluster Faults2024 27th International Conference on Computer Supported Cooperative Work in Design (CSCWD)10.1109/CSCWD61410.2024.10580429(2153-2158)Online publication date: 8-May-2024
https://doi.org/10.1109/CSCWD61410.2024.10580429
Bhanu ASharma HPathy SPonathil ARahimian HMadathil K(2023)Trust in Artificial Intelligent Agent while Completing a Procedural Construction TaskProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/2169506723119366867:1(2005-2006)Online publication date: 20-Oct-2023
https://doi.org/10.1177/21695067231193668
Phutane MJung CChen NAzenkot S(2023)Speaking with My Screen Reader: Using Audio Fictions to Explore Conversational Access to InterfacesProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608404(1-18)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3608404
Doherty ESpencer CEloy LKumar NDickler RHirshfield L(2023)Using Speech Patterns to Model the Dimensions of Teamness in Human-Agent TeamsProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614121(640-648)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614121
Momose KWeekes TMehta RWright CMoukpe JEskridge T(2023)Patterns of Effective Human-Agent TeamsExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585608(1-13)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585608
Park SKim HPark JLee Y(2023)Designing and Evaluating User Experience of an AI-Based Defense SystemIEEE Access10.1109/ACCESS.2023.332925711(122045-122056)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3329257
Wang KWu JSun YChen JPu YQi Y(2023)Trust in Human and Virtual Live Streamers: The Role of Integrity and Social PresenceInternational Journal of Human–Computer Interaction10.1080/10447318.2023.227941040:23(8274-8294)Online publication date: 21-Nov-2023
https://doi.org/10.1080/10447318.2023.2279410
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten