research-article

Experimental Investigation of Human Acceptance of AI Suggestions with Heatmap and Pointing-based XAI

Authors:

Akihiro Maehigashi,

Yosuke Fukuchi,

Seiji YamadaAuthors Info & Claims

HAI '23: Proceedings of the 11th International Conference on Human-Agent Interaction

Pages 291 - 298

https://doi.org/10.1145/3623809.3623834

Published: 04 December 2023 Publication History

Abstract

This study investigated how displaying an AI attention heatmap influences human acceptance of the AI’s suggestions in accordance with the interpretability of the heatmap. We conducted an experiment using a visual task where the participants were required to decide whether to accept or reject an AI’s suggestions. The participants could see the suggestions with an AI attention heatmap, the heatmap with the AI pointing to it (displayed as a laser dot cursor), the heatmap with a robot pointing (a robot using a stick to point to the AI heatmap displayed on a tablet), or no heatmap. The experimental results revealed that human acceptance of AI suggestions differed depending on the interpretability of the heatmap, especially when the heatmap was displayed with AI pointing. Also, additional analysis revealed an effect on acceptance due to the AI pointing to the heatmap that was found only in a high-task difficulty situation. An AI pointing to its attention heatmap is considered to trigger people to reason about particular AI processes and accept its suggestions. This study showed that an AI pointing to its attention heatmap could be used to control human behaviors in human-agent interaction.

References

[1]

Arjun R. Akula, Keze Wang, Changsong i Saba-Sadiya, Hongjing Lu, Sinisa Todorovic, Joyce Chai, and Song-Chun Zhu. 2022. CX-ToM: Counterfactual explanations with theory-of-mind for enhancing human trust in image recognition models. iScience 25, 1 (2022), 103581. https://doi.org/10.1016/j.isci.2021.103581

[2]

Anthony L. Baker, Elizabeth K. Phillips, Daniel Ullman, and Joseph R. Keebler. 2018. Toward an Understanding of Trust Repair in Human-Robot Interaction: Current Research and Future Directions. ACM Transactions on Interactive Intelligent Systems 8, 4 (2018), 1–30. https://doi.org/10.1145/3181671

Digital Library

[3]

Francisco Maria Calisto, João Fernandes, Margarida Morais, Carlos Santiago, João Maria Abrantes, Nuno Nunes, and Jacinto C. Nascimento. 2023. Assertiveness-based Agent Communication for a Personalized Medicine on Medical Imaging Diagnosis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems(CHI ’23). Association for Computing Machinery, New York, NY, 1–20. https://doi.org/10.1145/3544548.3580682

Digital Library

[4]

Francisco Maria Calisto, Carlos Santiago, Nuno Nunes, and Jacinto C. Nascimento. 2022. BreastScreening-AI: Evaluating Medical Intelligent Agents for Human-AI Interactions. Artificial Intelligence In Medicine 127, 102285 (2022). https://doi.org/10.1016/j.artmed.2022.102285

Digital Library

[5]

Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. 2018. Grad-CAM + +: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) (2018-03). IEEE. https://doi.org/10.1109/wacv.2018.00097

[6]

Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General 144, 1 (2015), 114–126. https://doi.org/10.1037/xge0000033

[7]

Mary T. Dzindolet, Scott A. Peterson, Regina A. Pomranky, and Linda Pierce. 2003. The Role of Trust in Automation Reliance. International Journal of Human-Computer Studies 58, 6 (2003), 697–718. https://doi.org/10.1016/S1071-5819(03)00038-7

Digital Library

[8]

Mary T. Dzindolet, Linda G. Pierce, Hall P. Beck, and Lloyd A. Dawe. 2002. The Perceived Utility of Human and Automated Aids in a Visual Detection Task. Human Factors 44, 1 (2002), 79–94. https://doi.org/10.1518/0018720024494856

[9]

Martha J. Farah. 1996. Is face recognition special? Evidence from neuropsychology. Behavioural Brain Research 76, 1-2 (1996), 181–189. https://doi.org/10.1016/0166-4328(95)00198-0

[10]

Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods 39, 2 (2007), 175–191. https://doi.org/10.3758/bf03193146

[11]

Reza Ghoddoosian, Marnim Galib, and Vassilis Athitsos. 2019. A Realistic Dataset and Baseline Temporal Model for Early Drowsiness Detection. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2019-06). 178–187. https://doi.org/10.1109/cvprw.2019.00027

[12]

David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Guang Zhong Yang. 2019. XAI–Explainable artificial intelligence. Science Robotics 4, 37 (2019). https://doi.org/10.1126/scirobotics.aay7120

[13]

David Gunning, Eric Vorm, Jennifer Yunyan Wang, and Matt Turek. 2021. DARPA’s explainable AI (XAI) program: A retrospective. Applied AI Letters 2, 4 (2021). https://doi.org/10.1002/ail2.61

Digital Library

[14]

Tatsuya Ishino, Mitsuhiro Goto, and Akihiro Kashihara. [n. d.]. A Robot for Reconstructing Presentation Behavior in Lecture. In Proceedings of the 6th International Conference on Human-Agent Interaction (2018-12). ACM. https://doi.org/10.1145/3284432.3284460

Digital Library

[15]

Ujwal Kayande, Arnaud De Bruyn, Gary L. Lilien, Arvind Rangaswamy, and Gerrit H. van Bruggen. 2009. How Incorporating Feedback Mechanisms in a DSS Affects DSS Evaluations. Information Systems Research 20, 4 (2009), 527–546. https://doi.org/10.1287/isre.1080.0198

Digital Library

[16]

Sunnie S. Y. Kim, Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández. 2023. "Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems(CHI ’23). Association for Computing Machinery, New York, NY, 1–17. https://doi.org/10.1145/3544548.3581001

Digital Library

[17]

Komiak and Benbasat. 2006. The Effects of Personalization and Familiarity on Trust and Adoption of Recommendation Agents. MIS Quarterly 30, 4 (2006), 941. https://doi.org/10.2307/25148760

[18]

John Lee and Katrina See. 2004. Trust in Automation: Designing for Appropriate Reliance. Human Factors 46, 1 (2004), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392

[19]

Jingyi Lu, Yiming Liang, and Hebing Duan. 2017. Justifying Decisions. Social Psychology 48, 2 (2017), 92–103. https://doi.org/10.1027/1864-9335/a000302

[20]

Akihiro Maehigashi, Yosuke Fukuchi, and Seiji Yamada. 2023. Empirical investigation of how robot’s pointing gesture influences trust in and acceptance of heatmap-based XAI. In Proceedings of the 32nd IEEE International Symposium on Robot and Human Interactive Communication(RO-MAN ’23).

[21]

Akihiro Maehigashi, Yosuke Fukuchi, and Seiji Yamada. 2023. Modeling Reliance on XAI Indicating Its Purpose and Attention. In Proceedings of the 45th Annual Meeting of the Cognitive Science Society(CogSci ’23). 1929–1936. https://escholarship.org/uc/item/1fx742xm

[22]

Hasan Mahmud, A.K.M. Najmul Islam, Syed Ishtiaque Ahmed, and Kari Smolander. 2021. What influences algorithmic decision-making? A systematic literature review on algorithm aversion. Technological Forecasting and Social Change 175 (2021), 121390. https://doi.org/10.1016/j.techfore.2021.121390

[23]

Arjun Mani, Nobline Yoo, Will Hinthorn, and Olga Russakovsky. 2022. Point and Ask: Incorporating Pointing into Visual Question Answering. arXiv:2011.13681v4 (2022), 12 pages. https://doi.org/10.48550/arXiv.2011.13681

[24]

Michael R. Maniaci and Ronald D. Rogge. 2014. Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality 48 (2014), 61–83. https://doi.org/10.1016/j.jrp.2013.09.008

[25]

Mohammad H. Rezazade Mehrizi, Ferdinand Mol, Marcel Peter, Erik Ranschaert, Daniel Pinto Dos Santos, Ramin Shahidi, Mansoor Fatehi, and Thomas Dratsch. 2023. The Impact of AI Suggestions on Radiologists’ Decisions: A Pilot Study of Explainability and Attitudinal Priming Interventions in Mammography Examination. Scientific Reports 13, 9230(2023) (2023). https://doi.org/10.1038/s41598-023-36435-3

[26]

Kazuo Okamura and Seiji Yamada. 2020. Empirical Evaluations of Framework for Adaptive Trust Calibration in Human-AI Cooperation. IEEE Access (2020), 1–1. https://doi.org/10.1109/access.2020.3042556

[27]

Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. [n. d.]. Multimodal Explanations: Justifying Decisions and Pointing to the Evidence. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018-06). IEEE. https://doi.org/10.1109/cvpr.2018.00915

[28]

Wolter Pieters. 2011. Explanation and trust: what to tell the user in security and AI?Ethics and Information Technology 1, 13 (2011), 53–64. https://doi.org/10.1007/s10676-010-9253-3

Digital Library

[29]

Arijit Ray, Michael Cogswell, Xiao Lin, Kamran Alipour, Ajay Divakaran, Yi Yao, and Giedrius Burachas. 2021. Knowing What VQA Does Not: Pointing to Error-Inducing Regions to Improve Explanation Helpfulness. arXiv:2103.14712v3 (2021), 10 pages. https://doi.org/10.48550/arXiv.2103.14712

[30]

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV2017) (2017-10). https://doi.org/10.1109/iccv.2017.74

[31]

Daniel Ullman and Bertram F Malle. 2019. Measuring Gains and Losses in Human-Robot Trust: Evidence for Differentiable Components of Trust. In Proceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction(HRI ’19). 618–619. https://doi.org/10.1109/HRI.2019.8673154

[32]

Weiquan Wang and Izak Benbasat. 2007. Recommendation agents for electronic commerce: Effects of explanation facilities on trusting beliefs. Journal of Management Information Systems 4, 23 (2007), 217–246. https://doi.org/10.2753/MIS0742-1222230410

Digital Library

[33]

Xun Wang, Mary-Anne Williams, Peter Gardenfors, Jonathan Vitale, Shaukat Abidi, Benjamin Johnston, Benjamin Kuipers, and Alan Huang. 2014. Directing human attention with pointing. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication (2014-08). IEEE. https://doi.org/10.1109/roman.2014.6926249

[34]

Douglas A. Wiegmann, Aaron Rich, and Hui Zhang. 2001. Automated Diagnostic Aids: The Effects of Aid Reliability on Users’ Trust and Reliance. Theoretical Issues in Ergonomics Science 2, 4 (2001), 352–367. https://doi.org/10.1080/14639220110110306

Cited By

Gajcin JJeromela JDusparic I(2024)Semifactual Explanations for Reinforcement LearningProceedings of the 12th International Conference on Human-Agent Interaction10.1145/3687272.3688324(167-175)Online publication date: 24-Nov-2024
https://dl.acm.org/doi/10.1145/3687272.3688324
Alharbi RLor PHerskovitz JSchoenebeck SBrewer R(2024)Misfitting With AI: How Blind People Verify and Contest AI ErrorsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675659(1-17)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675659
Müller R(2024)How Explainable AI Affects Human Performance: A Systematic Review of the Behavioural Consequences of Saliency MapsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2381929(1-32)Online publication date: 29-Jul-2024
https://doi.org/10.1080/10447318.2024.2381929

Index Terms

Experimental Investigation of Human Acceptance of AI Suggestions with Heatmap and Pointing-based XAI
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI

Recommendations

The Tower of Babel in Explainable Artificial Intelligence (XAI)
Machine Learning and Knowledge Extraction
Abstract
As machine learning (ML) has emerged as the predominant technological paradigm for artificial intelligence (AI), complex black box models such as GPT-4 have gained widespread adoption. Concurrently, explainable AI (XAI) has risen in significance ...
Human attention guided explainable artificial intelligence for computer vision models
Abstract
Explainable artificial intelligence (XAI) has been increasingly investigated to enhance the transparency of black-box artificial intelligence models, promoting better user understanding and trust. Developing an XAI that is faithful to models and ...
Highlights
- Human attention guided XAI is proposed for more faithful and plausible explanations.
- Two gradient-based XAI methods are presented for explaining object detection models.
- Human attention is adopted as an objective plausibility ...
Explainable Artificial Intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions
Abstract
Understanding black box models has become paramount as systems based on opaque Artificial Intelligence (AI) continue to flourish in diverse real-world applications. In response, Explainable AI (XAI) has emerged as a field of research with ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

HAI '23: Proceedings of the 11th International Conference on Human-Agent Interaction

December 2023

506 pages

ISBN:9798400708244

DOI:10.1145/3623809

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

JST CREST (JPMJCR21D4), Japan

Conference

HAI '23

HAI '23: International Conference on Human-Agent Interaction

December 4 - 7, 2023

Gothenburg, Sweden

Acceptance Rates

Overall Acceptance Rate 121 of 404 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
129
Total Downloads

Downloads (Last 12 months)107
Downloads (Last 6 weeks)2

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gajcin JJeromela JDusparic I(2024)Semifactual Explanations for Reinforcement LearningProceedings of the 12th International Conference on Human-Agent Interaction10.1145/3687272.3688324(167-175)Online publication date: 24-Nov-2024
https://dl.acm.org/doi/10.1145/3687272.3688324
Alharbi RLor PHerskovitz JSchoenebeck SBrewer R(2024)Misfitting With AI: How Blind People Verify and Contest AI ErrorsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675659(1-17)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675659
Müller R(2024)How Explainable AI Affects Human Performance: A Systematic Review of the Behavioural Consequences of Saliency MapsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2381929(1-32)Online publication date: 29-Jul-2024
https://doi.org/10.1080/10447318.2024.2381929

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents