AdaptiveVoice: Cognitively Adaptive Voice Interface for Driving Assistance
Authors: 
Shaoyue Wen, Songming Ping, Jialin Wang, Hai-Ning Liang, Xuhai Xu, Yukang YanAuthors Info & Claims
Shaoyue Wen, Songming Ping, Jialin Wang, Hai-Ning Liang, Xuhai Xu, Yukang YanAuthors Info & ClaimsArticle No.: 253, Pages 1 - 18
Abstract
Current voice assistants present messages in a predefined format without considering users’ mental states. This paper presents an optimization-based approach to alleviate this issue which adjusts the level of details and speech speed of the voice messages according to the estimated cognitive load of the user. In the first user study (N = 12), we investigated the impact of cognitive load on user performance. The findings reveal significant differences in preferred message formats across five cognitive load levels, substantiating the need for voice message adaptation. We then implemented AdaptiveVoice, an algorithm based on combinatorial optimization to generate adaptive voice messages in real time. In the second user study (N = 30) conducted in a VR-simulated driving environment, we compare AdaptiveVoice with a fixed format baseline, with and without visual guidance on the Heads-up display (HUD). Results indicate that users benefit from AdaptiveVoice with reduced response time and improved driving performance, particularly when it is augmented with HUD.
Supplemental Material
- Download
- 30.71 MB
- Transcript
- Download
- 285.39 MB
- Transcript
References
[1]
Ignacio Alvarez, Miren Karmele López-de Ipiña, and Juan E. Gilbert. 2012. The Voice User Help, a Smart Vehicle Assistant for the Elderly(UCAmI’12). Springer-Verlag, Berlin, Heidelberg, 314–321. https://doi.org/10.1007/978-3-642-35377-2_43
[2]
Aurélien Appriou, Andrzej Cichocki, and Fabien Lotte. 2018. Towards robust neuroadaptive HCI: exploring modern machine learning methods to estimate mental workload from EEG signals. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 1–6.
[3]
David Beattie, Lynne Baillie, Martin Halvey, and Rod McCall. 2014. What’s around the corner? Enhancing driver awareness in autonomous vehicles via in-vehicle spatial auditory displays. In Proceedings of the 8th nordic conference on human-computer interaction: fun, fast, foundational. 189–198.
[4]
Michael Braun, Anja Mainz, Ronee Chadowitz, Bastian Pfleging, and Florian Alt. 2019. At Your Service: Designing Voice Assistant Personalities to Improve Automotive User Interfaces. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300270
[5]
Yifei Cheng, Yukang Yan, Xin Yi, Yuanchun Shi, and David Lindlbauer. 2021. SemanticAdapt: Optimization-Based Adaptation of Mixed Reality Layouts Leveraging Virtual-Physical Semantic Connections. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 282–297. https://doi.org/10.1145/3472749.3474750
[6]
Yi Fei Cheng, Hang Yin, Yukang Yan, Jan Gugenheimer, and David Lindlbauer. 2022. Towards Understanding Diminished Reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 549, 16 pages. https://doi.org/10.1145/3491102.3517452
[7]
Rebecca Currano, So Yeon Park, Dylan James Moore, Kent Lyons, and David Sirkin. 2021. Little road driving hud: Heads-up display complexity influences drivers’ perceptions of automated vehicles. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–15.
[8]
Fred D Davis. 1985. A technology acceptance model for empirically testing new end-user information systems: Theory and results. Ph. D. Dissertation. Massachusetts Institute of Technology.
[9]
Tiffany D. Do, Ryan P. McMahan, and Pamela J. Wisniewski. 2022. A New Uncanny Valley? The Effects of Speech Fidelity and Human Listener Gender on Social Perceptions of a Virtual-Human Speaker(CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 424, 11 pages. https://doi.org/10.1145/3491102.3517564
[10]
Andrew T. Duchowski, Krzysztof Krejtz, Nina A. Gehrer, Tanya Bafna, and Per Bækgaard. 2020. The Low/High Index of Pupillary Activity(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376394
[11]
Andrew T Duchowski, Krzysztof Krejtz, Izabela Krejtz, Cezary Biele, Anna Niedzielska, Peter Kiefer, Martin Raubal, and Ioannis Giannopoulos. 2018. The index of pupillary activity: Measuring cognitive load vis-à-vis task difficulty with pupil oscillation. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–13.
[12]
Marie Eckert, Emanuël A. P. Habets, and Olli S. Rummukainen. 2021. Cognitive Load Estimation Based on Pupillometry in Virtual Reality with Uncontrolled Scene Lighting. In 2021 13th International Conference on Quality of Multimedia Experience (QoMEX). 73–76. https://doi.org/10.1109/QoMEX51781.2021.9465417
[13]
Roshan Fernandes, Arjun Gaonkar, Pratheek Shenoy, Anisha Rodrigues, Mohan A., and Vijaya Padmanabha. 2021. Efficient Virtual Reality-Based Platform for Virtual Concerts. 148–164. https://doi.org/10.4018/978-1-7998-4703-8.ch008
[14]
Lex Fridman, Bryan Reimer, Bruce Mehler, and William T Freeman. 2018. Cognitive load estimation in the wild. In Proceedings of the 2018 chi conference on human factors in computing systems. 1–9.
[15]
Anna-Katharina Frison, Philipp Wintersberger, Tianjia Liu, and Andreas Riener. 2019. Why do you like to drive automated? a context-dependent analysis of highly automated driving to elaborate requirements for intelligent user interfaces. In Proceedings of the 24th international conference on intelligent user interfaces. 528–537.
[16]
David Goedicke, Jamy Li, Vanessa Evers, and Wendy Ju. 2018. Vr-oom: Virtual reality on-road driving simulation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–11.
[17]
John F Golding. 1998. Motion sickness susceptibility questionnaire revised and its relationship to other forms of sickness. Brain Research Bulletin 47, 5 (1998), 507–516. https://doi.org/10.1016/S0361-9230(98)00091-4
[18]
John F Golding. 1998. Motion sickness susceptibility questionnaire revised and its relationship to other forms of sickness. Brain research bulletin 47, 5 (1998), 507–516.
[19]
Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.
[20]
S. G. Hart and L. E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Advances in Psychology 52, 6 (1988), 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9.
[21]
Nina Hollender, Cristian Hofmann, Michael Deneke, and Bernhard Schmitz. 2010. Integrating cognitive load theory and concepts of human–computer interaction. Computers in Human Behavior 26, 6 (2010), 1278–1288. https://doi.org/10.1016/j.chb.2010.05.031.
[22]
Jizhou Huang, Haifeng Wang, Shiqiang Ding, and Shaolei Wang. 2022. DuIVA: An Intelligent Voice Assistant for Hands-free and Eyes-free Voice Interaction with the Baidu Maps App. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3040–3050.
[23]
Jonathan Huyghe, Jan Derboven, and Dirk De Grooff. 2014-10-01. ALADIN: Adaptive Voice Interface for People with Disabilities.
[24]
Pascal Jansen, Julian Britten, Alexander Häusele, Thilo Segschneider, Mark Colley, and Enrico Rukzio. 2023. AutoVis: Enabling Mixed-Immersive Analysis of Automotive User Interface Interaction Studies. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–23.
[25]
Jingun Jung, Sangyoon Lee, Jiwoo Hong, Eunhye Youn, and Geehyuk Lee. 2020. Voice+ tactile: Augmenting in-vehicle voice user interface with tactile touchpad interaction. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.
[26]
Mohamed Kari, Tobias Grosse-Puppendahl, Alexander Jagaciak, David Bethge, Reinhard Schütte, and Christian Holz. 2021. SoundsRide: Affordance-Synchronized Music Mixing for In-Car Audio Augmented Reality. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 118–133. https://doi.org/10.1145/3472749.3474739
[27]
A Donald Keedwell and József Dénes. 2015. Latin squares and their applications. Elsevier.
[28]
Bret Kinsella and Ava Mutchler. 2019. In-car voice assistant consumer adoption report.
[29]
Patrick Langdon, Ioannis Politis, Mike Bradley, Lee Skrypchuk, Alex Mouzakitis, and John Clarkson. 2018. Obtaining design requirements from the public understanding of driverless technology. Springer, 749–759.
[30]
David Lindlbauer, Anna Maria Feit, and Otmar Hilliges. 2019. Context-Aware Online Adaptation of Mixed Reality Interfaces. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 147–160. https://doi.org/10.1145/3332165.3347945
[31]
Diane J Litman and Shimei Pan. 2002. Designing and evaluating an adaptive spoken dialogue system. User Modeling and User-Adapted Interaction 12, 2 (2002), 111–137.
[32]
Michal Luria, Guy Hoffman, and Oren Zuckerman. 2017. Comparing social robot, screen and voice interfaces for smart-home control. In Proceedings of the 2017 CHI conference on human factors in computing systems. 580–628.
[33]
Kirti Mahajan, David R Large, Gary Burnett, and Nagendra R Velaga. 2021. Exploring the benefits of conversing with a digital voice assistant during automated driving: A parametric duration model of takeover time. Transportation research part F: traffic psychology and behaviour 80 (2021), 104–126.
[34]
Florian Mathis, Kami Vaniea, and Mohamed Khamis. 2021. Replicueauth: Validating the use of a lab-based virtual reality setup for evaluating authentication systems. In Proceedings of the 2021 chi conference on human factors in computing systems. 1–18.
[35]
Florian Mathis, Kami Vaniea, and Mohamed Khamis. 2022. Can I Borrow Your ATM? Using Virtual Reality for (Simulated) In Situ Authentication Research. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 301–310. https://doi.org/10.1109/VR51125.2022.00049
[36]
Florian Mathis, John Williamson, Kami Vaniea, and Mohamed Khamis. 2020. Rubikauth: Fast and secure authentication in virtual reality. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–9.
[37]
Oussama Metatla, Alison Oldfield, Taimur Ahmed, Antonis Vafeas, and Sunny Miglani. 2019. Voice user interfaces in schools: Co-designing for inclusion with visually-impaired and sighted pupils. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–15.
[38]
Chelsea M. Myers, Anushay Furqan, and Jichen Zhu. 2019. The Impact of User Characteristics and Preferences on Performance with an Unfamiliar Voice User Interface(CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–9. https://doi.org/10.1145/3290605.3300277
[39]
Chihab Nadri, Seul Chan Lee, Siddhant Kekal, Yinjia Li, Xuan Li, Pasi Lautala, David Nelson, and Myounghoon Jeon. 2021. Effects of auditory display types and acoustic variables on subjective driver assessment in a rail crossing context. Transportation research record 2675, 9 (2021), 1457–1468.
[40]
Chihab Nadri, Siddhant Kekal, Yinjia Li, Xuan Li, Seul Chan Lee, David Nelson, Pasi Lautala, and Myounghoon Jeon. 2023. “Slow down. Rail crossing ahead. Look left and right at the crossing”: In-vehicle auditory alerts improve driver behavior at rail crossings. Applied ergonomics 106 (2023), 103912.
[41]
Jakob Nielsen. 1994. Usability engineering. Morgan Kaufmann.
[42]
Ronald K Pearson, Yrjö Neuvo, Jaakko Astola, and Moncef Gabbouj. 2016. Generalized hampel filters. EURASIP Journal on Advances in Signal Processing 2016, 1 (2016), 1–18.
[43]
S.M. Sarala, D.H. Sharath Yadav, and Asadullah Ansari. 2018. Emotionally Adaptive Driver Voice Alert System for Advanced Driver Assistance System (ADAS) Applications. Association for Computing Machinery, Tirunelveli, India. https://doi.org/10.1109/ICSSIT.2018.8748541
[44]
Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2021. MReD: A Meta-Review Dataset for Controllable Text Generation. arXiv preprint arXiv:2110.07474 (2021).
[45]
Yangming Shi, Jing Du, Eric Ragan, Kunhee Choi, and Shuo Ma. 2018. Social Influence on Construction Safety Behaviors: A Multi-User Virtual Reality Experiment. 174–183. https://doi.org/10.1061/9780784481288.018
[46]
EH Siegel, J Wei, A Gomes, M Oliviera, P Sundaramoorthy, K Smathers, M Vankipuram, S Ghosh, H Horii, J Bailenson, 2021. HP Omnicept Cognitive Load Database (HPO-CLD)–Developing a Multimodal Inference Engine for Detecting Real-time Mental Workload in VR. Technical Report. Technical report, HP Labs, Palo Alto.
[47]
Gustavo Silvera, Abhijat Biswas, and Henny Admoni. 2022. DReye VR: Democratizing Virtual Reality Driving Simulation for Behavioural & Interaction Research. In 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 639–643.
[48]
Hao Tan, Yaqi Zhou, Ruixiang Shen, Xiantao Chen, Xuning Wang, Moli Zhou, Daisong Guan, and Qin Zhang. 2019. A classification framework based on driver’s operations of in-car interaction. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings. 104–108.
[49]
MinJuan Wang, Sus Lundgren Lyckvi, and Fang Chen. 2016. Why and How Traffic Safety Cultures Matter When Designing Advisory Traffic Information Systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(CHI ’16). Association for Computing Machinery, New York, NY, USA, 2808–2818. https://doi.org/10.1145/2858036.2858467
[50]
Minghui Wang, Bi Zeng, and Qiujie Wang. 2021. Study of Motion Control and a Virtual Reality System for Autonomous Underwater Vehicles. Algorithms 14, 3 (2021). https://doi.org/10.3390.a14030093
[51]
Yukang Yan, Haohua Liu, Yingtian Shi, Jingying Wang, Ruici Guo, Zisu Li, Xuhai Xu, Chun Yu, Yuntao Wang, and Yuanchun Shi. 2023. ConeSpeech: Exploring Directional Speech Interaction for Multi-Person Remote Communication in Virtual Reality. IEEE Transactions on Visualization and Computer Graphics 29, 5 (2023), 2647–2657. https://doi.org/10.1109/TVCG.2023.3247085
[52]
Jiahong Yuan, Mark Liberman, and Christopher Cieri. 2006. Towards an integrated understanding of speaking rate in conversation. In Ninth International Conference on Spoken Language Processing.
[53]
NI Mohd Zaki, SM Che Husin, MK Abu Husain, N Abu Husain, A Ma’aram, SN Amilah Marmin, AF Adanan, Y Ahmad, and KA Abu Kassim. 2021. Auditory alert for in-vehicle safety technologies: a review. Journal of the Society of Automotive Engineers Malaysia 5, 1 (2021), 88–102.
[54]
Xin Zou, Steve O’Hern, Barrett Ens, Selby Coxon, Pascal Mater, Raymond Chow, Michael Neylan, and Hai L Vu. 2021. On-road virtual reality autonomous vehicle (VRAV) simulator: An empirical study on user experience. Transportation Research Part C: Emerging Technologies 126 (2021), 103090.
Index Terms
- AdaptiveVoice: Cognitively Adaptive Voice Interface for Driving Assistance
Recommendations
Adaptive Interfaces in Driving
FAC '09: Proceedings of the 5th International Conference on Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience: Held as Part of HCI International 2009The automotive domain is an excellent domain for investigating augmented cognition methods, and one of the domains that can provide the applications. We developed, applied and tested indirect (or derived) measures to estimate driver state risks, ...
The impact of an adaptive user interface on reducing driver distraction
AutomotiveUI '11: Proceedings of the 3rd International Conference on Automotive User Interfaces and Interactive Vehicular ApplicationsThis paper discusses the impact of an adaptive prototype in-car communication system (ICCS), called MIMI (Multimodal Interface for Mobile Info-communication), on driver distraction. Existing ICCSs attempt to minimise the visual and manual distraction, ...
Attuning in-car user interfaces to the momentary cognitive load
FAC'07: Proceedings of the 3rd international conference on Foundations of augmented cognitionCars, trucks and busses are more and more equipped with functions and services that drivers are supposed to operate and understand. The most important developments in this area are the Advanced Driver Assistance Systems (ADAS) and In Vehicle Information ...
Comments
Information & Contributors
Information
Published In
May 2024
18961 pages
ISBN:9798400703300
DOI:10.1145/3613904
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
Acceptance Rates
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%
Upcoming Conference
CHI 2025
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 1,267Total Downloads
- Downloads (Last 12 months)1,267
- Downloads (Last 6 weeks)164
Reflects downloads up to 24 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format