research-article

SmartHelm: Towards Multimodal Detection of Attention in an Outdoor Augmented Reality Biking Scenario

Authors:

Sromona Chatterjee,

Dennis Küster,

Johannes Schering,

Jorge Marx Gómez,

Tanja SchultzAuthors Info & Claims

ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction

Pages 426 - 432

https://doi.org/10.1145/3395035.3425207

Published: 27 December 2020 Publication History

Abstract

Driving and biking are complex and attention-demanding tasks for which distractions are a major safety hazard.

Modeling driving-related attention with regard to audio-visual distractions and assessing the attentional overload could help drivers to reduce stress and increase safety. In this work, we present a multimodal recording architecture using dry EEG-electrodes and the eye-tracking capability of the HoloLens 2 for an outdoor Augmented Reality (AR) scenario. The AR street scene contains visual distractions and moving cars and is shown to the subject in a simulation through the Hololens 2. The system records EEG and eye-tracking data to predict changes in the driver's attention. A preliminary case study is presented here to detail the data acquisition setup and to detect the occurrences of visual distractions in the simulation. Our first results suggest that this approach may overall be viable. However, further research is still required to refine our setup and models, as well as to evaluate the ability of the system to capture meaningful changes of attention in the field.

References

[1]

Josh Andres, m.c. schraefel, Nathan Semertzidis, Brahmi Dwivedi, Yutika C. Kulwe, Juerg von Kaenel, and Florian Floyd Mueller. 2020. Introducing Peripheral Awareness as a Neurological State for Human-computer Integration. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu HI USA, 2020-04--21). ACM, 1--13. https://doi.org/10.1145/3313831.3376128

Digital Library

[2]

Jürg Artho, S Schneider, and C Boss. 2012. Unaufmerksamkeit und Ablenkung: Was macht der Mensch am Steuer? (2012). https://doi.org/10.5167/UZH-66038 Publisher: Bundesamt für Strassen ASTRA.

[3]

Mathias Benedek, Rainer J. Schickel, Emanuel Jauk, Andreas Fink, and Aljoscha C. Neubauer. 2014. Alpha power increases in right parietal cortex reflects focused internal attention., Vol. 56 (2014), 393--400. https://doi.org/10.1016/j.neuropsychologia.2014.02.010

[4]

Mathias Benedek, Robert Stoiser, Sonja Walcher, and Christof Körner. 2017. Eye Behavior Associated with Internally versus Externally Directed Cognition., Vol. 8 (2017), 1092. https://doi.org/10.3389/fpsyg.2017.01092

[5]

Anne-Marie Brouwer, Maarten A. Hogervorst, Bob Oudejans, Anthony J. Ries, and Jonathan Touryan. 2017. EEG and Eye Tracking Signatures of Target Encoding during Structured Visual Search., Vol. 11 (2017), 264. https://doi.org/10.3389/fnhum.2017.00264

[6]

Zehong Cao, Chun-Hsiang Chuang, Jung-Kai King, and Chin-Teng Lin. 2019. Multi-channel EEG recordings during a sustained-attention driving task., Vol. 6, 1 (2019), 19. https://doi.org/10.1038/s41597-019-0027--4

[7]

Marvin M. Chun, Julie D. Golomb, and Nicholas B. Turk-Browne. 2011. A Taxonomy of External and Internal Attention., Vol. 62, 1 (2011), 73--101. https://doi.org/10.1146/annurev.psych.093008.100427

[8]

Arlene Ducao, Ilias Koen, and Zhiqi Guo. 2018. Multimer: validating multimodal, cognitive data in the city: towards a model of how the urban environment influences streetscape users. In Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data (Boulder Colorado, 2018--10--16). ACM, 1--8. https://doi.org/10.1145/3279810.3279853

Digital Library

[9]

Andrea Finke, Kai Essig, Giuseppe Marchioro, and Helge Ritter. 2016. Toward FRP-Based Brain-Machine Interfaces?Single-Trial Classification of Fixation-Related Potentials., Vol. 11, 1 (2016), e0146848. https://doi.org/10.1371/journal.pone.0146848

[10]

Nihan Karatas, Shintaro Tamura, Momoko Fushiki, and Michio Okado. 2018. Multi-party Conversation of Driving Agents: The Effects of Overhearing Information on Lifelikeness and Distraction. In Proceedings of the 6th International Conference on Human-Agent Interaction (Southampton, UK, December 2018). ACM, 84--91.

Digital Library

[11]

Ioanna Katidioti, Jelmer P. Borst, Douwe J. Bierens de Haan, Tamara Pepping, Marieke K. van Vugt, and Niels A. Taatgen. 2016. Interrupted by Your Pupil: An Interruption Management System Based on Pupil Dilation., Vol. 32, 10 (2016), 791--801. https://doi.org/10.1080/10447318.2016.1198525

[12]

Neale Kinnear and Alan Stevens. 2015. The battle for attention. Driver distraction -- A review of recent research knowledge. A report for Institute of Advanced Motorists (IAM), Vol. 30 (2015).

[13]

Josef F. Krems and Martin R. K. Baumann. 2009. Driving and Situation Awareness: A Cognitive Model of Memory-Update Processes. In Human Centered Design, Masaaki Kurosu (Ed.). Vol. 5619. Springer Berlin Heidelberg, 986--994. https://doi.org/10.1007/978--3--642-02806--9_113 Series Title: Lecture Notes in Computer Science.

[14]

Yulan Liang, William J. Horrey, and Joshua D. Hoffman. 2015. Reading Text While Driving: Understanding Drivers? Strategic and Tactical Adaptation to Distraction., Vol. 57, 2 (2015), 347--359. https://doi.org/10.1177/0018720814542974

[15]

Marie-Christin H. Oczko, Lukas Stratmann, Mario Franke, Julian Heinovski, Dominik S. Buse, Florian Klingler, and Falko Dressler. 2020. Integrating Haptic Signals with V2X-based Safety Systems for Vulnerable Road Users. In 2020 International Conference on Computing, Networking and Communications (ICNC) (Big Island, HI, USA, 2020-02). IEEE, 692--697. https://doi.org/10.1109/ICNC47757.2020.9049723

[16]

Marina Plavvsc, Heiner Bubb, Markus Duschul, Marcus Toennis, and Gundrun Klinker. 2009. Ergonomic Design and Evaluation of Augmented Reality Based Cautionary Warnings for Driving Assistance in Urban Environments. In Proceedings of 17th World Congress on Ergonomics (International Ergonomics Association, IEA) (Beijing, China).

[17]

Felix Putze, Jutta Hild, Rainer Kärgel, Christian Herff, Alexander Redmann, Jürgen Beyerer, and Tanja Schultz. 2013. Locating user attention using eye tracking and EEG for spatio-temporal event selection. In Proceedings of the 2013 international conference on Intelligent user interfaces - IUI '13 (Santa Monica, California, USA). ACM Press, 129. https://doi.org/10.1145/2449396.2449415

Digital Library

[18]

Felix Putze, Dennis Küster, Sonja Annerer-Walcher, and Mathias Benedek. 2018. Dozing Off or Thinking Hard?: Classifying Multi-dimensional Attentional States in the Classroom from Video. In Proceedings of the 2018 on International Conference on Multimodal Interaction - ICMI '18 (Boulder, CO, USA). ACM Press, 258--262. https://doi.org/10.1145/3242969.3243000

Digital Library

[19]

Felix Putze, Maximilian Scherer, and Tanja Schultz. 2016. Starring into the Void?: Classifying Internal vs. External Attention from EEG. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction (New York, NY, USA) (NordiCHI '16). ACM, 47:1--47:4. https://doi.org/10.1145/2971485.2971555

Digital Library

[20]

Lisa-Marie Vortmann, Felix Kroll, and Felix Putze. 2019. EEG-Based Classification of Internally- and Externally-Directed Attention in an Augmented Reality Paradigm. Frontiers in Human Neuroscience, Vol. 13 (2019). https://doi.org/10.3389/fnhum.2019.00348

[21]

Lisa-Marie Vortmann and Felix Putze. 2020. Attention-Aware Brain Computer Interface to Avoid Distractions in Augmented Reality. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu HI USA, 2020-04--25). ACM, 1--8. https://doi.org/10.1145/3334480.3382889

Digital Library

Cited By

Zhao GOrlosky JGabbard JKiyokawa K(2024)HazARdSnap: Gazed-Based Augmentation Delivery for Safe Information Access While CyclingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.333333630:9(6378-6389)Online publication date: Sep-2024
https://doi.org/10.1109/TVCG.2023.3333336
Nwagu CAlSlaity AOrji R(2023)EEG-Based Brain-Computer Interactions in Immersive Virtual and Augmented Reality: A Systematic ReviewProceedings of the ACM on Human-Computer Interaction10.1145/35932267:EICS(1-33)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593226
Kosch TMatviienko AMüller FBersch JKatins CSchön DMühlhäuser M(2022)NotiBike: Assessing Target Selection Techniques for Cyclist Notifications in Augmented RealityProceedings of the ACM on Human-Computer Interaction10.1145/35467326:MHCI(1-24)Online publication date: 20-Sep-2022
https://dl.acm.org/doi/10.1145/3546732
Show More Cited By

Index Terms

SmartHelm: Towards Multimodal Detection of Attention in an Outdoor Augmented Reality Biking Scenario

Index terms have been assigned to the content through auto-classification.

Recommendations

Attention-Aware Brain Computer Interface to Avoid Distractions in Augmented Reality
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

Recently, the idea of using BCIs in Augmented Reality settings to operate systems has emerged. One problem of such head-mounted displays is the distraction caused by an unavoidable display of control elements even when focused on internal thoughts. In ...
Multimodal augmented reality: the norm rather than the exception
MVAR '16: Proceedings of the 2016 workshop on Multimodal Virtual and Augmented Reality

Augmented reality (AR) is commonly seen as a technology that overlays virtual imagery onto a participant's view of the world. In line with this, most AR research is focused on what we see. In this paper, we challenge this focus on vision and make a case ...
Exploring head-up augmented reality interfaces for crash warning systems
AutomotiveUI '13: Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

Crash warning systems are designed to help avoid vehicle accidents by notifying drivers of potential hazards. In typical crash warning systems, primary warning information is provided through visual, audible and/or haptic cues. In general, the use of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction

October 2020

548 pages

ISBN:9781450380027

DOI:10.1145/3395035

General Chairs:
Khiet Truong
University of Twente, the Netherlands
,
Dirk Heylen
University of Twente, the Netherlands
,
Mary Czerwinski
Microsoft Research, USA
,
Program Chairs:
Nadia Berthouze
University College London, United Kingdom
,
Mohamed Chetouani
Sorbonne University, France
,
Mikio Nakano
C4A Research Institute, Japan

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 December 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '20

Sponsor:

SIGCHI

ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 25 - 29, 2020

Virtual Event, Netherlands

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
400
Total Downloads

Downloads (Last 12 months)71
Downloads (Last 6 weeks)10

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhao GOrlosky JGabbard JKiyokawa K(2024)HazARdSnap: Gazed-Based Augmentation Delivery for Safe Information Access While CyclingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.333333630:9(6378-6389)Online publication date: Sep-2024
https://doi.org/10.1109/TVCG.2023.3333336
Nwagu CAlSlaity AOrji R(2023)EEG-Based Brain-Computer Interactions in Immersive Virtual and Augmented Reality: A Systematic ReviewProceedings of the ACM on Human-Computer Interaction10.1145/35932267:EICS(1-33)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593226
Kosch TMatviienko AMüller FBersch JKatins CSchön DMühlhäuser M(2022)NotiBike: Assessing Target Selection Techniques for Cyclist Notifications in Augmented RealityProceedings of the ACM on Human-Computer Interaction10.1145/35467326:MHCI(1-24)Online publication date: 20-Sep-2022
https://dl.acm.org/doi/10.1145/3546732
Salous MKuster DScheck KDikfidan ANeumann TPutze FSchultz T(2022)SmartHelm: User Studies from Lab to Field for Attention Modeling2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC53654.2022.9945155(1012-1019)Online publication date: 9-Oct-2022
https://doi.org/10.1109/SMC53654.2022.9945155
Orji JHernandez ASelema BOrji R(2022)Virtual and Augmented Reality Applications for Promoting Safety and Security: A Systematic Review2022 IEEE 10th International Conference on Serious Games and Applications for Health(SeGAH)10.1109/SEGAH54908.2022.9978597(1-7)Online publication date: 10-Aug-2022
https://doi.org/10.1109/SEGAH54908.2022.9978597
Kurzhals KBecher MPathmanathan NReina G(2022)Evaluating Situated Visualization in AR with Eye Tracking2022 IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV)10.1109/BELIV57783.2022.00013(77-84)Online publication date: Oct-2022
https://doi.org/10.1109/BELIV57783.2022.00013
Küster DPutze FSt-Onge DFortin PUrrestilla NSchultz T(2021)3rd Workshop on Modeling Socio-Emotional and Cognitive Processes from Multimodal Data in the WildProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3480978(860-861)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3462244.3480978

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents