research-article

Multi-modal Multi-scale Attention Guidance in Cyber-Physical Environments

Authors:

Guillermo Reyes,

Alexandra AllesAuthors Info & Claims

IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

Pages 356 - 365

https://doi.org/10.1145/3397481.3450678

Published: 14 April 2021 Publication History

Abstract

This work proposes a new method for guiding a user’s attention towards objects of interest in a cyber-physical environment (CPE). CPEs are environments that contain several computing systems that interact with each other and with the physical world. These environments contain several sensors (cameras, eye trackers, etc.) and output devices (lamps, screens, speakers, etc.). These devices can be used to first track the user’s position, orientation, and focus of attention to then find the most suitable output device to guide the user’s attention towards a target object. We argue that the most suitable device in this context is the one that attracts attention closest to the target and is salient enough to capture the user’s attention. The method is implemented as a function which estimates the ”closeness” and ”salience” of each visual and auditive output device in the environment. Some parameters of this method are then evaluated through a user study in the context of a virtual reality supermarket. The results show that multi-modal guidance can lead to better guiding performance. However, this depends on the set parameters.

References

[1]

Reynold J. Bailey, Ann McNamara, Nisha Sudarsanam, and Cindy Grimm. 2009. Subtle gaze direction. ACM Trans. Graph. 28, 4 (2009), 100:1–100:14.

Digital Library

[2]

Carryl L. Baldwin, Charles Spence, James P. Bliss, J. Christopher Brill, Michael S. Wogalter, Christopher B. Mayhorn, and Thomas K. Ferris. 2012. Multimodal Cueing: The Relative Benefits of the Auditory, Visual, and Tactile Channels in Complex Environments. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 56, 1, 1431–1435.

[3]

Thomas Booth, Srinivas Sridharan, Ann McNamara, Cindy Grimm, and Reynold J. Bailey. 2013. Guiding attention in controlled real-world environments. In ACM Symposium on Applied Perception 2013, SAP’ 13, Dublin, Ireland, August 22-23, 2013, Ludovic Hoyet, Betsy Williams Sanders, Joe Geigel, and Jeanine Stefanucci (Eds.). ACM, 75–82.

Digital Library

[4]

Rudolph P. Darken and Barry Peterson. 2014. Spatial Orientation, Wayfinding, and Representation. In Handbook of Virtual Environments - Design, Implementation, and Applications, Second Edition, Kelly S. Hale and Kay M. Stanney (Eds.). CRC Press, 467–491.

[5]

Jon Driver and Charles Spence. 1998. Attention and the crossmodal construction of space. Trends in Cognitive Sciences 2, 7 (July 1998), 254–262.

[6]

John Duncan. 1984. Selective attention and the organization of visual information. Journal of Experimental Psychology: General 113, 4 (1984), 501–517.

[7]

Robert Egly, Jon Driver, and Robert D. Rafal. 1994. Shifting visual attention between objects and locations: Evidence from normal and parietal lesion subjects. Journal of Experimental Psychology: General 123, 2 (1994), 161–177.

[8]

Magy Seif El-Nasr, Athanasios V. Vasilakos, Chinmay Rao, and Joseph A. Zupko. 2009. Dynamic Intelligent Lighting for Directing Visual Attention in Interactive 3-D Scenes. IEEE Trans. Comput. Intell. AI Games 1, 2 (2009), 145–153.

[9]

Charles W. Eriksen and James D. St. James. 1986. Visual attention within and around the field of focal attention: A zoom lens model. Perception & Psychophysics 40, 4 (July 1986), 225–240.

[10]

Michael W. Eysenck and Mark T. Keane. 2010. Cognitive psychology : a students handbook(6 ed.). Psychology Press, Chapter 1: Approaches to Human Cognition, 1–32.

[11]

Gregory M. Fitch, Jonathan M. Hankey, Brian M. Kleiner, and Thomas A. Dingus. 2011. Driver comprehension of multiple haptic seat alerts intended for use in an integrated collision avoidance system. Transportation Research Part F: Traffic Psychology and Behaviour 14, 4 (July 2011), 278–290.

[12]

Hede Helfrich. 2019. Sprache und Kommunikation(2. aufl. 2019 ed.). Springer Berlin Heidelberg, Chapter 9: Sprache und Kommunikation, 121–136.

[13]

Andrew Hollingworth, Ashleigh M. Maxcey-Richard, and Shaun P. Vecera. 2012. The Spatial Distribution of Attention within and across Objects. Journal of Experimental Psychology: Human Perception and Performance 38, 1 (Feb. 2012), 135–151.

[14]

Liqiang Huang and Harold Pashler. 2005. Quantifying object salience by equating distractor effects. Vision Research 45, 14 (June 2005), 1909–1920.

[15]

Gerrit Kahl, Lübomira Spassova, Johannes Schöning, Sven Gehring, and Antonio Krüger. 2011. IRL SmartCart - a user-adaptive context-aware interface for shopping assistance. In Proceedings of the 16th International Conference on Intelligent User Interfaces, IUI 2011, Palo Alto, CA, USA, February 13-16, 2011, Pearl Pu, Michael J. Pazzani, Elisabeth André, and Doug Riecken (Eds.). ACM, 359–362.

Digital Library

[16]

James R. Marston, Jack M. Loomis, Roberta L. Klatzky, and Reginald G. Golledge. 2007. Nonvisual Route following with Guidance from a Simple Haptic or Auditory Display. Journal of Visual Impairment & Blindness 101, 4 (2007), 203–211.

[17]

Christina Meredith and Judy Edworthy. 1995. Are there too many alarms in the intensive care unit? An overview of the problems. Journal of Advanced Nursing 21, 1 (1995), 15–20.

[18]

Dinara Moura and Lyn Bartram. 2014. Investigating players’ responses to wayfinding cues in 3D video games. In CHI Conference on Human Factors in Computing Systems, CHI’14, Toronto, ON, Canada - April 26 - May 01, 2014, Extended Abstracts, Matt Jones, Philippe A. Palanque, Albrecht Schmidt, and Tovi Grossman (Eds.). ACM, 1513–1518.

Digital Library

[19]

Hermann J. Müller, Joseph Krummenacher, and Torsten Schubert. 2015. Aufmerksamkeit und Handlungssteuerung. Springer Berlin Heidelberg, Chapter 3: Selective Aufmerksamkeit, 19–38.

[20]

Hermann J. Müller, Joseph Krummenacher, and Torsten Schubert. 2015. Aufmerksamkeit und Handlungssteuerung. Springer Berlin Heidelberg, Chapter 5: Visuelle Suche, 45–56.

[21]

Michael I. Posner. 1980. Orienting of attention. Quarterly Journal of Experimental Psychology 32, 1(1980), 3–25.

[22]

Ragunathan Rajkumar, Insup Lee, Lui Sha, and John A. Stankovic. 2010. Cyber-physical systems: the next computing revolution. In Proceedings of the 47th Design Automation Conference, DAC 2010, Anaheim, California, USA, July 13-18, 2010, Sachin S. Sapatnekar (Ed.). ACM, 731–736.

Digital Library

[23]

Bill N. Schilit and Marvin M. Theimer. 1994. Disseminating Active Map Information to Mobile Hosts. IEEE Network: The Magazine of Global Internetworking 8, 5 (Sept. 1994), 22–32.

Digital Library

[24]

Vivek K. Singh and Ramesh Jain. 2009. Situation based control for cyber-physical environments. In MILCOM 2009 - 2009 IEEE Military Communications Conference. 1–7.

[25]

Charles Spence and Jon Driver. 1997. Audiovisual links in exogenous covert spatial orienting. Perception & Psychophysics 59, 1 (Jan. 1997), 1–22.

[26]

Christoph Stahl, Jörg Baus, Boris Brandherm, Michael Schmitz, and Tim Schwartz. 2005. Navigational- and Shopping Assistance on the Basis of User Interactions in Intelligent Environments. In Proceedings of the IEE International Workshop on Intelligent Environments (IE 2005). University of Essex, Colchester, UK, 182–191.

[27]

Karl E. Steiner and Lavanya Voruganti. 2004. A comparison of guidance cues in desktop virtual environments. Virtual Real. 7, 3-4 (2004), 140–147.

[28]

Rufin VanRullen. 2003. Visual saliency and spike timing in the ventral visual pathway. Journal of Physiology-Paris 97, 2 (March 2003), 365–377.

[29]

Jeremy M. Wolfe, Kyle R. Cave, and Susan L. Franzel. 1989. Guided search: an alternative to the feature integration model for visual search. Journal of Experimental Psychology: Human perception and performance 15, 3 (Aug. 1989), 419–433.

Cited By

Lee GLee DSu GManocha D(2024)“May I Speak?”: Multi-Modal Attention Guidance in Social VR Group ConversationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337211930:5(2287-2297)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372119

Index Terms

Multi-modal Multi-scale Attention Guidance in Cyber-Physical Environments
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Tracking
  2. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms

Index terms have been assigned to the content through auto-classification.

Recommendations

Multi-modal and multi-camera attention in smart environments
ICMI-MLMI '09: Proceedings of the 2009 international conference on Multimodal interfaces

This paper considers the problem of multi-modal saliency and attention. Saliency is a cue that is often used for directing attention of a computer vision system, e.g., in smart environments or for robots. Unlike the majority of recent publications on ...
Sailing through Signs: Attention Guidance towards Texts in Virtual Reality
FDG '23: Proceedings of the 18th International Conference on the Foundations of Digital Games

Virtual reality (VR) devices have technically matured during the last few years. Apart from offering better resolutions and a wider field of view, these devices provide novel interaction forms that resemble real-world interactions. Players can explore ...
Focusing computational visual attention in multi-modal human-robot interaction
ICMI-MLMI '10: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction

Identifying verbally and non-verbally referred-to objects is an important aspect of human-robot interaction. Most importantly, it is essential to achieve a joint focus of attention and, thus, a natural interaction behavior. In this contribution, we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

April 2021

618 pages

ISBN:9781450380171

DOI:10.1145/3397481

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 April 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Bundesministerium für Bildung und Forschung

Conference

IUI '21

Sponsor:

IUI '21: 26th International Conference on Intelligent User Interfaces

April 14 - 17, 2021

TX, College Station, USA

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
152
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)5

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lee GLee DSu GManocha D(2024)“May I Speak?”: Multi-Modal Attention Guidance in Social VR Group ConversationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337211930:5(2287-2297)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372119

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents