Towards Enhanced Context Awareness with Vision-based Multimodal Interfaces
Article No.: 41, Pages 1 - 3
Abstract
Vision-based Interfaces (VIs) are pivotal in advancing Human-Computer Interaction (HCI), particularly in enhancing context awareness. However, there are significant opportunities for these interfaces due to rapid advancements in multimodal Artificial Intelligence (AI), which promise a future of tight coupling between humans and intelligent systems. AI-driven VIs, when integrated with other modalities, offer a robust solution for effectively capturing and interpreting user intentions and complex environmental information, thereby facilitating seamless and efficient interactions. This PhD study explores three application cases of multimodal interfaces to augment context awareness, respectively focusing on three dimensions of visual modality: scale, depth, and time: a fine-grained analysis of physical surfaces via microscopic image, precise projection of the real world using depth data, and rendering haptic feedback from video background in virtual environments.
References
[1]
Don Samitha Elvitigala, Yunfan Wang, Yongquan Hu, and Aaron J Quigley. 2023. RadarFoot: Fine-grain Ground Surface Context Awareness for Smart Shoes. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–13.
[2]
Hans W Gellersen, Albrecht Schmidt, and Michael Beigl. 2002. Multi-sensor context-awareness in mobile devices and smart artifacts. Mobile Networks and Applications 7 (2002), 341–351.
[3]
Albert Haque, Arnold Milstein, and Li Fei-Fei. 2020. Illuminating the dark spaces of healthcare with ambient intelligence. Nature 585, 7824 (2020), 193–202.
[4]
Haitham Sabah Hasan and S Abdul Kareem. 2012. Human computer interaction for vision based hand gesture recognition: A survey. In 2012 International Conference on Advanced Computer Science Applications and Technologies (ACSAT). IEEE, 55–60.
[5]
Yongquan Hu, Wen Hu, and Aaron Quigley. 2023. Towards Using Generative AI for Facilitating Image Creation in Spatial Augmented Reality. In 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, 441–443.
[6]
Yongquan Hu, Hui-Shyong Yeo, Mingyue Yuan, Haoran Fan, Don Samitha Elvitigala, Wen Hu, and Aaron Quigley. 2023. Microcam: Leveraging smartphone microscope camera for context-aware contact surface sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 3 (2023), 1–28.
[7]
Yongquan Hu, Mingyue Yuan, Kaiqi Xian, Don Samitha Elvitigala, and Aaron Quigley. 2023. Exploring the design space of employing ai-generated content for augmented reality display. arXiv preprint arXiv:2303.16593 (2023).
[8]
Yongquan Hu, Dawen Zhang, and Aaron Quigley. 2023. GenAIR: Exploring Design Factor of Employing Generative AI for Augmented Reality. In Proceedings of the 2023 ACM Symposium on Spatial User Interaction. 1–3.
[9]
Ajune Wanis Ismail, Mark Billinghurst, and Mohd Shahrizal Sunar. 2015. Vision-based technique and issues for multimodal interaction in augmented reality. In Proceedings of the 8th International Symposium on Visual Information Communication and Interaction. 75–82.
[10]
Mathias Kolsch, Matthew Turk, and Tobias Hollerer. 2004. Vision-based interfaces for mobility. In The First Annual International Conference on Mobile and Ubiquitous Systems: Networking and Services, 2004. MOBIQUITOUS 2004. IEEE, 86–94.
[11]
Jarmo Makkonen, Ivan Avdouevski, Riitta Kerminen, and Ari Visa. 2009. Context awareness in human-computer interaction. In Human-Computer Interaction. IntechOpen.
[12]
Stefan Poslad. 2011. Ubiquitous computing: smart devices, environments and interactions. John Wiley & Sons.
[13]
Aaron Quigley and Peter Eades. 2000. Fade: Graph drawing, clustering, and visual abstraction. In International Symposium on Graph Drawing. Springer, 197–210.
[14]
Maximilian Schrapel, Philipp Etgeton, and Michael Rohs. 2021. SpectroPhone: Enabling Material Surface Sensing with Rear Camera and Flashlight LEDs. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–5.
[15]
Rajeev Sharma, Vladimir I Pavlovic, and Thomas S Huang. 1998. Toward multimodal human-computer interface. Proc. IEEE 86, 5 (1998), 853–869.
[16]
Gihan Shin and Junchul Chun. 2007. Vision-based multimodal human computer interface based on parallel tracking of eye and hand motion. In 2007 International Conference on Convergence Information Technology (ICCIT 2007). IEEE, 2443–2448.
[17]
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, 2024. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. arXiv preprint arXiv:2406.16860 (2024).
[18]
Xing-Dong Yang, Tovi Grossman, Daniel Wigdor, and George Fitzmaurice. 2012. Magic finger: always-available input through finger instrumentation. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 147–156.
[19]
Hui-Shyong Yeo, Juyoung Lee, Andrea Bianchi, David Harris-Birtill, and Aaron Quigley. 2017. Specam: Sensing surface color and material with the front-facing camera of a mobile device. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1–9.
[20]
Hui-Shyong Yeo, Ryosuke Minami, Kirill Rodriguez, George Shaker, and Aaron Quigley. 2018. Exploring tangible interactions with radar sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 1–25.
Index Terms
- Towards Enhanced Context Awareness with Vision-based Multimodal Interfaces
Recommendations
Context-aware querying for multimodal search engines
MMM'12: Proceedings of the 18th international conference on Advances in Multimedia ModelingMultimodal interaction provides the user with multiple modes of interacting with a system, such as gestures, speech, text, video, audio, etc. A multimodal system allows for several distinct means for input and output of data. In this paper, we present ...
Multimodal augmented reality: the norm rather than the exception
MVAR '16: Proceedings of the 2016 workshop on Multimodal Virtual and Augmented RealityAugmented reality (AR) is commonly seen as a technology that overlays virtual imagery onto a participant's view of the world. In line with this, most AR research is focused on what we see. In this paper, we challenge this focus on vision and make a case ...
Multimodal spatial reference in mediated environments: users' preferences and the pragmatics of pointing and talking
CHI EA '06: CHI '06 Extended Abstracts on Human Factors in Computing SystemsThis paper describes the current results and future developments of a project on multimodal spatial reference in mediated environments. The database consists of video-recorded sessions, with 120 participants in three experimental designs, contrasting ...
Comments
Information & Contributors
Information
Published In
September 2024
252 pages
ISBN:9798400705069
DOI:10.1145/3640471
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 September 2024
Check for updates
Author Tags
Qualifiers
- Extended-abstract
- Research
- Refereed limited
Conference
MobileHCI '24
Sponsor:
MobileHCI '24: 26th International Conference on Mobile Human-Computer Interaction
September 30 - October 3, 2024
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 202 of 906 submissions, 22%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 47Total Downloads
- Downloads (Last 12 months)47
- Downloads (Last 6 weeks)9
Reflects downloads up to 01 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format