research-article

Multimodal reference resolution for mobile spatial interaction in urban environments

Authors:

Mohammad Mehdi Moniri,

Christian MüllerAuthors Info & Claims

AutomotiveUI '12: Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

Pages 241 - 248

https://doi.org/10.1145/2390256.2390296

Published: 17 October 2012 Publication History

Abstract

We present results of a study on referring to the outside environment from within a moving vehicle. Reference resolution is the first necessary step in integrating the outside environment into the interactive system in the car. It is the problem of finding out which of the objects outside the users is interested in. In our study, we explored eye gaze, head pose, pointing gesture with a smart phone, and the user's view field. We implemented and tested everything in a moving vehicle in a real-life traffic. For safety reasons, the front-seat passenger used the system while the driver was concentrating completely on driving. For analysis and visualization of the user's interaction with the environment, 528 buildings of the city were modeled in 2.5D by using an airborne LIDAR scan, Google Earth, and a spatial database. As a result of our study, we propose in this paper a new algorithm for spatial reference resolution together with a scanning mechanism.

References

[1]

K. Amlacher, G. Fritz, P. Luley, A. Almer, and L. Paletta. Geo-contextual priors for attentive urban object recognition. In Proceedings of the 2009 IEEE international conference on Robotics and Automation, ICRA'09, pages 3015--3020, Piscataway, NJ, USA, 2009. IEEE Press.

Digital Library

[2]

L. Baillie, H. Kunczier, and H. Anegg. Rolling, rotating and imagining in a virtual mobile world. In Proceedings of the 7th international conference on Human computer interaction with mobile devices & services, MobileHCI '05, pages 283--286, New York, NY, USA, 2005. ACM.

Digital Library

[3]

M. Baldauf, P. Fröhlich, and S. Hutter. Kibitzer: a wearable system for eye-gaze-based mobile urban exploration. In Proceedings of the 1st Augmented Human International Conference, AH '10, pages 9:1--9:5, New York, NY, USA, 2010. ACM.

Digital Library

[4]

W. Barfield, C. M. Hendrix, O. Bjorneseth, K. A. Kaczmarek, and W. Lotens. Comparison of human sensory capabilities with technical specifications of virtual environment equipment. Teleoperators and Virtual Environments, 4:329--356, 1995.

Digital Library

[5]

J. D. Carswell. 3dq: Threat dome visibility querying on mobile devices. In GIM International, Vol.24, (8), 2010.

[6]

V. R. Chandrasekhar, D. M. Chen, S. S. Tsai, N.-M. Cheung, H. Chen, G. Takacs, Y. Reznik, R. Vedantham, R. Grzeszczuk, J. Bach, and B. Girod. The stanford mobile visual search data set. In Proceedings of the second annual ACM conference on Multimedia systems, MMSys '11, pages 117--122, New York, NY, USA, 2011. ACM.

Digital Library

[7]

E. De Ves, I. Coma, M. Fernï£¡ndez, and J. Gimeno. Intelligent eye: location-based multimedia information for mobile. In Conference iiWAS2010, Paris, France, 2010.

Digital Library

[8]

W. T. Fong, S. K. Ong, and A. Y. C. Nee. Computer vision centric hybrid tracking for augmented reality in outdoor urban environments. In Proceedings of the 8th International Conference on Virtual Reality Continuum and its Applications in Industry, VRCAI '09, pages 185--190, New York, NY, USA, 2009. ACM.

Digital Library

[9]

K. Gardiner and J. D. Carswell. Viewer-based directional querying for mobile applications. In Proceedings of the Fourth international conference on Web information systems engineering workshops, WISEW'03, pages 83--91, Washington, DC, USA, 2003. IEEE Computer Society.

Digital Library

[10]

K. Gardiner, J. Yin, and J. D. Carswell. Egoviz --- a mobile based spatial interaction system. In Proceedings of the 9th International Symposium on Web and Wireless Geographical Information Systems, W2GIS '09, pages 135--152, Berlin, Heidelberg, 2009. Springer-Verlag.

Digital Library

[11]

J. Löfström. Physical Mobile Interaction Design for Spatially-Aware Pointing. Master's thesis, KTH Computer Science and Communication, Stockholm, Sweden, 2010.

[12]

R. Simon and P. Fröhlich. A mobile application framework for the geospatial web. In Proceedings of the 16th international conference on World Wide Web, WWW '07, pages 381--390, New York, NY, USA, 2007. ACM.

Digital Library

[13]

R. Simon, P. Fröhlich, and H. Anegg. Beyond location based - the spatially aware mobile phone. In J. Carswell and T. Tezuka, editors, Web and Wireless Geographical Information Systems, volume 4295 of Lecture Notes in Computer Science, pages 12--21. Springer Berlin/Heidelberg, 2006. 10.1007/11935148_2.

Digital Library

[14]

R. Simon, P. Fröhlich, and T. Grechenig. Geopointing: evaluating the performance of orientation-aware location-based interaction under real-world conditions. J. Locat. Based Serv., 2:24--40, March 2008.

Digital Library

[15]

R. Simon, P. Fröhlich, G. Obernberger, and E. Wittowetz. The point to discover geowand. In Proceedings of the 9th International Conference on Ubiquitous Computing (UbiComp 07), September 2007, 2009.

[16]

R. Simon, H. Kunczier, and H. Anegg. Towards orientation-aware location based mobile services. In G. Gartner, W. Cartwright, and M. P. Peterson, editors, Location Based Services and TeleCartography, Lecture Notes in Geoinformation and Cartography, pages 279--290. Springer Berlin Heidelberg, 2007. 10.1007/978-3-540-36728-4_21.

[17]

L. Terissi and J. Gómez. 3d head pose and facial expression tracking using a single camera. j-jucs, 16(6):903--920, 2010.

Cited By

Gomaa AReyes GFeld MKrüger A(2024)Looking for a better fit? An Incremental Learning Multimodal Object Referencing Framework adapting to Individual DriversProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645152(1-13)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645152
Reyes GGomaa AFeld M(2023)It’s all about you: Personalized in-Vehicle Gesture Recognition with a Time-of-Flight CameraProceedings of the 15th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3580585.3607153(234-243)Online publication date: 18-Sep-2023
https://dl.acm.org/doi/10.1145/3580585.3607153
Gomaa AFeld M(2023)Towards Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous SystemsProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3616121(689-694)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3616121
Show More Cited By

Index Terms

Multimodal reference resolution for mobile spatial interaction in urban environments
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI theory, concepts and models
    2. Interaction devices
      1. Touch screens
  2. Interaction design
    1. Interaction design theory, concepts and paradigms

Recommendations

Making use of drivers' glances onto the screen for explicit gaze-based interaction
AutomotiveUI '10: Proceedings of the 2nd International Conference on Automotive User Interfaces and Interactive Vehicular Applications

Interaction with communication and infotainment systems in the car is common while driving. Our research investigates modalities and techniques that enable interaction with interactive applications while driving without compromising safety. In this ...
Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle
ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction

Hand pointing and eye gaze have been extensively investigated in automotive applications for object selection and referencing. Despite significant advances, existing outside-the-vehicle referencing methods consider these modalities separately. Moreover, ...
Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze
ICMI '19: 2019 International Conference on Multimodal Interaction

Continuous emotion recognition is of great significance in affective computing and human-computer interaction. Most of existing methods for video based continuous emotion recognition utilize facial expression. However, besides facial expression, other ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AutomotiveUI '12: Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

October 2012

280 pages

ISBN:9781450317511

DOI:10.1145/2390256

General Chair:
Andrew L. Kun
University of New Hampshire
,
Program Chairs:
Linda Boyle
University of Washington
,
Bryan Reimer
Massachusetts Institute of Technology
,
Andreas Riener
University of Linz
,
Jennifer Healey
Intel Corporation
,
Wei Zhang
Tsinghua University
,
Publications Chairs:
Bastian Pfleging
University of Stuttgart
,
Marc Kurz
University of Linz

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Bundesministerium für Bildung und Forschung

Conference

AutomotiveUI '12

AutomotiveUI '12: International Conference on Automotive User Interfaces and Interactive Vehicular Applications

October 17 - 19, 2012

New Hampshire, Portsmouth

Acceptance Rates

Overall Acceptance Rate 248 of 566 submissions, 44%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
235
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gomaa AReyes GFeld MKrüger A(2024)Looking for a better fit? An Incremental Learning Multimodal Object Referencing Framework adapting to Individual DriversProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645152(1-13)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645152
Reyes GGomaa AFeld M(2023)It’s all about you: Personalized in-Vehicle Gesture Recognition with a Time-of-Flight CameraProceedings of the 15th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3580585.3607153(234-243)Online publication date: 18-Sep-2023
https://dl.acm.org/doi/10.1145/3580585.3607153
Gomaa AFeld M(2023)Towards Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous SystemsProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3616121(689-694)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3616121
Gomaa A(2022)Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive InterfacesProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3557034(690-695)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536221.3557034
Aftab Avon der Beeck M(2022)Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the VehicleProceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511142(483-495)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490099.3511142
Gomaa AReyes GFeld M(2021)ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving VehicleProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479910(318-327)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3462244.3479910
Moniri MFeld M(2017)Eye and Head Tracking for Focus of Attention Control in the CockpitAutomotive User Interfaces10.1007/978-3-319-49448-7_9(249-272)Online publication date: 28-Feb-2017
https://doi.org/10.1007/978-3-319-49448-7_9
Trösterer SWuchse MDöttlinger CMeschtscherjakov ATscheligi MBurnett GGabbard JGreen POsswald SEren AAntrobus V(2015)Light my wayProceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/2799250.2799258(196-203)Online publication date: 1-Sep-2015
https://dl.acm.org/doi/10.1145/2799250.2799258
Trösterer SGärtner MWuchse MMaurer BBaumgartner AMeschtscherjakov ATscheligi M(2015)Four Eyes See More Than Two: Shared Gaze in the CarHuman-Computer Interaction – INTERACT 201510.1007/978-3-319-22668-2_26(331-348)Online publication date: 30-Aug-2015
https://doi.org/10.1007/978-3-319-22668-2_26
Moniri MFeld MMuller C(2012)Personalized In-Vehicle Information SystemsProceedings of the 2012 Eighth International Conference on Intelligent Environments10.1109/IE.2012.40(379-382)Online publication date: 26-Jun-2012
https://dl.acm.org/doi/10.1109/IE.2012.40

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten