ViRgilites: Multilevel Feedforward for Multimodal Interaction in VR
Authors: 
Valentino Artizzu, 
Kris Luyten, 
Gustavo Rovelo Ruiz, 
Lucio Davide SpanoAuthors Info & Claims
Valentino Artizzu, Kris Luyten, Gustavo Rovelo Ruiz, Lucio Davide SpanoAuthors Info & ClaimsArticle No.: 247, Pages 1 - 24
Abstract
Navigating the interaction landscape of Virtual Reality (VR) and Augmented Reality (AR) presents significant complexities due to the plethora of available input hardware and interaction modalities, compounded by spatially diverse visual interfaces. Such complexities elevate the likelihood of user errors, necessitating frequent backtracking. To address this, we introduce ViRgilites, a virtual guidance framework that delivers multi-level feedforward information covering the available interaction techniques as well as the future possibilities to interact with virtual objects, anticipating the interaction effects and how they fit with the overall user's goal. ViRgilites is engineered to facilitate task execution, empowering users to make informed decisions about action methodologies and alternative courses of action. This paper presents the architecture and functionality of ViRgilites and demonstrates its efficacy through evaluation with a formative user study.
Supplemental Material
MP4 File
Demonstration video
- Download
- 89.29 MB
References
[1]
Valentino Artizzu, Gianmarco Cherchi, Davide Fara, Vittoria Frau, Riccardo Macis, Luca Pitzalis, Alessandro Tola, Ivan Blecic, and Lucio Davide Spano. 2022. Defining Configurable Virtual Reality Templates for End Users. Proc. ACM Hum.-Comput. Interact. 6, EICS, Article 163 (jun 2022), 35 pages. https://doi.org/10.1145/3534517
[2]
SK Bailey, Cheryl I Johnson, Bradford L Schroeder, and Matthew D Marraffino. 2017. Using virtual reality for training maintenance procedures. In Proceedings of the Interservice/Industry Training, Simulation and Education Conference.
[3]
Marc Baloup, Thomas Pietrzak, and Géry Casiez. 2019. RayCursor: A 3D Pointing Facilitation Technique Based on Raycasting. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300331
[4]
Soumya C. Barathi, Daniel J. Finnegan, Matthew Farrow, Alexander Whaley, Pippa Heath, Jude Buckley, Peter W. Dowrick, Burkhard C. Wuensche, James L. J. Bilzon, Eamonn O'Neill, and Christof Lutteroth. 2018. Interactive Feedforward for Improving Performance and Maintaining Intrinsic Motivation in VR Exergaming. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3173574.3173982
[5]
Olivier Bau and Wendy E. Mackay. 2008. OctoPocus: A Dynamic Guide for Learning Gesture-Based Command Sets. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST '08). Association for Computing Machinery, New York, NY, USA, 37--46. https://doi.org/10.1145/1449715.1449724
[6]
Mark Billinghurst. 2013. Hands and Speech in Space: Multimodal Interaction with Augmented Reality Interfaces. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction (Sydney, Australia) (ICMI '13). Association for Computing Machinery, New York, NY, USA, 379--380. https://doi.org/10.1145/2522848.2532202
[7]
Jeremy Boy, Louis Eveillard, Françoise Detienne, and Jean-Daniel Fekete. 2015. Suggested interactivity: Seeking perceived affordances for information visualization. IEEE transactions on visualization and computer graphics 22, 1 (2015), 639--648.
[8]
Eric Burns, Sharif Razzaque, Abigail T Panter, Mary C Whitton, Matthew R McCallus, and Frederick P Brooks. 2005. The hand is slower than the eye: A quantitative exploration of visual dominance over proprioception. In IEEE Proceedings. VR 2005. Virtual Reality, 2005. IEEE, 3--10.
[9]
Alisa Burova, John Mäkelä, Jaakko Hakulinen, Tuuli Keskinen, Hanna Heinonen, Sanni Siltanen, and Markku Turunen. 2020. Utilizing VR and Gaze Tracking to Develop AR Solutions for Industrial Maintenance. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3313831.3376405
[10]
Edwige Chauvergne, Martin Hachet, and Arnaud Prouzeau. 2023. User Onboarding in Virtual Reality: An Investigation of Current Practices. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 711, 15 pages. https://doi.org/10.1145/3544548.3581211
[11]
Feng-Kuang Chiang, Xiaojing Shang, and Lu Qiao. 2022. Augmented reality in vocational training: A systematic review of research and applications. Computers in Human Behavior 129 (2022), 107125. https://doi.org/10.1016/j.chb.2021.107125
[12]
Sven Coppers, Kris Luyten, Davy Vanacken, David Navarre, Philippe Palanque, and Christine Gris. 2019. Fortunettes: Feedforward about the Future State of GUI Widgets. 3, EICS, Article 20 (jun 2019), 20 pages. https://doi.org/10.1145/3331162
[13]
Joëlle Coutaz. 2006. Meta-User Interfaces for Ambient Spaces. In Proceedings of the 5th International Conference on Task Models and Diagrams for Users Interface Design (Hasselt, Belgium) (TAMODIA'06). Springer-Verlag, Berlin, Heidelberg, 1--15.
[14]
Anind K Dey, Gregory D Abowd, and Daniel Salber. 2001. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Human-Computer Interaction 16, 2-4 (2001), 97--166.
[15]
Tom Djajadiningrat, Kees Overbeeke, and Stephan Wensveen. 2002. But how, Donald, tell us how? on the creation of meaning in interaction design through feedforward and inherent feedback. In Proceedings of the 4th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (London, England) (DIS '02). Association for Computing Machinery, New York, NY, USA, 285--291. https://doi.org/10.1145/778712.778752
[16]
Katherine Fennedy, Jeremy Hartmann, Quentin Roy, Simon Tangi Perrault, and Daniel Vogel. 2021. Octopocus in VR: using a dynamic guide for 3D mid-air gestures in virtual reality. IEEE Transactions on Visualization and Computer Graphics 27, 12 (2021), 4425--4438.
[17]
Andrea Ferracani, Marco Faustino, Gabriele Xavier Giannini, Lea Landucci, and Alberto Del Bimbo. 2017. Natural experiences in museums through virtual reality and voice commands. In Proceedings of the 25th ACM international conference on Multimedia. 1233--1234.
[18]
UNESCO Institute for Statistics. 2012. International standard classification of education: ISCED 2011. Comparative Social Research 30 (2012).
[19]
Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1995. Design patterns: elements of reusable object-oriented software. Pearson Deutschland GmbH.
[20]
Tovi Grossman and George Fitzmaurice. 2010. ToolClips: An Investigation of Contextual Video Assistance for Functionality Understanding. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '10). Association for Computing Machinery, New York, NY, USA, 1515--1524. https://doi.org/10.1145/1753326.1753552
[21]
Daniel Hepperle, Yannick Weiß, Andreas Siess, and Matthias Wölfel. 2019. 2D, 3D or speech? A case study on which user interface is preferable for what kind of object interaction in immersive virtual reality. Computers & Graphics 82 (2019), 321--331.
[22]
Teresa Hirzle, Jan Gugenheimer, Florian Geiselhart, Andreas Bulling, and Enrico Rukzio. 2019. A Design Space for Gaze Interaction on Head-Mounted Displays. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300855
[23]
Robert J.K. Jacob, Audrey Girouard, Leanne M. Hirshfield, Michael S. Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-Based Interaction: A Framework for Post-WIMP Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI '08). Association for Computing Machinery, New York, NY, USA, 201--210. https://doi.org/10.1145/1357054.1357089
[24]
Tiffany C.K. Kwok, Peter Kiefer, Victor R. Schinazi, Benjamin Adams, and Martin Raubal. 2019. Gaze-Guided Narratives: Adapting Audio Guide Content to Gaze in Virtual and Real Environments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300721
[25]
Jérémy Lacoche, Thierry Duval, Bruno Arnaldi, Eric Maisel, and Jérome Royan. 2015. Plasticity for 3D user interfaces: new models for devices and interaction techniques. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Duisburg, Germany) (EICS '15). Association for Computing Machinery, New York, NY, USA, 28--33. https://doi.org/10.1145/2774225.2775073
[26]
Jérémy Lacoche, Thierry Duval, Bruno Arnaldi, Eric Maisel, and Jérôme Royan. 2019. 3DPlasticToolkit: Plasticity for 3D User Interfaces. In Virtual Reality and Augmented Reality, Patrick Bourdot, Victoria Interrante, Luciana Nedel, Nadia Magnenat-Thalmann, and Gabriel Zachmann (Eds.). Springer International Publishing, Cham, 62--83.
[27]
Giorgia Lallai, Giovanni Loi Zedda, Célia Martinie, Philippe Palanque, Mauro Pisano, and Lucio Davide Spano. 2021. Engineering Task-based Augmented Reality Guidance: Application to the Training of Aircraft Flight Procedures. Interacting with Computers 33, 1 (03 2021), 17--39. https://doi.org/10.1093/iwcomp/iwab007 arXiv:https://academic.oup.com/iwc/article-pdf/33/1/17/38495857/iwab007.pdf
[28]
Joseph J LaViola Jr, Ernst Kruijff, Ryan P McMahan, Doug Bowman, and Ivan P Poupyrev. 2017. 3D user interfaces: theory and practice. Addison-Wesley Professional.
[29]
Klemen Lilija, S⊘ren Kyllingsbæk, and Kasper Hornbæk. 2021. Correction of Avatar Hand Movements Supports Learning of a Motor Skill. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 1--8. https://doi.org/10.1109/VR50410.2021.00069
[30]
Paul Lukowicz, Sandy Pentland, and Alois Ferscha. 2011. From context awareness to socially aware computing. IEEE pervasive computing 11, 1 (2011), 32--41.
[31]
Daniel Martin, Sandra Malpica, Diego Gutierrez, Belen Masia, and Ana Serrano. 2022. Multimodality in VR: A Survey. ACM Comput. Surv. 54, 10s, Article 216 (sep 2022), 36 pages. https://doi.org/10.1145/3508361
[32]
Microsoft. 2022. Mixed Reality Toolkit Unity Plugin. https://github.com/microsoft/MixedRealityToolkit-Unity [Online; accessed 27-July-2023].
[33]
Florian 'Floyd' Mueller, Tuomas Kari, Rohit Khot, Zhuying Li, Yan Wang, Yash Mehta, and Peter Arnold. 2018. Towards Experiencing Eating as a Form of Play. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (Melbourne, VIC, Australia) (CHI PLAY '18 Extended Abstracts). Association for Computing Machinery, New York, NY, USA, 559--567. https://doi.org/10.1145/3270316.3271528
[34]
Andreea Muresan, Jess McIntosh, and Kasper Hornbæk. 2023. Using Feedforward to Reveal Interaction Possibilities in Virtual Reality. ACM Trans. Comput.-Hum. Interact. (jun 2023). https://doi.org/10.1145/3603623 Just Accepted.
[35]
Allen Newell and Herbert Alexander Simon. 1972. Human problem solving. Prentice-hall Englewood Cliffs, NJ.
[36]
Donald A Norman. 1999. Affordance, conventions, and design. interactions 6, 3 (1999), 38--43.
[37]
Donald A. Norman. 2002. The Design of Everyday Things. Basic Books, Inc., USA.
[38]
Donald A Norman. 2008. The way I see IT signifiers, not affordances. interactions 15, 6 (2008), 18--19.
[39]
Charith Perera, Arkady Zaslavsky, Peter Christen, and Dimitrios Georgakopoulos. 2013. Context aware computing for the internet of things: A survey. IEEE communications surveys & tutorials 16, 1 (2013), 414--454.
[40]
Krzysztof Pietroszek. 2018. Raycasting in Virtual Reality. Springer International Publishing, Cham, 1--3. https://doi.org/10.1007/978-3-319-08234-9_180-1
[41]
Krzysztof Pietroszek. 2018. Virtual Hand Metaphor in Virtual Reality. Springer International Publishing, Cham, 1--3. https://doi.org/10.1007/978-3-319-08234-9_178-1
[42]
Amotz Perlman Rafael Sacks and Ronen Barak. 2013. Construction safety training using immersive virtual reality. Construction Management and Economics 31, 9 (2013), 1005--1017. https://doi.org/10.1080/01446193.2013.828844
[43]
Nimesha Ranasinghe, Pravar Jain, Nguyen Thi Ngoc Tram, Koon Chuan Raymond Koh, David Tolley, Shienny Karwita, Lin Lien-Ya, Yan Liangkun, Kala Shamaiah, Chow Eason Wai Tung, Ching Chiuan Yen, and Ellen Yi-Luen Do. 2018. Season Traveller: Multisensory Narration for Enhancing the Virtual Reality Experience. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3173574.3174151
[44]
Rajinder Sodhi, Hrvoje Benko, and Andrew Wilson. 2012. LightGuide: Projected Visualizations for Hand Movement Guidance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 179--188. https://doi.org/10.1145/2207676.2207702
[45]
Charles Spence, Jae Lee, and Nathan Van der Stoep. 2020. Responding to sounds from unseen locations: crossmodal attentional orienting in response to sounds presented from the rear. European Journal of Neuroscience 51, 5 (2020), 1137--1150.
[46]
Anthony Steed, Tuukka M. Takala, Daniel Archer, Wallace Lages, and Robert W. Lindeman. 2021. Directions for 3D User Interface Research from Consumer VR Games. IEEE Transactions on Visualization and Computer Graphics 27, 11 (2021), 4171--4182. https://doi.org/10.1109/TVCG.2021.3106431
[47]
Richard Stone, Elease McLaurin, Peihan Zhong, and Kristopher Watts. 2013. Full virtual reality vs. integrated virtual reality training in welding. (2013).
[48]
Michael Terry and Elizabeth D. Mynatt. 2002. Side Views: Persistent, on-Demand Previews for Open-Ended Tasks. In Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology (Paris, France) (UIST '02). Association for Computing Machinery, New York, NY, USA, 71--80. https://doi.org/10.1145/571985.571996
[49]
David Thevenin and Joëlle Coutaz. 1999. Plasticity of user interfaces: Framework and research agenda. In Interact, Vol. 99. 110--117.
[50]
Jo Vermeulen, Kris Luyten, Elise van den Hoven, and Karin Coninx. 2013. Crossing the Bridge over Norman's Gulf of Execution: Revealing Feedforward's True Identity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). Association for Computing Machinery, New York, NY, USA, 1931--1940. https://doi.org/10.1145/2470654.2466255
[51]
S. A. G. Wensveen, J. P. Djajadiningrat, and C. J. Overbeeke. 2004. Interaction Frogger: A Design Framework to Couple Action and Function through Feedback and Feedforward. In Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (Cambridge, MA, USA) (DIS '04). Association for Computing Machinery, New York, NY, USA, 177--184. https://doi.org/10.1145/1013115.1013140
[52]
Heli Wigelius and Heli Väätäjä. 2009. Dimensions of context affecting user experience in mobile work. In Human-Computer Interaction-INTERACT 2009: 12th IFIP TC 13 International Conference, Uppsala, Sweden, August 24-28, 2009, Proceedings, Part II 12. Springer, 604--617.
[53]
Huiyue Wu, Yu Wang, Jiali Qiu, Jiayi Liu, and Xiaolong Zhang. 2019. User-defined gesture interaction for immersive VR shopping applications. Behaviour & Information Technology 38, 7 (2019), 726--741.
[54]
Hui-Yin Wu, Florent Robert, Théo Fafet, Brice Graulier, Barthelemy Passin-Cauneau, Lucile Sassatelli, and Marco Winckler. 2022. Designing Guided User Tasks in VR Embodied Experiences. Proc. ACM Hum.-Comput. Interact. 6, EICS, Article 158 (jun 2022), 24 pages. https://doi.org/10.1145/3532208
[55]
LI Yang, Jin Huang, TIAN Feng, WANG Hong-An, and DAI Guo-Zhong. 2019. Gesture interaction in virtual reality. Virtual Reality & Intelligent Hardware 1, 1 (2019), 84--112.
[56]
Difeng Yu, Xueshi Lu, Rongkai Shi, Hai-Ning Liang, Tilman Dingler, Eduardo Velloso, and Jorge Goncalves. 2021. Gaze-Supported 3D Object Manipulation in Virtual Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 734, 13 pages. https://doi.org/10.1145/3411764.3445343
[57]
Özgür Yürür, Chi Harold Liu, Zhengguo Sheng, Victor CM Leung, Wilfrido Moreno, and Kin K Leung. 2014. Context-awareness for mobile sensing: A survey and future directions. IEEE Communications Surveys & Tutorials 18, 1 (2014), 68--93.
Index Terms
- ViRgilites: Multilevel Feedforward for Multimodal Interaction in VR
Recommendations
Using Feedforward to Reveal Interaction Possibilities in Virtual Reality
In virtual reality (VR), interactions may fail when users encounter new, unknown, or unexpected objects. We propose using feedforward in VR to help users interact with objects by revealing how such objects work. Feedforward lets users know what to do and ...
Direct Feedforward Techniques for the ViRgilites System
EICS '24 Companion: Companion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing SystemsIn this poster we propose an implementation of direct feedforward for the ViRgilites system. The project defines two alternative uses, with respect to the current implementation, that only shows in an indirect way (icons, target object images, text) how ...
Multimodal prop-based interaction with virtual mock-up: CAD model integration and human performance evaluation
VRST '08: Proceedings of the 2008 ACM symposium on Virtual reality software and technologyThis paper presents a methodology for the efficient integration of CAD models in a physical-based virtual reality simulation that provides the user with multi-modal feedback. User interacts with virtual mock-up using a string-based haptic interface. ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 17 June 2024
Accepted: 01 March 2024
Revised: 01 January 2024
Received: 01 October 2023
Published in PACMHCI Volume 8, Issue EICS
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Funding Sources
- NextGenerationEU
- The National Operational Program Research and Innovation 2014-2020 (PONRI)
- Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 288Total Downloads
- Downloads (Last 12 months)288
- Downloads (Last 6 weeks)42
Reflects downloads up to 24 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in