research-article

VRhook: A Data Collection Tool for VR Motion Sickness Research

Authors:

Tharindu Indrajith Kaluarachchi,

Shamane Siriwardhana,

Mark Billinghurst,

Robert W. Lindeman,

Suranga NanayakkaraAuthors Info & Claims

UIST '22: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

Article No.: 82, Pages 1 - 9

https://doi.org/10.1145/3526113.3545656

Published: 28 October 2022 Publication History

Abstract

Despite the increasing popularity of VR games, one factor hindering the industry’s rapid growth is motion sickness experienced by the users. Symptoms such as fatigue and nausea severely hamper the user experience. Machine Learning methods could be used to automatically detect motion sickness in VR experiences, but generating the extensive labeled dataset needed is a challenging task. It needs either very time consuming manual labeling by human experts or modification of proprietary VR application source codes for label capturing. To overcome these challenges, we developed a novel data collection tool, VRhook, which can collect data from any VR game without needing access to its source code. This is achieved by dynamic hooking, where we can inject custom code into a game’s run-time memory to record each video frame and its associated transformation matrices. Using this, we can automatically extract various useful labels such as rotation, speed, and acceleration. In addition, VRhook can blend a customized screen overlay on top of game contents to collect self-reported comfort scores. In this paper, we describe the technical development of VRhook, demonstrate its utility with an example, and describe directions for future research.

References

[1]

[n.d.]. Virtual reality in gaming market size. https://www.fortunebusinessinsights.com/industry-reports/virtual-reality-gaming-market-100271

[2]

Isayas Berhe Adhanom, Nathan Navarro Griffin, Paul MacNeilage, and Eelke Folmer. 2020. The effect of a foveated field-of-view restrictor on VR sickness. In 2020 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, 645–652.

[3]

Frederick Bonato, Andrea Bubka, and Stephen Palmisano. 2009. Combined pitch and roll and cybersickness in a virtual environment. Aviation, space, and environmental medicine 80, 11 (2009), 941–945.

[4]

Frederick Bonato, Andrea Bubka, Stephen Palmisano, Danielle Phillip, and Giselle Moreno. 2008. Vection change exacerbates simulator sickness in virtual environments. Presence: Teleoperators and Virtual Environments 17, 3(2008), 283–292.

Digital Library

[5]

Matthew S Brennesholtz. 2018. 3-1: Invited Paper: VR Standards and Guidelines. In SID Symposium Digest of Technical Papers, Vol. 49. Wiley Online Library, 1–4.

[6]

Kieran Carnegie and Taehyun Rhee. 2015. Reducing visual discomfort with HMDs using dynamic depth of field. IEEE computer graphics and applications 35, 5 (2015), 34–41.

Digital Library

[7]

Eunhee Chang, Hyun Taek Kim, and Byounghyun Yoo. 2020. Virtual reality sickness: a review of causes and measurements. International Journal of Human–Computer Interaction 36, 17(2020), 1658–1682.

[8]

Jean-Rémy Chardonnet, Mohammad Ali Mirzaei, and Frédéric Mérienne. 2015. Visually induced motion sickness estimation and prediction in virtual reality using frequency components analysis of postural sway signal. In International conference on artificial reality and telexistence eurographics symposium on virtual environments. 9–16.

[9]

Minghan Du, Hui Cui, Yuan Wang, and Henry Duh. 2021. Learning from Deep Stereoscopic Attention for Simulator Sickness Prediction. IEEE Transactions on Visualization and Computer Graphics (2021).

[10]

HB-L Duh, JW Lin, Robert V Kenyon, Donald E Parker, and Thomas A Furness. 2001. Effects of field of view on balance in an immersive environment. In Proceedings IEEE Virtual Reality 2001. IEEE, 235–240.

[11]

Haoqi Fan, Tullie Murrell, Heng Wang, Kalyan Vasudev Alwala, Yanghao Li, Yilei Li, Bo Xiong, Nikhila Ravi, Meng Li, Haichuan Yang, 2021. PyTorchVideo: A deep learning library for video understanding. In Proceedings of the 29th ACM International Conference on Multimedia. 3783–3786.

Digital Library

[12]

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision. 6202–6211.

[13]

Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. 2021. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems 34 (2021), 27503–27516.

[14]

John F Golding, Kim Doolan, Amish Acharya, Maryame Tribak, and Michael A Gresty. 2012. Cognitive cues and visually induced motion sickness. Aviation, space, and environmental medicine 83, 5 (2012), 477–482.

[15]

Donald Hearn, M Pauline Baker, and M Pauline Baker. 2004. Computer graphics with OpenGL. Vol. 3. Pearson Prentice Hall Upper Saddle River, NJ:.

[16]

Stefan Hell and Vasileios Argyriou. 2018. Machine learning architectures to predict motion sickness using a virtual reality rollercoaster simulation tool. In 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). IEEE, 153–156.

[17]

Dae Kyo Jeong, Sangbong Yoo, and Yun Jang. 2018. VR sickness measurement with EEG using DNN algorithm. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology. 1–2.

Digital Library

[18]

David M Johnson. 2005. Introduction to and review of simulator sickness research. (2005).

[19]

Behrang Keshavarz and Heiko Hecht. 2011. Axis rotation and visually induced motion sickness: the role of combined roll, pitch, and yaw motion. Aviation, space, and environmental medicine 82, 11 (2011), 1023–1029.

[20]

Behrang Keshavarz, Bernhard E Riecke, Lawrence J Hettinger, and Jennifer L Campos. 2015. Vection and visually induced motion sickness: how are they related?Frontiers in psychology 6 (2015), 472.

[21]

Hak Gu Kim, Wissam J Baddar, Heoun-taek Lim, Hyunwook Jeong, and Yong Man Ro. 2017. Measurement of exceptional motion in vr video contents for vr sickness assessment using deep convolutional autoencoder. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology. 1–7.

Digital Library

[22]

Hyun K Kim, Jaehyun Park, Yeongcheol Choi, and Mungyeong Choe. 2018. Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. Applied ergonomics 69(2018), 66–73.

[23]

Kihyun Kim, Sangmin Lee, Hak Gu Kim, Minho Park, and Yong Man Ro. 2019. Deep objective assessment model based on spatio-temporal perception of 360-degree video for VR sickness prediction. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 3192–3196.

[24]

Seongyeop Kim, Sangmin Lee, and Yong Man Ro. 2020. Estimating VR Sickness Caused By Camera Shake in VR Videography. In 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 3433–3437.

[25]

Ouren X Kuiper, Jelte E Bos, Cyriel Diels, and Eike A Schmidt. 2020. Knowing what’s coming: Anticipatory audio cues can mitigate motion sickness. Applied Ergonomics 85(2020), 103068.

[26]

Po-Chen Kuo, Li-Chung Chuang, Dong-Yi Lin, and Ming-Sui Lee. 2021. VR Sickness Assessment with Perception Prior and Hybrid Temporal Features. In 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 5558–5564.

[27]

Sangmin Lee, Seongyeop Kim, Hak Gu Kim, Min Seob Kim, Seokho Yun, Bumseok Jeong, and Yong Man Ro. 2019. Physiological fusion net: Quantifying individual vr sickness with content stimulus and physiological response. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 440–444.

[28]

Tae Min Lee, Jong-Chul Yoon, and In-Kwon Lee. 2019. Motion sickness prediction in stereoscopic videos using 3d convolutional neural networks. IEEE transactions on visualization and computer graphics 25, 5(2019), 1919–1927.

[29]

JJ-W Lin, Henry Been-Lirn Duh, Donald E Parker, Habib Abi-Rached, and Thomas A Furness. 2002. Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment. In Proceedings ieee virtual reality 2002. IEEE, 164–171.

[30]

Cheng-Li Liu and Shiaw-Tsyr Uang. 2012. A study of sickness induced within a 3D virtual store and combated with fuzzy control in the elderly. In 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery. IEEE, 334–338.

[31]

Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering (2021).

[32]

Juan Lopez, Leonardo Babun, Hidayet Aksu, and A Selcuk Uluagac. 2017. A survey on function and system call hooking approaches. Journal of Hardware and Systems Security 1, 2 (2017), 114–136.

[33]

Frank Luna. 2012. Introduction to 3D game programming with DirectX 11. Stylus Publishing, LLC.

[34]

Sergo Martirosov and Pavel Kopecek. 2017. CYBER SICKNESS IN VIRTUAL REALITY-LITERATURE REVIEW.Annals of DAAAM & Proceedings 28 (2017).

[35]

Natalie McHugh, Sungchul Jung, Simon Hoermann, and Robert W Lindeman. 2019. Investigating a physical dial as a measurement tool for cybersickness in virtual reality. In 25th ACM Symposium on Virtual Reality Software and Technology. 1–5.

Digital Library

[36]

Michelle Menard and Bryan Wagstaff. 2012. Game development with Unity. Course Technology.

[37]

Heeseok Oh and Wookho Son. 2022. Cybersickness and Its Severity Arising from Virtual Reality Content: A Comprehensive Study. Sensors 22, 4 (2022), 1314.

[38]

SDK OpenVR. [n.d.]. URL: https://github. com. ValveSoftware/openvr (visited on 02/02/2019) ([n. d.]).

[39]

Nitish Padmanaban, Timon Ruban, Vincent Sitzmann, Anthony M Norcia, and Gordon Wetzstein. 2018. Towards a machine-learning approach for sickness prediction in 360 stereoscopic videos. IEEE transactions on visualization and computer graphics 24, 4(2018), 1594–1603.

Digital Library

[40]

Dimitrios Saredakis, Ancret Szpak, Brandon Birckhead, Hannah AD Keage, Albert Rizzo, and Tobias Loetscher. 2020. Factors associated with virtual reality sickness in head-mounted displays: a systematic review and meta-analysis. Frontiers in human neuroscience 14 (2020), 96.

[41]

Richard HY So, Andy Ho, and WT Lo. 2001. A metric to quantify virtual scene movement for the study of cybersickness: Definition, implementation, and verification. Presence 10, 2 (2001), 193–215.

Digital Library

[42]

Richard HY So, WT Lo, and Andy TK Ho. 2001. Effects of navigation speed on motion sickness caused by an immersive virtual environment. Human factors 43, 3 (2001), 452–461.

[43]

Lorenzo Terenzi and Peter Zaal. 2020. Rotational and translational velocity and acceleration thresholds for the onset of cybersickness in virtual reality. In AIAA Scitech 2020 Forum. 0171.

[44]

Eoin Verling. 2005. Development of a User-Space Application for an HID Device, Using libhid. Linux Journal138(2005), 42–46.

Cited By

Martinez EMalik AMcMahan R(2024)CLOVR: Collecting and Logging OpenVR Data from SteamVR Applications2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00095(485-492)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00095
Wen EGupta CSasikumar PBillinghurst MWilmott JSkow EDey ANanayakkara S(2024)VR.net: A Real-world Dataset for Virtual Reality Motion Sickness ResearchIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337204430:5(2330-2336)Online publication date: May-2024
https://doi.org/10.1109/TVCG.2024.3372044
Nigatu HPickoff-White LCanny JChasins S(2023)Co-Designing for Transparency: Lessons from Building a Document Organization Tool in the Criminal Justice DomainProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594093(1463-1478)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594093

Index Terms

VRhook: A Data Collection Tool for VR Motion Sickness Research

Recommendations

How Visual Motion Cues Can Influence Sickness For In-Car VR
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

This video demonstrates our research into the use of VR Head Mounted Displays (HMDs) in-car and in-motion. Immersive HMDs offer new possibilities for entertainment and productivity during travel. However, their use is confounded by motion sickness, ...
Investigating motion sickness techniques for immersive virtual environments
PETRA '19: Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments

Motion sickness is one of important issues in immersive virtual environments. In some cases it may last for hours after participation in the virtual experience. Reducing the amount of motion sickness in healthcare applications is of great importance. ...
Demand Characteristics in Assessing Motion Sickness in a Virtual Environment: Or Does Taking a Motion Sickness Questionnaire Make You Sick?

The experience of motion sickness in a virtual environment may be measured through pre and postexperiment self-reported questionnaires such as the Simulator Sickness Questionnaire (SSQ). Although research provides converging evidence that users of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '22: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

October 2022

1363 pages

ISBN:9781450393201

DOI:10.1145/3526113

Editors:
Maneesh Agrawala
Stanford University, USA
,
Jacob O. Wobbrock
University of Washington, USA
,
Eytan Adar
University of Michigan, USA
,
Vidya Setlur
Tableau Research, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

UIST '22

Sponsor:

UIST '22: The 35th Annual ACM Symposium on User Interface Software and Technology

October 29 - November 2, 2022

OR, Bend, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
682
Total Downloads

Downloads (Last 12 months)203
Downloads (Last 6 weeks)20

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Martinez EMalik AMcMahan R(2024)CLOVR: Collecting and Logging OpenVR Data from SteamVR Applications2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00095(485-492)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00095
Wen EGupta CSasikumar PBillinghurst MWilmott JSkow EDey ANanayakkara S(2024)VR.net: A Real-world Dataset for Virtual Reality Motion Sickness ResearchIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337204430:5(2330-2336)Online publication date: May-2024
https://doi.org/10.1109/TVCG.2024.3372044
Nigatu HPickoff-White LCanny JChasins S(2023)Co-Designing for Transparency: Lessons from Building a Document Organization Tool in the Criminal Justice DomainProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594093(1463-1478)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594093

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents