skip to main content
research-article

Modeling the User Experience of Watching 360° Videos with Head-Mounted Displays

Published: 27 January 2022 Publication History

Abstract

Conducting user studies to quantify the Quality of Experience (QoE) of watching the increasingly more popular 360° videos in Head-Mounted Displays (HMDs) is time-consuming, tedious, and expensive. Deriving QoE models, however, is very challenging because of the diverse viewing behaviors and complex QoE features and factors. In this article, we compile a wide spectrum of QoE features and factors that may contribute to the overall QoE. We design and conduct a user study to build a dataset of the overall QoE, QoE features, and QoE factors. Using the dataset, we derive the QoE models for both the Mean Opinion Score (MOS) and Individual Score (IS), where MOS captures the aggregated QoE across all subjects, while IS captures the QoE of individual subjects. Our derived overall QoE models achieve 0.98 and 0.91 in Pearson’s Linear Correlation Coefficient (PLCC) for MOS and IS, respectively. Besides, we make several new observations on our user study results, such as (1) content factors dominate the overall QoE across all factor categories, (2) Video Multi-Method Assessment Fusion (VMAF) is the dominating factor among content factors, and (3) the perceived cybersickness is affected by human factors more among others. Our proposed user study design is useful for QoE modeling (specifically) and subjective evaluations (in general) of emerging 360° tiled video streaming to HMDs.

References

[1]
Muhammad Shahid Anwar, Jing Wang, Sadique Ahmad, Wahab Khan, Asad Ullah, Mudassir Shah, and Zesong Fei. 2020. Impact of the impairment in 360-degree videos on users VR involvement and machine learning-based QoE predictions. IEEE Access 8 (2020), 204585–204596.
[2]
Muhammad Shahid Anwar, Jing Wang, Wahab Khan, Asad Ullah, Sadique Ahmad, and Zesong Fei. 2020. Subjective QoE of 360-degree virtual reality videos and machine learning predictions. IEEE Access (2020), 148084–148099.
[3]
Sana Aroussi and Abdelhamid Mellouk. 2014. Survey on machine learning-based QoE-QoS correlation models. In Proc. of IEEE International Conference on Computing, Management and Telecommunications (ComManTel’14).
[4]
Debasish Basak, Srimanta Pal, and Dipak Chandra Patranabis. 2007. Support vector regression. Neural Information Processing-Letters and Reviews 11, 10 (2007), 203–224.
[5]
Maximino Bessa, Miguel Melo, David Narciso, Luis Barbosa, and Jose Vasconcelos-Raposo. 2016. Does 3D 360 video enhance user’s VR experience: An evaluation study. In Proc. of International Conference on Human Computer Interaction (Interaction’16).
[6]
Khadija Bouraqia, Essaid Sabir, Mohamed Sadik, and Latif Ladid. 2020. Quality of experience for streaming services: Measurements, challenges and insights. IEEE Access 8 (2020), 13341–13361.
[7]
Lauren Bush, Ursula Hess, and George Wolford. 1993. Transformations for within-subject designs: A Monte Carlo investigation.APA Psychological Bulletin 113, 3 (1993), 566–579.
[8]
Sumohana S. Channappayya, Alan C. Bovik, Constantine Caramanis, and Robert W. Heath Jr.2008. SSIM-Optimal linear image restoration. In Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’08).
[9]
HTC Co.2019. VIVE: Discover Virtual Reality Beyond Imagination. Retrieved April 8, 2019, from https://www.vive.com/us/.
[10]
HTC Corporation. 2020. Eye Tracking SDK (SRanipal). Retrieved January 22, 2020, from https://developer.vive.com/resources/knowledgebase/vive-sranipal-sdk/.
[11]
HTC Corporation. 2020. VIVE PRO EYE. Retrieved January 22, 2020, from https://enterprise.vive.com/us/product/vive-pro-eye/.
[12]
Simone Croci, Cagri Ozcinar, Emin Zerman, Julián Cabrera, and Aljosa Smolic. 2019. Voronoi-based objective quality metrics for omnidirectional video. In Proc. of International Conference on Quality of Multimedia Experience (QoMEX’19).
[13]
Simone Croci, Cagri Ozcinar, Emin Zerman, Sebastian Knorr, Julián Cabrera, and Aljosa Smolic. 2020. Visual attention-aware quality estimation framework for omnidirectional video using spherical Voronoi diagram. Springer Quality and User Experience 5, 1 (2020), 4:1–4:17.
[14]
Darragh Egan, Sean Brennan, John Barrett, Yuansong Qiao, Christian Timmerer, and Niall Murray. 2016. An evaluation of heart rate and electrodermal activity as an objective QoE evaluation method for immersive virtual reality environments. In Proc. of International Conference on Quality of Multimedia Experience (QoMEX’16).
[15]
C. Fan, W. Lo, Y. Pai, and C. Hsu. 2019. A survey on 360° video streaming: Acquisition, transmission, and display. Computing Surveys 52, 4 (2019), 71:1–71:36.
[16]
C. Fan, S. Yen, C. Huang, and C. Hsu. 2020. On the optimal encoding ladder of tiled 360° videos for head-mounted virtual reality. IEEE Transactions on Circuits and Systems for Video Technology 31, 4 (July 2020), 1632–1647.
[17]
Ajoy S. Fernandes and Steven K. Feiner. 2016. Combating VR sickness through subtle dynamic Field-of-View modification. In Proc. of IEEE Symposium on 3D User Interfaces (3DUI’16).
[18]
David A. Freedman. 2009. Statistical Models: Theory and Practice. Cambridge University Press.
[19]
Stephan Fremerey, Steve Göring, Rakesh Rao Ramachandra Rao, Rachel Huang, and Alexander Raake. 2020. Subjective test dataset and meta-data-based models for 360° streaming video quality. In Proc. of IEEE International Workshop on Multimedia Signal Processing (MMSP’20).
[20]
Jerome H. Friedman. 2001. Greedy function approximation: A gradient boosting machine. JSTOR Annals of Statistics 29, 5 (2001), 1632–1647.
[21]
Jukka Hakkinen, Tero Vuori, and M. Paakka. 2002. Postural stability and sickness symptoms after HMD use. In IEEE International Conference on Systems, Man and Cybernetics, Vol. 1.
[22]
Tobias Hoßfeld, Poul Heegaard, Martín Varela, and Sebastian Möller. 2016. QoE beyond the MOS: An in-depth look at QoE via better metrics and their relation to MOS. Quality and User Experience 1, 1 (2016), 2:1–2:23.
[23]
Chih-Fan Hsu, Anthony Chen, Cheng-Hsin Hsu, Chun-Ying Huang, Chin-Laung Lei, and Kuan-Ta Chen. 2017. Is foveated rendering perceivable in virtual reality: Exploring the efficiency and consistency of quality assessment methods. In Proc. of ACM International Conference on Multimedia (MM’17).
[24]
Mingkai Huang, Qiu Shen, Zhan Ma, Alan Conrad Bovik, Praful Gupta, Rongbing Zhou, and Xun Cao. 2018. Modeling the perceptual quality of immersive images rendered on head mounted displays: Resolution and compression. IEEE Transactions on Image Processing 27, 12 (2018), 6039–6050.
[25]
Netflix Inc.2019. VMAF - Video Multi-Method Assessment Fusion. https://github.com/Netflix/vmaf.
[26]
ISO 23009-1:2012 2012. Information Technology – Dynamic Adaptive Streaming Over HTTP (DASH) – Part 1: Media Presentation Description and Segment Formats. (March 2012).
[27]
ISO/IEC JTC1/SC29/WG11/N17197 2017. Algorithm Descriptions of Projection Format Conversion and Video Quality Metrics in 360Lib. Standard. International Telecommunication Union.
[29]
ITU, COM 9-80-E 2000. Final Report from the Video Quality Experts Group on the Validation of Objective Models of Video Quality Assessment. Technical Report. VQEG.
[30]
ITU Telecommunication Standardization Sector. 1999. Subjective video quality assessment methods for multimedia applications. ITU-T Recommendation P.910 (September 1999).
[31]
Brendan John, Pallavi Raiturkar, Olivier Le Meur, and Eakta Jain. 2019. A benchmark of four methods for generating 360° saliency maps from eye tracking data. World Scientific International Journal of Semantic Computing 13, 3 (2019), 329–341.
[32]
Parikshit Juluri, Venkatesh Tamarapalli, and Deep Medhi. 2016. Measurement of quality of experience of video-on-demand services: A survey. IEEE Communications Surveys & Tutorials 18, 1 (2016), 401–418.
[33]
Satu Jumisko-Pyykkö and Teija Vainio. 2010. Framing the context of use for mobile HCI. IGI Global International Journal of Mobile Human Computer Interaction 2, 4 (2010), 1–28.
[34]
Robert S. Kennedy, Norman E. Lane, Kevin S. Berbaum, and Michael G. Lilienthal. 1993. Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. International Journal of Aviation Psychology 3, 3 (1993), 203–220.
[35]
István Ketykó, Katrien De Moor, Toon De Pessemier, Adrián Juan Verdejo, Kris Vanhecke, Wout Joseph, Luc Martens, and Lieven De Marez. 2010. QoE measurement of mobile YouTube video streaming. In Proc. of Workshop on Mobile Video Delivery (MoVid’10).
[36]
Hak Gu Kim, Heoun-Taek Lim, Sangmin Lee, and Yong Man Ro. 2018. VRSA net: VR sickness assessment considering exceptional motion for 360 VR video. IEEE Transactions on Image Processing 28, 4 (2018), 1646–1660.
[37]
Jaekvung Kim, Woojae Kim, Sewoong Ahn, Jinwoo Kim, and Sanghoon Lee. 2018. Virtual reality sickness predictor: Analysis of visual-vestibular conflict and VR contents. In Proc. of IEEE International Conference on Quality of Multimedia Experience (QoMEX’18).
[38]
Tae Min Lee, Jong-Chul Yoon, and In-Kwon Lee. 2019. Motion sickness prediction in stereoscopic videos using 3D convolutional neural networks. IEEE Transactions on Visualization and Computer Graphics 25, 5 (2019), 1919–1927.
[39]
Andy Liaw, Matthew Wiener, et al. 2002. Classification and Regression by RandomForest. R News 2, 3 (2002), 18–22.
[40]
Wen-Chih Lo, Ching-Ling Fan, Shou-Cheng Yen, and Cheng-Hsin Hsu. 2017. Performance measurements of 360° video streaming to head-mounted displays over live 4G cellular networks. In Proc. of Asia-Pacific Network Operations and Management Symposium (APNOMS’17).
[41]
Bruce D. Lucas and Takeo Kanade. 1981. An iterative image registration technique with an application to stereo vision. In Proc. of the International Joint Conference on Artificial Intelligence (IJCAI’81).
[42]
MATLAB 2020. MATLAB - MathWorks. Retrieved August 4, 2020, from https://www.mathworks.com/products/matlab.html.
[43]
Kiran Misra, Andrew Segall, Michael Horowitz, Shilin Xu, Arild Fuldseth, and Minhua Zhou. 2013. An overview of tiles in HEVC. IEEE Journal of Selected Topics in Signal Processing 7, 6 (December 2013), 969–977.
[44]
Sebastian Moller, Marcel Waltermann, and Marie-Neige Garcia. 2014. Features of quality of experience. In Quality of Experience, Sebastian Moller and Alexander Raake Editors (Eds.). Springer US, Chapter 5.
[45]
Bonnie A. Nardi. 1996. Context and Consciousness: Activity Theory and Human-Computer Interaction. MIT Press.
[47]
nmsl nthu. 2020. QoE-Modeling-for-360-Degree-Videos-Dataset. Retrieved August 11, 2020, from https://github.com/nmsl-nthu/QoE-Modeling-for-360-Degree-Videos-Dataset.
[48]
LLC. Oculus VR. 2019. Facebook Oculus Rift. Retrieved April 8, 2019, from https://www.oculus.com/.
[49]
Marta Orduna, César Díaz, Lara Muñoz, Pablo Pérez, Ignacio Benito, and Narciso García. 2019. Video multimethod assessment fusion (VMAF) on 360VR contents. IEEE Transactions on Consumer Electronics 66, 1 (2019), 22–31.
[50]
Nitish Padmanaban, Timon Ruban, Vincent Sitzmann, Anthony M. Norcia, and Gordon Wetzstein. 2018. Towards a machine-learning approach for sickness prediction in 360 stereoscopic videos. IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1594–1603.
[51]
Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Mathieu Blondel, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.
[52]
Alexander Raake, Ashutosh Singla, Rakesh Rao Ramachandra Rao, Werner Robitza, and Frank Hofmeyer. 2020. SiSiMo: Towards simulator sickness modeling for 360° videos viewed with an HMD. In Proc. of IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW’20).
[53]
Md Abdur Rahman, Abdulmotaleb El Saddik, and Wail Gueaieb. 2010. Augmenting context awareness by combining body sensor networks and social networks. IEEE Transactions on Instrumentation and Measurement 60, 2 (2010), 345–353.
[54]
Yashas Rai, Patrick Le Callet, and Philippe Guillotel. 2017. Which saliency weighting for omni directional image quality assessment? In Proc. of IEEE International Conference on Quality of Multimedia Experience (QoMEX’17).
[55]
Georg Regal, Raimund Schatz, Johann Schrammel, and Stefan Suette. 2018. VRate: A Unity3D asset for integrating subjective assessment questionnaires in virtual environments. In Proc. of IEEE International Conference on Quality of Multimedia Experience (QoMEX’18).
[56]
Ulrich Reiter, Kjell Brunnström, Katrien De Moor, Mohamed-Chaker Larabi, Manuela Pereira, Antonio Pinheiro, Junyong You, and Andrej Zgank. 2014. Factors influencing quality of experience. In Quality of Experience, Sebastian Moller and Alexander Raake Editors (Eds.). Springer US, Chapter 4.
[57]
Raimund Schatz, Georg Regal, Stephanie Schwarz, Stefan Suettc, and Marina Kempf. 2018. Assessing the QoE impact of 3D rendering style in the context of VR-based training. In Proc. of IEEE International Conference on Quality of Multimedia Experience (QoMEX’18).
[58]
ITU Telecommunication Standardization Sector. 2008. Subjective video quality assessment methods for multimedia applications. ITU-T Recommendation P.910 (April 2008).
[59]
ITU Telecommunication Standardization Sector. 2016. Mean opinion score (MOS) terminology. ITU-T Recommendation P.800.1 (July 2016).
[60]
ITU Telecommunication Standardization Sector. 2016. Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment. ITU-T Recommendation P.913 (March 2016).
[61]
ITU Telecommunication Standardization Sector. 2016. Series P: Terminals and subjective and objective assessment methods. ITU-T Recommendation P.800.2 (July 2016).
[62]
Ronald E. Shiffler. 1988. Maximum Z scores and outliers. American Statistician 42, 1 (1988), 79–80.
[63]
Ashutosh Singla, Stephan Fremerey, Werner Robitza, Pierre Lebreton, and Alexander Raake. 2017. Comparison of subjective quality evaluation for HEVC encoded omnidirectional videos at different bit-rates for UHD and FHD resolution. In Proc. of ACM Multimedia Thematic Workshops.
[64]
Ashutosh Singla, Stephan Fremerey, Werner Robitza, and Alexander Raake. 2017. Measuring and comparing QoE and simulator sickness of omnidirectional videos in different head mounted displays. In Proc. of International Conference on Quality of Multimedia Experience (QoMEX’17).
[65]
Ashutosh Singla, Steve Goring, Alexander Raake, Britta Meixner, Rob Koenen, and Thomas Buchholz. 2019. Subjective quality evaluation of tile-based streaming for omnidirectional videos. In Proc. of ACM Conference on Multimedia Systems (MMSys’19).
[66]
Iraj Sodagar. 2011. The MPEG-DASH standard for multimedia streaming over the Internet. IEEE Multimedia 18, 4 (2011), 62–67.
[67]
Wei Song, Yao Xiao, Dian Tjondronegoro, and Antonio Liotta. 2015. QoE modeling for VP9 and H.265 videos on mobile devices. In Proc. of ACM International Conference on Multimedia (MM’15).
[68]
Charles Spearman. 1904. The proof and measurement of association between two things. American Journal of Psychology 15, 1 (1904), 72–101.
[69]
Katy Tcha-Tokey, Emilie Loup-Escande, Olivier Christmann, and Simon Richir. 2016. A questionnaire to measure the user experience in immersive virtual environments. In Proc. of ACM Virtual Reality International Conference (VRIC’16).
[70]
[71]
Trejkaz. 2020. Equi-Angular Cubemap Skybox for Unity. Retrieved August 11, 2020, from https://github.com/trejkaz/EACSkyboxShader.
[72]
Izabela Tucker. 2011. Perceptual Video Quality Dimensions. Master’s thesis. Technische, Universität Berlin, Berlin, Germany.
[73]
Unity 2020. Unity. Retrieved May 22, 2020, from https://unity3d.com/.
[74]
Evgeniy Upenik, Martin Rerabek, and Touradj Ebrahimi. 2016. Testbed for subjective evaluation of omnidirectional visual content. In Proc. of Picture Coding Symposium (PCS’16).
[75]
Evgeniy Upenik, Martin Rerabek, and Touradj Ebrahimi. 2017. On the performance of objective metrics for omnidirectional visual content. In Proc. of International Conference on Quality of Multimedia Experience (QoMEX’17).
[76]
Martín Varela, Lea Skorin-Kapov, and Touradj Ebrahimi. 2014. Quality of service versus quality of experience. In Quality of Experience, Sebastian Moller and Alexander Raake Editors (Eds.). Springer US, Chapter 6.
[77]
Marko Viitanen, Ari Koivula, Ari Lemmetti, Arttu Yla-Outinen, Jarno Vanne, and Timo D. Hamalainen. 2016. Kvazaar: Open-Source HEVC/H.265 Encoder. In Proc. of ACM International Conference on Multimedia (MM’16).
[78]
Sara Vlahovic, Mirko Suznjevic, and Lea Skorin-Kapov. 2018. Subjective assessment of different locomotion techniques in virtual reality environments. In Proc. of IEEE International Conference on Quality of Multimedia Experience (QoMEX’18).
[79]
Hui Wang, Vu-Thanh Nguyen, Wei Tsang Ooi, and Mun Choon Chan. 2014. Mixing tile resolutions in tiled video: A perceptual quality assessment. In Proc. of ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV’14).
[80]
Martin Weier, Thorsten Roth, Ernst Kruijff, André Hinkenjann, Arsène Pérard-Gayot, Philipp Slusallek, and Yongmin Li. 2016. Foveated real-time ray tracing for head-mounted displays. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 289–298.
[81]
Frank W. Weymouth. 1958. Visual sensory units and the minimal angle of resolution. Elsevier American Journal of Ophthalmology 46, 1 (1958), 102–113.
[82]
Lan Xie, Zhimin Xu, Yixuan Ban, Xinggong Zhang, and Zongming Guo. 2017. 360ProbDASH: Improving QoE of 360 Video streaming using Tile-Based HTTP adaptive streaming. In Proc. of ACM International Conference on Multimedia (MM’17).
[83]
Shaowei Xie, Yiling Xu, Qiaojian Qian, Qiu Shen, Zhan Ma, and Wenjun Zhang. 2018. Modeling the perceptual impact of viewport adaptation for immersive video. In Proc. of IEEE International Symposium on Circuits and Systems (ISCAS’18).
[84]
Shun-Huai Yao, Ching-Ling Fan, and Cheng-Hsin Hsu. 2019. Towards quality-of-experience models for watching 360° videos in head-mounted virtual reality. In Proc. of International Conference on Quality of Multimedia Experience (QoMEX’19).
[85]
Matt Yu, Haricharan Lakshman, and Bernd Girod. 2015. Content adaptive representations of omnidirectional videos for cinematic virtual reality. In Proc. of ACM International Workshop on Immersive Media Experiences (ImmersiveMe’15).

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications
ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 1
January 2022
517 pages
ISSN:1551-6857
EISSN:1551-6865
DOI:10.1145/3505205
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2022
Accepted: 01 April 2021
Revised: 01 February 2021
Received: 01 August 2020
Published in TOMM Volume 18, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Virtual reality
  2. user study
  3. mean opinion score
  4. individual score

Qualifiers

  • Research-article
  • Refereed

Funding Sources

  • Ministry of Science and Technology of Taiwan
  • NOVATEK Fellowship

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)158
  • Downloads (Last 6 weeks)6
Reflects downloads up to 14 Sep 2024

Other Metrics

Citations

Cited By

View all

View Options

Get Access

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media