skip to main content
10.1145/3625468.3647625acmconferencesArticle/Chapter ViewAbstractPublication PagesmmsysConference Proceedingsconference-collections
research-article

User Study-based Models of Game Player Quality of Experience with Frame Display Time Variation

Published: 17 April 2024 Publication History

Abstract

Computer games are often rendered with inconsistent frame timing (frame jitter), particularly in cloud-based game streaming where frames traverse network bottlenecks before being rendered. While previous studies have helped understand the Quality of Experience (QoE) with frame jitter, derived models have tended to be limited in their prediction ability for conditions not yet tested. This paper combines results from four different user studies that assess QoE based on frame jitter, the studies differing in games, game systems, and methods of induced frame time variation. Analysis of the results shows the degree to which frame jitter degrades QoE, and that playout interruption sizes matter while interrupt frequencies do not. The rich user study-based data set provides the basis for models for predicting game player QoE with frame jitter - models which should be predictive for both cloud-based game streaming and traditional games, and for a wide range of player actions and game genres.

References

[1]
Hamed Ahmadi, Sepideh Khoshnood, Mahmoud Reza Hashemi, and Shervin Shirmohammadi. 2013. Efficient Bitrate Reduction using a Game Attention Model in Cloud Gaming. In Proceedings of the IEEE International Symposium on Haptic Audio Visual Environments and Games (HAVE). Ottowa, ON, Canada, 103--108.
[2]
Josh Allard, Andrew Roskuski, and Mark Claypool. 2020. Measuring and Modeling the Impact of Buffering and Interrupts on Streaming Video Quality of Experience. In Proceedings of the 18th International Conference on Advances in Mobile Computing & Multimedia (MoMM). Chiang Mai, Thailand.
[3]
Nabajeet Barman, Saman Zadtootaghaj, Steven Schmidt, Maria G. Martini, and Sebastian Möller. 2018. GamingVideoSET: A Dataset for Gaming Video Streaming Applications. In Workshop on Network and Systems Support for Games (NetGames). Amsterdam, The Netherlands, 1--6. https://doi.org/10.1109/NetGames.2018.8463362
[4]
Kashif Bilal and Aiman Erbad. 2017. Impact of Multiple Video Representations in Live Streaming: A Cost, Bandwidth, and QoE Analysis. In IEEE International Conference on Cloud Engineering (IC2E). Vancouver, Canada, 88--94. https://doi.org/10.1109/IC2E.2017.20
[5]
J.H. Brockmyer, C.M. Fox, K.A. Curtiss, E. McBroom, K.M. Burkhart, and J.N. Pidruzny. 2009. The Development of the Game Engagement Questionaire: A Measure of Engagement in Video Game Playing. Journal of Experimental Social Psychology (2009), 624--634.
[6]
Kajal Claypool and Mark Claypool. 2007. On Frame Rate and Player Performance in First Person Shooter Games. Springer Multimedia Systems Journal (MMSJ) 13 (2007), 3--17.
[7]
Mark Claypool. 2009. Motion and Scene Complexity for Streaming Video Games. In Proceedings of the 4th ACM International Conference on the Foundations of Digital Games (FDG). Florida, USA.
[8]
Mark Claypool and Kajal Claypool. 2009. Perspectives, Frame Rates and Resolutions: It's all in the Game. In Proceedings of the 4th ACM International Conference on the Foundations of Digital Games (FDG). Florida, USA.
[9]
Mark Claypool, Kajal Claypool, and Feissal Damaa. 2006. The Effects of Frame Rate and Resolution on Users Playing First Person Shooter Games. In Proceedings ACM/SPIE Multimedia Computing and Networking (MMCN) Conference. San Jose, CA, USA.
[10]
Mark Claypool, David Finkel, Alexander Grant, and Michael Solano. 2012. Thin to Win? Network Performance Analysis of the OnLive Thin Client Game System. In Proceedings of the 11th ACM Network and System Support for Games (NetGames). Venice, Italy.
[11]
A.L. Cox, C. Jennett, C. Paul, S. Dhoparee, A. Epps, T. Tijs, and A. Walton. 2008. Measuring and Defining the Experience of Immersion in Games. International Journal of Human-Computer Studies 66, 9 (2008).
[12]
(Anonymous for double-blind review). [n. d.].
[13]
GameTechDev. 2022. PresentMon V1.8.0. Online: https://github.com/GameTechDev/PresentMon.
[14]
Su Guan-Ming, Su Xiao, Bai Yan, Wang Mea, Athanasios V. Vasilakos, and Haohong Wang. 2015. QoE in Video Streaming over Wireless Networks: Perspectives and Research Challenges. Wireless Networking (2015). https://doi.org/10.1007/s11276-015-1028-7
[15]
Cameron Gutman. 2022. Moonlight PC v4.3.1. Online: https://github.com/moonlight-stream.
[16]
Ravin Balakrishnan Géry Casiez, Daniel Vogel and Andy Cockburn. 2008. The Impact of Control-Display Gain on User Performance in Pointing Tasks. Human-Computer Interaction 23, 3 (2008), 215--250. https://doi.org/10.1080/07370020802278163
[17]
International Telecommunication Union. 2008. ""ITU-T Recommendation P.910: Subjective Video Quality Assessment Methods for Multimedia Applications. https://www.itu.int/rec/T-REC-P.910-200804-S/en.
[18]
LizardByte. 2022. Sunshine. Online: https://github.com/LizardByte/Sunshine.
[19]
M. Manzano, J. A. Hernández, M. Urueña, and E. Calle. 2012. An Empirical Study of Cloud Gaming. In Proceeding of the 11th Annual Workshop on Network and Systems Support for Games (NetGames). 1--2.
[20]
Alexey Nicolaychuk. 2013. RivaTuner Statistics Server. Online: https://www.guru3d.com/content-page/rivatuner.html.
[21]
Péter Orosz, Tamás Skopkó, Zoltán Nagy, Pál Varga, and László Gyimóthi. 2014. A Case Study on Correlating Video QoS and QoE. In IEEE Network Operations and Management Symposium (NOMS). Krakow, Poland, 1--5. https://doi.org/10.1109/NOMS.2014.6838399
[22]
N. Rao, A. Maleki, F. Chen, W. Chen, C. Zhang, N. Kaur, and A. Haque. 2019. Analysis of the Effect of QoS on Video Conferencing QoE. In 15th International Wireless Communications & Mobile Computing Conference (IWCMC). 1267--1272. https://doi.org/10.1109/IWCMC.2019.8766591
[23]
Henrique Souza Rossi, Niclas Ögren, Karan Mitra, Irina Cotanis, Christer Åhlund, and Per Johansson. 2022. Subjective Quality of Experience Assessment in Mobile Cloud Games. In GLOBECOM - IEEE Global Communications Conference. Rio de Janeiro, Brazil, 1918--1923. https://doi.org/10.1109/GLOBECOM48099.2022.10001407
[24]
Saeed Shafiee Sabet, Steven Schmidt, Saman Zadtootaghaj, Carsten Griwodz, and Sebastian Möller. 2020. Delay Sensitivity Classification of Cloud Gaming Content. In Proceedings of the International Workshop on Immersive Mixed and Virtual Environment Systems (MMVE). Istanbul, Turkey.
[25]
S. Schmidt. 2022. Assessing the Quality of Experience of Cloud Gaming Services. Springer International Publishing, Switzerland. https://books.google.com/books?id=LRaIEAAAQBAJ
[26]
S. Schmidt, S. Zadtootaghaj, and S. Moller. 2017. Towards the Delay Sensitivity of Games: There is More than Genres. In Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). Erfurt, Germany.
[27]
Ivan Slivar, Mirko Suznjevic, and Lea Skorin-Kapov. 2015. The Impact of Video Encoding Parameters and Game Type on QoE for Cloud Gaming: A Case Study Using the Steam Platform. In Seventh International Workshop on Quality of Multimedia Experience (QoMEX). Costa Navarino, Messinia, Greece.
[28]
Ivan Slivar, Mirko Suznjevic, Lea Skorin-Kapov, and Maja Matijasevic. 2014. Empirical QoE Study of In-home Streaming of Online Games. In 13th Annual Workshop on Network and Systems Support for Games. Nagoya, Japan, 1--6. https://doi.org/10.1109/NetGames.2014.7010133
[29]
Josef Spjut, Ben Boudaoud, Kamran Binaee, Jonghyun Kim, Alexander Majercik, Morgan McGuire, David Luebke, and Joohwan Kim. 2019. Latency of 30 ms Benefits First Person Targeting Tasks More Than Refresh Rate Above 60 Hz. In SIGGRAPH Asia Technical Briefs. ACM, New York, NY, USA.
[30]
Steve and Oliver Baker. 2022. SuperTuxKart. https://supertuxkart.net/ (Accessed Sept 10, 2023).
[31]
Ya-Fan Su, Yi-Hsuan Yang, Meng-Ting Lu, and Homer H. Chen. 2009. Smooth Control of Adaptive Media Playout for Video Streaming. IEEE Transactions on Multimedia 11, 7 (2009), 1331--1339. https://doi.org/10.1109/TMM.2009.2030543
[32]
Mirko Suznjevic, Ivan Slivar, and Lea Skorin-Kapov. 2016. Analysis and QoE Evaluation of Cloud Gaming Service Adaptation under Different Network Conditions: The Case of NVIDIA GeForce NOW. In Eighth International Conference on Quality of Multimedia Experience (QoMEX). Lisbon, Portugal, 1--6. https://doi.org/10.1109/QoMEX.2016.7498968
[33]
Ayano Tatematsu, Yutaka Ishibashi, Norishige Fukushima, and Shinji Sugawara. 2010. QoE Assessment in Haptic Media, Sound and Video Transmission: Influences of Network Latency. In IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR 2010). Vancouver, British Columbia, Canada, 1--6. https://doi.org/10.1109/CQR.2010.5619913
[34]
International Telecommunication Union. 2001. Perceptual Evaluation of Speech Quality (PESQ) - An Objective Method for End-to-end Speech Quality Assessment of Narrow-band Telephone Networks and Speech Codecs. ITU Recommendation ITU-T P.862.
[35]
wepc.com. 2022. Video Game Industry Statistics, Trends and Data In 2022. Online: https://www.wepc.com/news/video-game-statistics/. (Accessed October 26, 2022).
[36]
Wikipedia contributors. [n.d.]. Counter-strike: Global Offensive (CS:GO). https://en.wikipedia.org/wiki/Counter-Strike:_Global_Offensive [Accessed 28-Nov-2023].
[37]
Wikipedia contributors. 2017. Hollow Knight. https://en.wikipedia.org/wiki/Hollow_Knight [Accessed 12-Sep-2023].
[38]
Wikipedia contributors. 2018. Bloons TD 6. https://en.wikipedia.org/wiki/Bloons_TD_6 [Accessed 12-Sep-2023].
[39]
Wikipedia contributors. 2020. Hades (video game). https://en.wikipedia.org/wiki/Hades_(video_game) [Accessed 12-Sep-2023].
[40]
Wikipedia contributors. 2022. Network emulation. https://en.wikipedia.org/wiki/Network_emulation [Accessed 20-Oct-2022].
[41]
Xiaokun Xu and Mark Claypool. 2022. Measurement of Cloud-Based Game Streaming System Response to Competing TCP Cubic or TCP BBR Flows. In Proceedings of the 22nd ACM Internet Measurement Conference. Nice, France, 305--316.
[42]
Saman Zadtootaghaj, Steven Schmidt, and Sebastian Möller. 2018. Modeling Gaming QoE: Towards the Impact of Frame Rate and Bit Rate on Cloud Gaming. In Tenth International Conference on Quality of Multimedia Experience (QoMEX). Sardinia, Italy, 1--6.

Cited By

View all
  • (2024)Preventing performance dips at the intersection of Nomophobia and behavioral regulation in organizations: an experimental studyJournal of Economic and Administrative Sciences10.1108/JEAS-06-2024-0189Online publication date: 16-Dec-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MMSys '24: Proceedings of the 15th ACM Multimedia Systems Conference
April 2024
557 pages
ISBN:9798400704123
DOI:10.1145/3625468
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2024

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

MMSys '24
Sponsor:

Acceptance Rates

Overall Acceptance Rate 176 of 530 submissions, 33%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)78
  • Downloads (Last 6 weeks)4
Reflects downloads up to 01 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Preventing performance dips at the intersection of Nomophobia and behavioral regulation in organizations: an experimental studyJournal of Economic and Administrative Sciences10.1108/JEAS-06-2024-0189Online publication date: 16-Dec-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media