research-article

Cloud Gaming with Foveated Video Encoding

Authors:

Gazi Karam Illahi,

Thomas Van Gemert,

Matti Siekkinen,

Antti Oulasvirta,

Antti Ylä-JääskiAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 16, Issue 1

Article No.: 7, Pages 1 - 24

https://doi.org/10.1145/3369110

Published: 17 February 2020 Publication History

Abstract

Cloud gaming enables playing high-end games, originally designed for PC or game console setups, on low-end devices such as netbooks and smartphones, by offloading graphics rendering to GPU-powered cloud servers. However, transmitting the high-resolution video requires a large amount of network bandwidth, even though it is a compressed video stream. Foveated video encoding (FVE) reduces the bandwidth requirement by taking advantage of the non-uniform acuity of human visual system and by knowing where the user is looking. Based on a consumer-grade real-time eye tracker and an open source cloud gaming platform, we provide a cloud gaming FVE prototype that is game-agnostic and requires no modifications to the underlying game engine. In this article, we describe the prototype and its evaluation through measurements with representative games from different genres to understand the effect of parametrization of the FVE scheme on bandwidth requirements and to understand its feasibility from the latency perspective. We also present results from a user study on first-person shooter games. The results suggest that it is possible to find a “sweet spot” for the encoding parameters so the users hardly notice the presence of foveated encoding but at the same time the scheme yields most of the achievable bandwidth savings.

References

[1]

Hamed Ahmadi, Saman Zad Tootaghaj, Mahmoud Reza Hashemi, and Shervin Shirmohammadi. 2014. A game attention model for efficient bit rate allocation in cloud gaming. Multim. Syst. 20, 5 (Oct. 2014), 485--501.

Digital Library

[2]

Zahid Akhtar and Tiago H. Falk. 2017. Audio-visual multimedia quality assessment: A comprehensive survey. IEEE Access 5 (2017), 21090--21117.

[3]

Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency requirements for foveated rendering in virtual reality. ACM Trans. Appl. Percept. 14, 4, Article 25 (Sept. 2017), 13 pages.

Digital Library

[4]

Ayub Bokani. 2014. Empirical evaluation of real-time video foveation. In Proceedings of the Workshop on Design, Quality and Deployment of Adaptive Video Streaming. ACM, 45--46.

Digital Library

[5]

Speedtest by Ookla. 2019. Monthly Comparisons of Internet Speeds from Around the World. Retrieved from https://www.speedtest.net/global-index.

[6]

Wei Cai, Min Chen, and Victor C. M. Leung. 2014. Toward gaming as a service. IEEE Internet Comput. 18, 3 (May 2014), 12--18.

[7]

Wei Cai, Conghui Zhou, Victor C. M. Leung, and Min Chen. 2013. A cognitive platform for mobile cloud gaming. In Proceedings of the IEEE 5th International Conference on Cloud Computing Technology and Science, Vol. 1. 72--79.

Digital Library

[8]

Kuan-Ta Chen, Yu-Chun Chang, Hwai-Jung Hsu, De-Yu Chen, Chun-Ying Huang, and Cheng-Hsin Hsu. 2014. On the quality of service of cloud gaming systems. IEEE Trans. Multim. 16, 2 (2014), 480--495.

Digital Library

[9]

Kuan-Ta Chen, Yu-Chun Chang, Po-Han Tseng, Chun-Ying Huang, and Chin-Laung Lei. 2011. Measuring the latency of cloud gaming systems. In Proceedings of the 19th ACM International Conference on Multimedia (MM’11). ACM, New York, NY, 1269--1272.

Digital Library

[10]

Zhenzhong Chen and Christine Guillemot. 2010. Perceptually-friendly H.264/AVC video coding based on foveated just-noticeable-distortion model. IEEE Trans. Circ. Syst. Vid. Technol. 20, 6 (June 2010), 806--819.

[11]

Sharon Choy, Bernard Wong, Gwendal Simon, and Catherine Rosenberg. 2012. The brewing storm in cloud gaming: A measurement study on cloud to end-user latency. In Proceedings of the 11th Workshop on Network and Systems Support for Games (NetGames’12). IEEE Press, Piscataway, NJ, Article 2, 6 pages. Retrieved from http://dl.acm.org/citation.cfm?id=2501560.2501563.

[12]

Mark Claypool and Kajal Claypool. 2006. Latency and player actions in online games. Commun. ACM 49, 11 (Nov. 2006), 40--45.

Digital Library

[13]

Jonathan Deber, Ricardo Jota, Clifton Forlines, and Daniel Wigdor. 2015. How much faster is fast enough? User perception of latency 8 latency improvements in direct and indirect touch. In Proceedings of the 33rd ACM Conference on Human Factors in Computing Systems (CHI’15), 1, 1 (2015), 1827--1836.

[14]

Agostino Gibaldi, Mauricio Vanegas, Peter J. Bex, and Guido Maiello. 2017. Evaluation of the Tobii EyeX eye tracking controller and Matlab toolkit for research. Behav. Res. Meth. 49, 3 (01 June 2017), 923--946.

[15]

Chun-Ying Huang, Cheng-Hsin Hsu, Yu-Chun Chang, and Kuan-Ta Chen. 2013. GamingAnywhere: An open cloud gaming system. In Proceedings of the 4th ACM Multimedia Systems Conference (MMSys’13). ACM, New York, NY, 36--47.

Digital Library

[16]

Chun-Ying Huang, Cheng-Hsin Hsu, Yu-Chun Chang, and Kuan-Ta Chen. 2013. GamingAnywhere: An open cloud gaming system. In Proceedings of the 4th ACM Multimedia Systems Conference (MMSys’13). ACM, New York, NY, 36--47.

Digital Library

[17]

Wijnand Ijsselsteijn, Yvonne de Kort, Karolien Poels, Audrius Jurgelionis, and Francesco Bellotti. 2007. Characterising and measuring user experiences in digital games. In Proceedings of the International Conference on Advances in Computer Entertainment.

[18]

Gazi Illahi, Matti Siekkinen, and Enrico Masala. 2017. Foveated video streaming for cloud gaming. In Proceedings of the IEEE 19th International Workshop on Multimedia Signal Processing (MMSP’17). 1--6.

[19]

Laurent Itti. 2004. Automatic foveation for video compression using a neurobiological model of visual attention. IEEE Trans. Image Proc. 13, 10 (Oct. 2004), 1304--1318.

Digital Library

[20]

JVT. 2003. Joint video team (JVT) of ISO/IEC MPEG and ITU-T VCEG. Draft ITU-T recommendation and final draft international standard of joint video specification (ITU-T Rec. H264 ISO/IEC 14496--10 AVC), document JVT-G050d35, 7th Meeting.

[21]

M. Jarschel, D. Schlosser, S. Scheuring, and T. Hoßfeld. 2011. An evaluation of QoE in cloud gaming based on subjective tests. In Proceedings of the 5th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing. 330--335.

[22]

Teemu Kämäräinen, Matti Siekkinen, Antti Ylä-Jääski, Wenxiao Zhang, and Pan Hui. 2017. A measurement study on achieving imperceptible latency in mobile cloud gaming. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys’17). ACM, New York, NY, 88--99.

Digital Library

[23]

Zhicheng Li, Shiyin Qin, and Laurent Itti. 2011. Visual attention guided bit allocation in video compression. Image Vis. Comput. 29, 1 (2011), 1--14.

Digital Library

[24]

Pietro Lungaro, Rickard Sjöberg, Alfredo J. F. Valero, Ashutosh Mittal, and Konrad Tollmar. 2018. Gaze-aware streaming solutions for the next generation of mobile VR experiences. IEEE Trans. Vis. Comput. Graph. 24, 4 (Apr. 2018), 1535--1544.

Digital Library

[25]

Pietro Lungaro and Konrad Tollmar. 2017. QoE design tradeoffs for foveated content provision. In Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX’17).

[26]

Twan Maintz. 2005. Digital and medical image processing. Universiteit Utrecht (2005).

[27]

Loren Merritt and Rahul Vanam. 2006. x264: A high performance H. 264/AVC encoder. Retrieved from http://neuron2.net/library/avc/overview_x264_v8_5.pdf.

[28]

Iman S. Mohammadi, Mahmoud-Reza Hashemi, and Mohammad Ghanbari. 2015. An object-based framework for cloud gaming using player’s visual attention. In Proceedings of the IEEE International Conference on Multimedia Expo Workshops (ICMEW’15). 1--6.

[29]

Sebastian Möller, Dennis Pommer, Justus Beyer, and Jannis Rake-Revelant. 2013. Factors influencing gaming QoE: Lessons learned from the evaluation of cloud gaming services. In Proceedings of the 4th International Workshop on Perceptual Quality of Systems (PQS’13). 1--5.

[30]

Jim Mullin, Lucy Smallwood, Anna Watson, and Gillian Wilson. 2001. New techniques for assessing audio and video quality in real-time interactive communications. IHM-HCI Tutorial 2001, Lille, France. 1--63.

[31]

Shruti Patil, Yu Chen, and Tajana Simunic Rosing. 2015. GazeTube: Gaze-Based adaptive video playback for bandwidth and power optimizations. In Proceedings of the IEEE Global Communications Conference (GLOBECOM’15). IEEE, 1--6.

[32]

Romass Pauliks, Konstantins Tretjaks, Kirils Belahs, and Romass Pauliks. 2013. A survey on some measurement methods for subjective video quality assessment. In Proceedings of the World Congress on Computer and Information Technology (WCCIT’13).

[33]

Feng Qian, Lusheng Ji, Bo Han, and Vijay Gopalakrishnan. 2016. Optimizing 360 video delivery over cellular networks. In Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges. ACM, 1--6.

Digital Library

[34]

Yashas Rai, Marcus Barkowsky, and Patrick Le Callet. 2016. Role of spatio-temporal distortions in the visual periphery in disrupting natural attention deployment. In Proceedings of the Conference on Human Vision and Electronic Imaging (HVEI’16). 1--6.

[35]

Yashas Rai and Patrick Le Callet. 2017. Do gaze disruptions indicate the perceived quality of non-uniformly coded natural scenes? Electron. Imag. 14 (2017), 104--109.

[36]

Michele Rucci, Paul V. McGraw, and Richard J. Krauzlis. 2016. Fixational eye movements and perception. Vis. Res. 118 (2016), 1--4.

[37]

Jihoon Ryoo, Kiwon Yun, Dimitris Samaras, Samir R. Das, and Gregory Zelinsky. 2016. Design and evaluation of a foveated video streaming service for commodity client devices. In Proceedings of the 7th International Conference on Multimedia Systems (MMSys’16). ACM, New York, NY, Article 6, 11 pages.

Digital Library

[38]

Natela Shanidze, Saeideh Ghahghaei, and Preeti Verghese. 2016. Accuracy of eye position for saccades and smooth pursuit. J. Vis. 16, 15 (2016), 23--23.

[39]

Ryan Shea, Jiangchuan Liu, Edith C.-H. Ngai, and Yong Cui. 2013. Cloud gaming: Architecture and performance. IEEE Netw. 27, 4 (2013), 16--21.

[40]

Ivan Slivar, Mirko Suznjevic, Lea Skorin-Kapov, and Maja Matijasevic. 2014. Empirical QoE study of in-home streaming of online games. In Proceedings of the 13th Annual Workshop on Network and Systems Support for Games. 1--6.

Digital Library

[41]

Rand S. Swenson. 2006. Review of Clinical and Functional Neuroscience. Retrieved from https://www.dartmouth.edu/rswenson/NeuroSci/index.html.

[42]

Tobii. 2015. Developer’s Guide Tobii EyeX SDK for C/C++. Retrieved from http://developer-files.tobii.com/wp-content/uploads/2016/03/Developers-Guide-C-Cpp.pdf.

[43]

Niraj Tolia, Mahadev Satyanarayanan, and David G. Andersen. 2006. Quantifying interactive user experience on thin clients. Computer 39, 3 (2006), 46--52.

Digital Library

[44]

Brian A. Wandell. 1995. Foundations of Vision. Vol. 8. Sinauer Associates, Sunderland, MA.

[45]

Zhou Wang and Alan C. Bovik. 2006. Foveated image and video coding. In Digital Video, Image Quality, and Perceptual Coding, Hong Ren Wu and Kamisetty Ramamohan Rao (Eds.). CRC Press, Boca Raton, FL, 431--457.

[46]

Zhou Wang, Ligang Lu, and Alan. C. Bovik. 2003. Foveation scalable video coding with automatic fixation selection. IEEE Trans. Image Proc. 12, 2 (Feb. 2003), 243--254.

[47]

Alireza Zare, Alireza Aminlou, Miska M. Hannuksela, and Moncef Gabbouj. 2016. HEVC-compliant tile-based streaming of panoramic video for virtual reality applications. In Proceedings of the ACM on Multimedia Conference (MM’16). ACM, New York, NY, 601--605.

Digital Library

Cited By

Wang YYang FMurn LSock JGorriz Blanch MWan SZhang WYang FHerranz L(2025)Enhanced neural video compression for cloud gaming videos with aligned frame generationExpert Systems with Applications10.1016/j.eswa.2024.125535261(125535)Online publication date: Feb-2025
https://doi.org/10.1016/j.eswa.2024.125535
Jumani AShi JLaghari AEstrela VSampedro GAlmadhor AKryvinska Nul Nabi A(2024)Quality of Experience That Matters in Gaming Graphics: How to Blend Image Processing and Virtual RealityElectronics10.3390/electronics1315299813:15(2998)Online publication date: 30-Jul-2024
https://doi.org/10.3390/electronics13152998
Arellano-Uson JMagaña EMorato DIzal M(2024)Survey on Quality of Experience Evaluation for Cloud-Based Interactive ApplicationsApplied Sciences10.3390/app1405198714:5(1987)Online publication date: 28-Feb-2024
https://doi.org/10.3390/app14051987
Show More Cited By

Index Terms

Cloud Gaming with Foveated Video Encoding

Recommendations

Content-aware video encoding for cloud gaming
MMSys '19: Proceedings of the 10th ACM Multimedia Systems Conference

Cloud gaming allows users with thin-clients to play complex games on their end devices as the bulk of processing is offloaded to remote servers. A thin-client is only required to have basic decoding capabilities which exist on most modern devices. The ...
Game Categorization for Deriving QoE-Driven Video Encoding Configuration Strategies for Cloud Gaming
Special Section on Delay-Sensitive Video Computing in the Cloud and Special Section on Extended MMSys-NOSSDAV Best Papers

Cloud gaming has been recognized as a promising shift in the online game industry, with the aim of implementing the “on demand” service concept that has achieved market success in other areas of digital entertainment such as movies and TV shows. The ...
A game attention model for efficient bit rate allocation in cloud gaming

The widespread availability of broadband internet access and the growth in server-based processing have provided an opportunity to run games away from the player into the cloud and offer a new promising service known as cloud gaming. The concept of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 16, Issue 1

February 2020

363 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3384216

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 February 2020

Accepted: 01 October 2019

Revised: 01 June 2019

Received: 01 August 2018

Published in TOMM Volume 16, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Nokia Center for Advanced Research
European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme
Academy of Finland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

49
Total Citations
View Citations
1,285
Total Downloads

Downloads (Last 12 months)116
Downloads (Last 6 weeks)11

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang YYang FMurn LSock JGorriz Blanch MWan SZhang WYang FHerranz L(2025)Enhanced neural video compression for cloud gaming videos with aligned frame generationExpert Systems with Applications10.1016/j.eswa.2024.125535261(125535)Online publication date: Feb-2025
https://doi.org/10.1016/j.eswa.2024.125535
Jumani AShi JLaghari AEstrela VSampedro GAlmadhor AKryvinska Nul Nabi A(2024)Quality of Experience That Matters in Gaming Graphics: How to Blend Image Processing and Virtual RealityElectronics10.3390/electronics1315299813:15(2998)Online publication date: 30-Jul-2024
https://doi.org/10.3390/electronics13152998
Arellano-Uson JMagaña EMorato DIzal M(2024)Survey on Quality of Experience Evaluation for Cloud-Based Interactive ApplicationsApplied Sciences10.3390/app1405198714:5(1987)Online publication date: 28-Feb-2024
https://doi.org/10.3390/app14051987
Ishioka TFukui TFujiwara TNarikawa SFujihashi TSaruwatari SWatanabe T(2024)Traffic Reduction for Speculative Video Transmission in Cloud Gaming SystemsIEICE Transactions on Communications10.23919/transcom.2023EBP3108E107-B:5(408-418)Online publication date: May-2024
https://doi.org/10.23919/transcom.2023EBP3108
Mohanty SBhasi VSon MKandemir MDas C(2024)FAAStloop: Optimizing Loop-Based Applications for Serverless ComputingProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698560(943-960)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698560
Alhilal AWu ZTsui YHui P(2024)FovOptixProceedings of the 15th ACM Multimedia Systems Conference10.1145/3625468.3647612(67-77)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3625468.3647612
Zhang YLiu QWang HChen DHan K(2024)Crowdsourcing Live High Definition Map via Collaborative Computation in Automotive Edge ComputingIEEE Transactions on Vehicular Technology10.1109/TVT.2024.339372873:9(13569-13583)Online publication date: Sep-2024
https://doi.org/10.1109/TVT.2024.3393728
Oliveira SSouza CCarvalho S(2024)A Multiplayer Cloud Gaming Architecture for Scalable Physics2024 IEEE Conference on Games (CoG)10.1109/CoG60054.2024.10645556(1-8)Online publication date: 5-Aug-2024
https://doi.org/10.1109/CoG60054.2024.10645556
Lim JTreder MChadha AAndreopoulos Y(2024)Adaptive Render-Video Streaming for Virtual Environments2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00583(5742-5751)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00583
He ZYang YLi ZChen NZhang ASu Y(2024)CERender: Real-Time Cloud Rendering Based on Cloud-Edge CollaborationComputer Supported Cooperative Work and Social Computing10.1007/978-981-99-9637-7_36(491-501)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-981-99-9637-7_36
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents