research-article

Open access

Theia: Gaze-driven and Perception-aware Volumetric Content Delivery for Mixed Reality Headsets

Authors:

Puqi ZhouAuthors Info & Claims

MOBISYS '24: Proceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services

Pages 70 - 84

https://doi.org/10.1145/3643832.3661858

Published: 04 June 2024 Publication History

Abstract

Minimizing bandwidth consumption while maintaining satisfactory visual quality becomes the holy grail of volumetric content delivery. However, due to the huge amount of 3D data to stream, the stringent latency requirement, and the high computational workload, achieving this ambitious goal could be challenging for mobile mixed reality headsets, which can naturally enable viewers' motion with six degrees of freedom but have limited computing power. Motivated by our critical observations from a user study of eye movements with 50+ participants, in this paper, we present Theia, a first-of-its-kind gaze-driven and perception-aware volumetric content delivery system that effectively incorporates the following innovations into a holistic system: (1) real-time creation of foveated volumetric content to reduce network data usage; (2) efficient augmentation of foveal content to boost user experience; and (3) adaptive omission of peripheral content for further bandwidth savings based on eye movements. We implement a prototype of Theia using Microsoft HoloLens 2 headsets and extensively evaluate its performance. Our results reveal that compared to the state-of-the-art, Theia can drastically reduce bandwidth consumption by up to 67.0% and enhance visual quality by up to 92.5%.

References

[1]

2017. 8i Voxelized Full Bodies (8iVFB v2) - Dynamic Voxelized Point Cloud Dataset. http://plenodb.jpeg.org/pc/8ilabs. [accessed on 11/30/2023].

[2]

2019. Microsoft HoloLens 2. https://www.microsoft.com/en-us/hololens. [accessed on 11/30/2023].

[3]

2023. About HoloLens 2. https://learn.microsoft.com/en-us/hololens/hololens2-hardware. [accessed on 11/30/2023].

[4]

2023. CUDA Toolkit. https://developer.nvidia.com/cuda-toolkit. [accessed on 11/30/2023].

[5]

2023. Draco 3D Data Compression. https://google.github.io/draco/. [accessed on 11/30/2023].

[6]

2023. Extended Eye Tracking in Native Engine. https://learn.microsoft.com/en-us/windows/mixed-reality/develop/native/extended-eye-tracking-native. [accessed on 11/30/2023].

[7]

2023. Eye Tracking on HoloLens 2. https://docs.microsoft.com/en-us/windows/mixed-reality/eye-tracking. [accessed on 11/30/2023].

[8]

2023. Magic Leap 2. https://www.magicleap.com/magic-leap-2. [accessed on 11/30/2023].

[9]

2023. tc(8) - Linux man page. https://linux.die.net/man/8/tc. [accessed on 11/30/2023].

[10]

2023. Unreal Engine. https://www.unrealengine.com. [accessed on 11/30/2023].

[11]

Lukas Ahrenberg, Philip Benzie, Marcus Magnor, and John Watson. 2008. Computer Generated Holograms from Three Dimensional Meshes using an Analytic Light Transport Modell. Applied Optics 47, 10 (2008), 1567--1574.

[12]

Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency Requirements for Foveated Rendering in Virtual Reality. ACM Transactions on Applied Perception 14, 4 (2017), 1--13.

Digital Library

[13]

Elena Arabadzhiyska, Cara Tursun, Hans-Peter Seidel, and Piotr Didyk. 2023. Practical Saccade Prediction for Head-Mounted Displays: Towards a Comprehensive Model. ACM Transactions on Applied Perception 20, 1 (2023), 1--23.

Digital Library

[14]

Elena Arabadzhiyska, Okan Tarhan Tursun, Karol Myszkowski, Hans-Peter Seidel, and Piotr Didyk. 2017. Saccade Landing Position Prediction for Gaze-Contingent Rendering. ACM Transactions on Graphics 36, 4 (2017), 1--12.

Digital Library

[15]

Helder Araujo and Jorge MDias. 1996. An Introduction to the Log-polar Mapping [Image Sampling]. In Proceedings of II Workshop on Cybernetic Vision.

[16]

Paramvir Bahl and Venkata N Padmanabhan. 2000. RADAR: An In-Building RF-based User Location and Tracking System. In Proceedings of IEEE INFOCOM.

[17]

Behnam Bastani, Eric Turner, Carlin Vieri, Haomiao Jiang, Brian Funt, and Nikhil Balram. 2017. Foveated Pipeline for AR/VR Head-Mounted Displays. Information Display 33, 6 (2017), 14--35.

[18]

Marcelo Bertalmio, Andrea L Bertozzi, and Guillermo Sapiro. 2001. Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]

Efe Bozkir, Ali Burak Ünal, Mete Akgün, Enkelejda Kasneci, and Nico Pfeifer. 2020. Privacy Preserving Gaze Estimation Using Synthetic Images via a Randomized Encoding Based Framework. In Proceedings of ACM Symposium on Eye Tracking Research and Applications (ETRA).

Digital Library

[20]

G. Bradski. 2000. The OpenCV Library. Dr. Dobb's Journal of Software Tools (2000).

[21]

David R. Bull and Fan Zhang. 2021. Chapter 4 - Digital Picture Formats and Representations. In Intelligent Image and Video Compression (Second Edition). Academic Press, Oxford, 107--142.

[22]

Rick H-Y Chen and Timothy D Wilkinson. 2009. Computer Generated Hologram from Point Cloud using Graphics Processor. Applied Optics 48, 6 (2009), 6841--6850.

[23]

Ruizhi Cheng, Kaiyan Liu, Nan Wu, and Bo Han. 2023. Enriching Telepresence with Semantic-driven Holographic Communication. In Proceedings of ACM Workshop on Hot Topics in Networks.

Digital Library

[24]

Alexander Clemm, Maria Torres Vega, Hemanth Kumar Ravuri, Tim Wauters, and Filip De Turck. 2020. Toward Truly Immersive Holographic-type Communication: Challenges and Solutions. IEEE Communications Magazine 58, 1 (2020), 93--99.

Digital Library

[25]

Eduardo Cuervo, Alec Wolman, Landon P. Cox, Kiron Lebeck, Ali Razeen, Stefan Saroiu, and Madanlal Musuvathi. 2015. Kahawai: High-Quality Mobile Gaming Using GPU Offload. In Proceedings of ACM MobiSys.

Digital Library

[26]

Brendan David-John, Diane Hosfelt, Kevin Butler, and Eakta Jain. 2021. A Privacy-Preserving Approach to Streaming Eye-Tracking Data. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021), 2555--2565.

[27]

Leila De Floriani, Franco Morando, and Enrico Puppo. 2003. Representation of Non-Manifold Objects. In Proceedings of ACM Symposium on Solid Modeling and Applications.

Digital Library

[28]

Mark R Diamond, John Ross, and Maria C Morrone. 2000. Extraretinal Control of Saccadic Suppression. Journal of Neuroscience 20, 9 (2000), 3449--3455.

[29]

Omar Elharrouss, Noor Almaadeed, Somaya Al-Maadeed, and Younes Akbari. 2020. Image Inpainting: A Review. Neural Processing Letters 51 (2020), 2007--2028.

Digital Library

[30]

Ching-Ling Fan, Jean Lee, Wen-Chih Lo, Chun-Ying Huang, Kuan-Ta Chen, and Cheng-Hsin Hsu. 2017. Fixation Prediction for 360° Video Streaming in Head-Mounted Virtual Reality. In Proceedings of ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV).

Digital Library

[31]

Yu Fang, Ryoichi Nakashima, Kazumichi Matsumiya, Ichiro Kuriki, and Satoshi Shioiri. 2015. Eye-Head Coordination for Visual Cognitive Processing. PLOS ONE 10, 3 (2015), e0121035.

[32]

Clarence Errol Ferree, Gertrude Rand, and C Hardy. 1931. Refraction For The Peripheral Field of Vision. Archives of Ophthalmology 5, 5 (1931), 717--731.

[33]

James D Foley. 1996. Computer Graphics: Principles and Practice. In Intelligent Image and Video Compression (Second Edition). Addison-Wesley Professional.

[34]

Yongjie Guan, Xueyu Hou, Nan Wu, Bo Han, and Tao Han. 2023. MetaStream: Live Volumetric Content Capture, Creation, Delivery, and Rendering in Real Time. In Proceedings of the International Conference on Mobile Computing and Networking.

Digital Library

[35]

Yu Guan, Chengyuan Zheng, Xinggong Zhang, Zongming Guo, and Junchen Jiang. 2019. Pano: Optimizing 360 Video Streaming with a Better Understanding of Quality Perception. In Proceedings of ACM SIGCOMM.

Digital Library

[36]

Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D Graphics. ACM Transactions on Graphics 31, 6 (2012), 1--10.

Digital Library

[37]

Serhan Gül, Dimitri Podborski, Jangwoo Son, Gurdeep Singh Bhullar, Thomas Buchholz, Thomas Schierl, and Cornelius Hellge. 2020. Cloud Rendering-based Volumetric Video Streaming System for Mixed Reality Services. In Proceedings of ACM MMSys.

Digital Library

[38]

Bo Han, Yu Liu, and Feng Qian. 2020. ViVo: Visibility-Aware Mobile Volumetric Video Streaming. In Proceedings of ACM MobiCom.

Digital Library

[39]

John Paulin Hansen, Alexandre Alapetite, Martin Thomsen, Zhongyu Wang, Katsumi Minakata, and Guangtao Zhang. 2018. Head and Gaze Control of a Telepresence Robot with an HMD. In Proceedings of ACM Symposium on Eye Tracking Research & Applications (ETRA).

Digital Library

[40]

Joy Hirsch and Christine A Curcio. 1989. The Spatial Resolution Capacity of Human Foveal Retina. Vision Research 29, 9 (1989), 1095--1101.

[41]

Luke Hsiao, Brooke Krajancich, Philip Levis, Gordon Wetzstein, and Keith Winstein. 2022. Towards Retina-Quality VR Video Streaming: 15ms Could Save You 80% of Your Bandwidth. ACM SIGCOMM Computer Communication Review 52, 1 (2022), 10--19.

Digital Library

[42]

Chih-Fan Hsu, Anthony Chen, Cheng-Hsin Hsu, Chun-Ying Huang, Chin-Laung Lei, and Kuan-Ta Chen. 2017. Is Foveated Rendering Perceivable in Virtual Reality?: Exploring the Efficiency and Consistency of Quality Assessment Methods. In Proceedings of ACM International Conference on Multimedia.

Digital Library

[43]

Miao Hu, Zhenxiao Luo, Yipeng Zhou, Xuezheng Liu, and Di Wu. 2022. Otus: A Gaze Model-based Privacy Control Framework for Eye Tracking Applications. In Proceedings of IEEE INFOCOM.

Digital Library

[44]

Zhiming Hu, Sheng Li, Congyi Zhang, Kangrui Yi, Guoping Wang, and Dinesh Manocha. 2020. DGaze: CNN-Based Gaze Prediction in Dynamic Scenes. IEEE Transactions on Visualization and Computer Graphics 26, 5 (2020), 1902--1911.

[45]

Zhiming Hu, Congyi Zhang, Sheng Li, Guoping Wang, and Dinesh Manocha. 2019. SGaze: A Data-Driven Eye-Head Coordination Model for Realtime Gaze Prediction. IEEE Transactions on Visualization and Computer Graphics 25, 5 (2019), 32002--2010.

[46]

Gazi Karam Illahi, Matti Siekkinen, Teemu Kämäräinen, and Antti Ylä-Jääski. 2022. Real-time Gaze Prediction in Virtual Reality. In Proceedings of ACM International Workshop on Immersive Mixed and Virtual Environment Systems (MMVE).

Digital Library

[47]

Alireza Javaheri, Catarina Brites, Fernando Pereira, and Joao Ascenso. 2020. Point Cloud Rendering After Coding: Impacts on Subjective and Objective Quality. IEEE Transactions on Multimedia 23 (2020), 4049--4064.

[48]

Junchen Jiang, Vyas Sekar, and Hui Zhang. 2012. Improving Fairness, Efficiency, and Stability in HTTP-based Adaptive Video Streaming with FESTIVE. In Proceedings of ACM CoNEXT.

Digital Library

[49]

Donald H Kelly. 1979. Motion and vision. II. Stabilized spatio-temporal threshold surface. Journal of the Optical Society of America 69, 10 (1979), 1340--1349.

[50]

Conor Kelton, Jihoon Ryoo, Aruna Balasubramanian, and Samir Das. 2017. Improving User Perceived Page Load Time Using Gaze. In Proceedings of USENIX NSDI.

Digital Library

[51]

Sehwan Ki, Sung-Ho Bae, Munchurl Kim, and Hyunsuk Ko. 2018. Learning-Based Just-Noticeable-Quantization- Distortion Modeling for Perceptual Video Coding. IEEE Transactions on Image Processing 27, 7 (2018), 3178--3193.

[52]

Joohwan Kim, Michael Stengel, Alexander Majercik, Shalini De Mello, David Dunn, Samuli Laine, Morgan McGuire, and David Luebke. 2019. NVGaze: An Anatomically-Informed Dataset for Low-Latency, Near-Eye Gaze Estimation. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI).

Digital Library

[53]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. https://arxiv.org/abs/1412.6980. [accessed on 11/30/2023].

[54]

Earl J Kirkland and Earl J Kirkland. 2010. Bilinear Interpolation. Advanced Computing in Electron Microscopy (2010), 261--263.

[55]

Petrus EJ Kivi, Markku J Mäkitalo, Jakub Žádník, Julius Ikkala, Vinod Kumar Malamal Vadakital, and Pekka O Jääskeläinen. 2022. Real-Time Rendering of Point Clouds With Photorealistic Effects: A Survey. IEEE Access 10 (2022), 13151--13173.

[56]

Kyungjin Lee, Juheon Yi, Youngki Lee, Sunghyun Choi, and Young Min Kim. 2020. GROOT: a Real-time Streaming System of High-fidelity Volumetric Videos. In Proceedings of ACM MobiCom.

Digital Library

[57]

Marc Levoy and Ross Whitaker. 1990. Gaze-directed Volume Rendering. In Proceedings of Symposium on Interactive 3D Graphics (I3D).

Digital Library

[58]

David Li, Ruofei Du, Adharsh Babu, Camelia D Brumar, and Amitabh Varshney. 2021. A Log-Rectilinear Transformation for Foveated 360-degree Video Streaming. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021), 2638--2647.

[59]

Jingjie Li, Amrita Roy Chowdhury, Kassem Fawaz, and Younghyun Kim. 2021. Kalϵido: Real-time Privacy Control for Eye-tracking Systems. In Proceedings of USENIX Security Symposium. https://www.usenix.org/conference/usenixsecurity21/presentation/li-jingjie

[60]

Ruihui Li, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. 2019. PU-GAN: A Point Cloud Upsampling Adversarial Network. In Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV).

[61]

Tianxing Li, Qiang Liu, and Xia Zhou. 2017. Ultra-Low Power Gaze Tracking for Virtual Reality. In Proceedings of ACM SenSys.

Digital Library

[62]

Tianxing Li and Xia Zhou. 2018. Battery-Free Eye Tracker on Glasses. In Proceedings of ACM MobiCom.

Digital Library

[63]

Zhicheng Li, Shiyin Qin, and Laurent Itti. 2011. Visual Attention Guided Bit Allocation in Video Compression. Image and Vision Computing 29, 1 (2011), 1--14.

Digital Library

[64]

Daniel J Liebling and Sören Preibusch. 2014. Privacy Considerations for a Pervasive Eye Tracking World. In Proceedings of ACM UbiComp.

Digital Library

[65]

Weisi Lin and C.-C. Jay Kuo. 2011. Perceptual visual quality metrics: A survey. Journal of Visual Communication and Image Representation 22, 4 (2011), 297--312.

Digital Library

[66]

Junhua Liu, Boxiang Zhu, Fangxin Wang, Yili Jin, Wenyi Zhang, Zihan Xu, and Shuguang Cui. 2023. CaV3: Cache-assisted Viewport Adaptive Volumetric Video Streaming. In Proceedings of IEEE Conference Virtual Reality and 3D User Interfaces (VR).

[67]

Yu Liu, Bo Han, Feng Qian, Arvind Narayanan, and Zhi-Li Zhang. 2022. Vues: Practical Mobile Volumetric Video Streaming Through Multiview Transcoding. In Proceedings of ACM MobiCom.

Digital Library

[68]

Lester C Loschky and Gary S Wolverton. 2007. How Late Can You Update Gaze-contingent Multiresolutional Displays Without Detection? ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 3, 4 (2007), 1--10.

Digital Library

[69]

Pietro Lungaro, Rickard Sjöberg, Alfredo José Fanghella Valero, Ashutosh Mittal, and Konrad Tollmar. 2018. Gaze-Aware Streaming Solutions for the Next Generation of Mobile VR Experiences. IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1535--1544.

Digital Library

[70]

Adrien Maglo, Guillaume Lavoué, Florent Dupont, and Céline Hudelot. 2015. 3D Mesh Compression: Survey, Comparisons, and Emerging Trends. ACM Computing Surveys 47, 3 (2015).

Digital Library

[71]

Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. 2017. Neural Adaptive Video Streaming with Pensieve. In Proceedings of ACM SIGCOMM.

Digital Library

[72]

Rachel McAfee, Cole Haxton, Matthew Harrison, and Joshua Gess. 2020. Thermal Characterization of a Virtual Reality Headset during Transient and Resting Operation. In Proceedings of Semiconductor Thermal Measurement, Modeling & Management Symposium (SEMI-THERM).

[73]

Fang Meng and Hongbin Zha. 2003. Efficient Streaming of Point-based Models for Interactive 3-D Geometry Transmission. In Proceedings of IEEE International Symposium on Computational Intelligence in Robotics and Automation.

[74]

Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney. 2018. Kernel Foveated Rendering. In Proceedings of ACM on Computer Graphics and Interactive Techniques.

Digital Library

[75]

Zili Meng, Tingfeng Wang, Yixin Shen, Bo Wang, Mingwei Xu, Rui Han, Honghao Liu, Venkat Arun, Hongxin Hu, and Xue Wei. 2023. Enabling High Quality Real-Time Communications with Adaptive Frame-Rate. In Proceedings of USENIX NSDI. https://www.usenix.org/conference/nsdi23/presentation/meng

[76]

Aythami Morales, Francisco M Costela, and Russell L Woods. 2021. Saccade Landing Point Prediction Based on Fine-Grained Learning Method. IEEE Access 9 (2021), 52474--52484.

[77]

Yana Nehmé, Johanna Delanoy, Florent Dupont, Jean-Philippe Farrugia, Patrick Le Callet, and Guillaume Lavoué. 2023. Textured Mesh Quality Assessment: Large-Scale Dataset and Deep Learning-based Quality Metric. ACM Transactions on Graphics (2023).

Digital Library

[78]

Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, et al. 2016. Holoportation: Virtual 3D Teleportation in Real-time. In Proceedings of ACM Symposium on User Interface Software and Technology (UIST).

Digital Library

[79]

Rafael Pagés, Konstantinos Amplianitis, Jan Ondrej, Emin Zerman, and Aljosa Smolic. 2022. Volograms & V-SENSE Volumetric Video Dataset. (2022).

[80]

Rafael Pagés, Emin Zerman, Konstantinos Amplianitis, Jan Ondřej, and Aljosa Smolic. 2021. Volograms & V-SENSE Volumetric Video Dataset. ISO/IEC JTC1/SC29/WG07 MPEG2021/m56767 (2021).

[81]

Jounsup Park, Philip A. Chou, and Jenq-Neng Hwang. 2019. Rate-Utility Optimized Streaming of Volumetric Media for Augmented Reality. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 1 (2019), 149--162.

[82]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proceedings of NeurIPS.

Digital Library

[83]

Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards Foveated Rendering for Gaze-tracked Virtual Reality. ACM Transactions on Graphics 35, 6 (2016), 179:1--179:12.

Digital Library

[84]

Snježana Rimac-Drlje, Goran Martinović, and Branka Zovko-Cihlar. 2011. Foveation-based Content Adaptive Structural Similarity Index. In Proceedings of International Conference on Systems, Signals and Image Processing.

[85]

Jihoon Ryoo, Kiwon Yun, Dimitris Samaras, Samir R Das, and Gregory Zelinsky. 2016. Design and evaluation of a foveated video streaming service for commodity client devices. In Proceedings of ACM MMSys.

Digital Library

[86]

William Sentosa, Balakrishnan Chandrasekaran, P. Brighten Godfrey, Haitham Hassanieh, and Bruce Maggs. 2023. DChannel: Accelerating Mobile Applications With Parallel High-bandwidth and Low-latency Channels. In Proceedings of USENIX NSDI. https://www.usenix.org/conference/nsdi23/presentation/sentosa

[87]

Hans Strasburger, Ingo Rentschler, and Martin Jüttner. 2011. Peripheral Vision and Pattern Recognition: A Review. Journal of Vision 11, 5 (2011), 13--13.

[88]

Emilio Calvanese Strinati, Sergio Barbarossa, Jose Luis Gonzalez-Jimenez, Dimitri Ktenas, Nicolas Cassiau, Luc Maret, and Cedric Dehos. 2019. 6G: The Next Frontier: From Holographic Messaging to Artificial Intelligence Using Subterahertz and Visible Light Communication. IEEE Vehicular Technology Magazine 14, 3 (2019), 42--50.

[89]

Gail M Sullivan and Anthony R Artino Jr. 2013. Analyzing and Interpreting Data From Likert-Type Scales. Journal of Graduate Medical Education 5, 4 (2013), 541--542.

[90]

Faisal Tariq, Muhammad RA Khandaker, Kai-Kit Wong, Muhammad A Imran, Mehdi Bennis, and Merouane Debbah. 2020. A Speculative Study on 6G. IEEE Wireless Communications 27, 4 (2020), 118--125.

[91]

Flora Ponjou Tasse, Jiri Kosinka, and Neil Dodgson. 2015. Cluster-Based Point Set Saliency. In Proceedings of IEEE International Conference on Computer Vision (ICCV).

Digital Library

[92]

Y Tefera, Dario Mazzanti, Sara Anastasi, Darwin Caldwell, Paolo Fiorini, and Nikhil Deshpande. 2022. Towards Foveated Rendering For Immersive Remote Telerobotics. In Proceedings of the International Workshop on Virtual, Augmented, and Mixed-Reality for Human-Robot Interactions at HRI. https://hal.science/hal-03610377/file/foveated.pdf

[93]

Graham Upton and Ian Cook. 2014. A Dictionary of Statistics. Oxford University Press, USA.

[94]

Christian J Van den Branden Lambrecht and Olivier Verscheure. 1996. Perceptual Quality Measure Using a Spatiotemporal Model of the Human Visual System. In Proceedings of Digital Video Compression: Algorithms and Technologies.

[95]

Frances C Volkmann, Lorrin A Riggs, Keith D White, and Robert K Moore. 1978. Contrast Sensitivity during Saccadic Eye Movements. Vision Research 18, 9 (1978), 1193--1199.

[96]

Dongzhu Xu, Anfu Zhou, Xinyu Zhang, Guixian Wang, Xi Liu, Congkai An, Yiming Shi, Liang Liu, and Huadong Ma. 2020. Understanding Operational 5G: A First Measurement Study on Its Coverage, Performance and Energy Consumption. In Proceedings of ACM SIGCOMM. 479--494.

Digital Library

[97]

Yanyu Xu, Yanbing Dong, Junru Wu, Zhengzhong Sun, Zhiru Shi, Jingyi Yu, and Shenghua Gao. 2018. Gaze Prediction in Dynamic 360° Immersive Videos. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[98]

Jingteng Xue and Chang Wen Chen. 2014. Mobile Video Perception: New Insights and Adaptation Strategies. IEEE Journal of Selected Topics in Signal Processing 8, 3 (2014), 390--401.

[99]

Hyunho Yeo, Chan Ju Chong, Youngmok Jung, Juncheol Ye, and Dongsu Han. 2020. NEMO: Enabling Neural-enhanced Video Streaming on Commodity Mobile Devices. In Proceedings of ACM MobiCom.

Digital Library

[100]

Anlan Zhang, Chendong Wang, Bo Han, and Feng Qian. 2022. YuZu: Neural-enhanced Volumetric Video Streaming. In Proceedings of USENIX NSDI. https://www.usenix.org/conference/nsdi22/presentation/zhang-anlan

[101]

Ding Zhang, Bo Han, Parth Pathak, and Haoliang Wang. 2021. Innovating Multiuser Volumetric Video Streaming through Cross-layer Design. In Proceedings of ACM HotNets.

Digital Library

[102]

Ding Zhang, Puqi Zhou, Bo Han, and Parth Pathak. 2022. M5: Facilitating MultiUser Volumetric Content Delivery with Multi-Lobe Multicast over mmWave. In Proceedings of ACM SenSys.

Digital Library

[103]

Shulin Zhao, Haibo Zhang, Cyan Subhra Mishra, Sandeepa Bhuyan, Ziyu Ying, Mahmut Taylan Kandemir, Anand Sivasubramaniam, and Chita Das. 2021. HoloAR: On-the-Fly Optimization of 3D Holographic Processing for Augmented Reality. In Proceedings of Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

Digital Library

[104]

Tianhang Zheng, Changyou Chen, Junsong Yuan, Bo Li, and Kui Ren. 2019. PointCloud Saliency Maps. In Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV). 1598--1606.

Cited By

Cheng RWu NLe VChai EVarvello MHan BShu YLiu JTan RHe YChen J(2024)MagicStream: Bandwidth-conserving Immersive Telepresence via Semantic CommunicationProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699344(365-379)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699344
Cheng RWu NVarvello MChai EChen SHan BVallina-Rodríguez NSuarez-Tángil GLevin DPelsser C(2024)A First Look at Immersive Telepresence on Apple Vision ProProceedings of the 2024 ACM on Internet Measurement Conference10.1145/3646547.3689006(555-562)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3646547.3689006

Index Terms

Theia: Gaze-driven and Perception-aware Volumetric Content Delivery for Mixed Reality Headsets
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Mixed / augmented reality
2. Information systems
  1. Information systems applications
    1. Multimedia information systems
      1. Multimedia streaming

Recommendations

MuV2: Scaling up Multi-user Mobile Volumetric Video Streaming via Content Hybridization and Sharing
ACM MobiCom '24: Proceedings of the 30th Annual International Conference on Mobile Computing and Networking

Volumetric videos offer a unique interactive experience and have the potential to enhance social virtual reality and telepresence. Streaming volumetric videos to multiple users remains a challenge due to its tremendous requirements of network and ...
MetaStream: Live Volumetric Content Capture, Creation, Delivery, and Rendering in Real Time
ACM MobiCom '23: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking

While recent work explored streaming volumetric content on-demand, there is little effort on live volumetric video streaming that bears the potential of bringing more exciting applications than its on-demand counterpart. To fill this critical gap, in ...
Discomfort of Visual Perception in Virtual and Mixed Reality Systems
Abstract
The importance of the topic of this paper is caused by the rapid development of virtual, mixed, and augmented reality systems. A problem in designing virtual reality (VR), augmented reality (AR) and mixed reality (MR) systems is the assessment of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MOBISYS '24: Proceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services

June 2024

778 pages

ISBN:9798400705816

DOI:10.1145/3643832

Chairs:
Tadashi Okoshi,
JeongGil Ko,
Program Chair:
Robert LiKamWa

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

In-Cooperation

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 June 2024

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Conference

MOBISYS '24

Sponsor:

SIGMOBILE

MOBISYS '24: 22nd Annual International Conference on Mobile Systems, Applications and Services

June 3 - 7, 2024

Minato-ku, Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 274 of 1,679 submissions, 16%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
803
Total Downloads

Downloads (Last 12 months)803
Downloads (Last 6 weeks)174

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cheng RWu NLe VChai EVarvello MHan BShu YLiu JTan RHe YChen J(2024)MagicStream: Bandwidth-conserving Immersive Telepresence via Semantic CommunicationProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699344(365-379)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699344
Cheng RWu NVarvello MChai EChen SHan BVallina-Rodríguez NSuarez-Tángil GLevin DPelsser C(2024)A First Look at Immersive Telepresence on Apple Vision ProProceedings of the 2024 ACM on Internet Measurement Conference10.1145/3646547.3689006(555-562)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3646547.3689006

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents