research-article

On Objective and Subjective Quality of 6DoF Synthesized Live Immersive Videos

Authors:

Sheng-Ming Tang,

Ching-Ting Wang,

Cheng-Hsin HsuAuthors Info & Claims

QoEVMA '22: Proceedings of the 2nd Workshop on Quality of Experience in Visual Multimedia Applications

Pages 49 - 56

https://doi.org/10.1145/3552469.3555709

Published: 10 October 2022 Publication History

Abstract

We address the problem of quantifying the perceived quality in 6DoF (Degree-of-Freedom) live immersive video in two steps. First, we develop a set of tools to generate (or collect) datasets in a photorealistic simulator, AirSim. Using these tools, we get to change diverse settings of live immersive videos, such as scenes, trajectories, camera placements, and encoding parameters. Second, we develop objective and subjective evaluation procedures, and carry out evaluations on a sample immersive video codec, MPEG MIV, using our own dataset. Several insights were found through our experiments: (1) the two synthesizers in TMIV produce comparable target view quality, but RVS runs 2 times faster; (2) Quantization Parameter (QP) is a good control knob to exercise target view quality and bitrate, but camera placements (or trajectories) also impose significant impacts; and (3) overall subjective quality has strong linear/rank correlation with subjective similarity, sharpness, and color. These findings shed some light on the future research problems for the development of emerging applications relying on immersive interactions.

Supplementary Material

MP4 File (QoEVMA22-13.mp4)

Presentation video - short version

Download
657.76 MB

References

[1]

Federica Battisti, Emilie Bosc, Marco Carli, Patrick Le Callet, and Simone Perugia. 2015. Objective image quality assessment of 3D synthesized views. Signal Processing: Image Communication 30 (2015), 78--88. https://doi.org/10.1016/j. image.2014.10.005

[2]

Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. In Noise reduction in speech processing. Springer, 1--4.

Digital Library

[3]

Gisle Bjontegaard. 2001. Calculation of average PSNR differences between RDcurves. VCEG-M33 (2001).

[4]

J. Boyce, R. Dore, A. Dziembowski, J. Fleureau, J. Jung, B. Kroon, B. Salahieh, V. Vadakital, and L. Yu. 2021. MPEG Immersive Video Coding Standard. Proc. IEEE (2021).

[5]

Yangang Cai, Ronggang Wang, Ke Qiu, Rui Peng, Zhipeng Cheng, and Qi Wang. 2021. Depth Map Video Compression Performance Evaluation For Ieee 1857.9. In Proc. of IEEE International Conference on Multimedia Expo Workshops (ICMEW'21). Shenzhen, China, 1--6. https://doi.org/10.1109/ICMEW53276.2021.9455977

[6]

Epic Games. 2019. Unreal Engine. Retrieved July 7, 2021 from https://www. unrealengine.com

[7]

Epic Games. 2021. Unreal Engine marketplace. Retrieved Sep 3, 2021 from https://www.unrealengine.com/marketplace/en-US/store

[8]

Sarah Fachada, Daniele Bonatto, Arnaud Schenkel, and Gauthier Lafruit. 2018. Free Navigation in Natural Scenery With DIBR: RVS and VSRS in MPEG-I Standardization. In Proc. of IEEE International Conference on 3D Immersion (IC3D). 1--6. https://doi.org/10.1109/IC3D.2018.8657912

[9]

FFmpeg. 2021. FFmpeg. Retrieved Sep 3, 2021 from https://www.ffmpeg.org/

[10]

Julien Fleureau, Bertrand Chupeau, Franck Thudor, Gerard Briand, Thierry Tapie, and Renaud Dore. 2020. An Immersive Video Experience with Real-Time View Synthesis Leveraging the Upcoming MIV Distribution Standard. In Proc. of IEEE International Conference on Multimedia Expo Workshops (ICMEW). 1--2. https: //doi.org/10.1109/ICMEW46912.2020.9105948

[11]

Fortune Business Insights. 2022. Virtual Reality Market Size, Share & COVID-19 Impact Analysis, By Component (Hardware, Software, Content), By Device Type (Head Mounted Display, VR Simulator, VR Glasses, Treadmills & Haptic Gloves, Others), By Industry (Gaming, Entertainment, Automotive, Retail, Healthcare, Education, Aerospace & Defense, Manufacturing, Others), and Regional Forecast, 2022--2029. Retrieved May. 29, 2022 from https://www.fortunebusinessinsights.com/industryreports/virtual-reality-market-101378

[12]

J. Hooft, M. Vega, T. Wauters, C. Timmerer, A. Begen, F. Turck, and R. Schatz. 2020. From Capturing to Rendering: Volumetric Media Delivery with Six Degrees of Freedom. IEEE Communications Magazine 58, 10 (2020), 49--55. https://doi. org/10.1109/MCOM.001.2000242

[13]

Chih-Fan Hsu, Tse-Hou Hung, and Cheng-Hsin Hsu. 2022. Optimizing Immersive Video Coding Configurations Using Deep Learning: A Case Study on TMIV. ACM Trans. Multimedia Comput. Commun. Appl. 18, 1, Article 19 (jan 2022), 25 pages. https://doi.org/10.1145/3471191

Digital Library

[14]

J. Jeong, S. Lee, I. Ryu, T. Le, and E. Ryu. 2020. Towards Viewport-Dependent 6DoF 360 Video Tiled Streaming for Virtual Reality Systems. In Proc. of the ACM International Conference on Multimedia (MM '20). New York, NY, USA, 3687--3695. https://doi.org/10.1145/3394171.3413712

Digital Library

[15]

Jong-Beom Jeong, Soonbin Lee, and Eun-Seok Ryu. 2021. DWS-BEAM: DecoderWise Subpicture Bitstream Extracting and Merging for MPEG Immersive Video. In 2021 International Conference on Visual Communications and Image Processing (VCIP). 1--5. https://doi.org/10.1109/VCIP53242.2021.9675419

[16]

Jong-Beom Jeong, Soonbin Lee, and Eun-Seok Ryu. 2021. Sub-bitstream packing based lightweight tiled streaming for 6 degree of freedom immersive video. Electronics Letters 57, 25 (2021), 973--976. https://doi.org/10.1049/ell2.12329 arXiv:https://ietresearch.onlinelibrary.wiley.com/doi/pdf/10.1049/ell2.12329

[17]

Jong-Beom Jeong, Soonbin Lee, and Eun-Seok Ryu. 2022. Rethinking FatigueAware 6DoF Video Streaming: Focusing on MPEG Immersive Video. In 2022 International Conference on Information Networking (ICOIN). 304--309. https: //doi.org/10.1109/ICOIN53446.2022.9687247

[18]

Chongchong Jin, Zongju Peng, Fen Chen, and Gangyi Jiang. 2022. Subjective and Objective Video Quality Assessment for Windowed-6DoF Synthesized Videos. IEEE Transactions on Broadcasting (2022), 1--15. https://doi.org/10.1109/TBC. 2022.3165473

[19]

Joël Jung and Patrick Boissonade. 2020. VVS: Versatile View Synthesizer for 6-DoF Immersive Video. working paper or preprint.

[20]

J Jung, B Kroon, and J Boyce. 2020. Common Test Conditions for MPEG Immersive Video. ISO/IEC JTC 1/SC 29/WG 11 N19484 (2020).

[21]

Sangwoon Kwak, Joungil Yun, Jun-Young Jeong, Youngwook Kim, Insung Ihm, Won-Sik Cheong, and Jeongil Seo. 2022. View synthesis with sparse light field for 6DoF immersive video. ETRI Journal 44, 1 (2022), 24--37. https://doi.org/10.4218/ etrij.2021-0205 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.4218/etrij.2021- 0205

[22]

Heesub Lee, Alan H. Rowberg, Mark S. Frank M.D., Hyung-Sik Choi M.D., and Yongmin Kim. 1992. Subjective evaluation of compressed image quality. In Medical Imaging VI: Image Capture, Formatting, and Display, Yongmin Kim (Ed.), Vol. 1653. International Society for Optics and Photonics, SPIE, 241 -- 251. https: //doi.org/10.1117/12.59503

[23]

Soonbin Lee, Jong-Beom Jeong, and Eun-Seok Ryu. 2022. Efficient Group-Based Packing Strategy for 6DoF Immersive Video Streaming. In 2022 International Conference on Information Networking (ICOIN). 310--314. https://doi.org/10.1109/ ICOIN53446.2022.9687139

[24]

Leida Li, Yipo Huang, Jinjian Wu, Ke Gu, and Yuming Fang. 2021. Predicting the Quality of View Synthesis With Color-Depth Image Fusion. IEEE Transactions on Circuits and Systems for Video Technology 31, 7 (2021), 2509--2521. https: //doi.org/10.1109/TCSVT.2020.3024882

[25]

Leida Li, Yu Zhou, Ke Gu, Weisi Lin, and Shiqi Wang. 2018. Quality Assessment of DIBR-Synthesized Images by Measuring Local Geometric Distortions and Global Sharpness. IEEE Transactions on Multimedia 20, 4 (2018), 914--926. https: //doi.org/10.1109/TMM.2017.2760062

Digital Library

[26]

Xiangkai Liu, Yun Zhang, Sudeng Hu, Sam Kwong, C-C Jay Kuo, and Qiang Peng. 2015. Subjective and objective video quality assessment of 3D synthesized views with texture/depth compression distortion. IEEE Transactions on Image Processing 24, 12 (2015), 4847--4861.

Digital Library

[27]

Dawid Mieloch, Patrick Garus, Marta Milovanovi?, Joël Jung, Jun Young Jeong, Smitha Lingadahalli Ravi, and Basel Salahieh. 2022. Overview and Efficiency of Decoder-Side Depth Estimation in MPEG Immersive Video. IEEE Transactions on Circuits and Systems for Video Technology (2022).

[28]

Sebastian Möller, Marcel Wältermann, and Marie-Neige Garcia. 2014. Features of Quality of Experience. Springer International Publishing, Cham, 73--84. https: //doi.org/10.1007/978--3--319-02681--7_5

[29]

MPEG. 2019. The gitlab of MPEG test model for immersive video. Retrieved Sep 3, 2021 from https://gitlab.com/mpeg-i-visual/tmiv/-/tree/v10.0.1

[30]

Netflix. 2021. VMAF - Video Multi-Method Assessment Fusion. Retrieved Sep 3, 2021 from https://github.com/Netflix/vmaf

[31]

K. Nevelsteen. 2018. Virtual world, defined from a technological perspective and applied to video games, mixed reality, and the Metaverse. Computer Animation and Virtual Worlds 29, 1 (2018), e1752. https://doi.org/10.1002/cav.1752

[32]

B. Ray, J. Jung, and M. Larabi. 2018. On the possibility to achieve 6-DoF for 360 video using divergent multi-view content. In Proc. of European Signal Processing Conference (EUSIPCO'18). Rome, Italy, 211--215. https://doi.org/10.23919/ EUSIPCO.2018.8553397

[33]

Sadbhawna, Vinit Jakhetiya, Deebha Mumtaz, Badri Narayan Subudhi, and Sharath Chandra Guntuku. 2022. Stretching Artifacts Identification for Quality Assessment of 3D-Synthesized Views. IEEE Transactions on Image Processing 31 (2022), 1737--1750. https://doi.org/10.1109/TIP.2022.3145997

[34]

Basel Salahieh, Sumit Bhatia, and Jill Boyce. 2019. Multi-Pass Renderer in MPEG Test Model for Immersive Video. In Proc. of IEEE International Conference on Picture Coding Symposium (PCS). 1--5. https://doi.org/10.1109/PCS48520.2019. 8954515

[35]

Sebastian Schwarz and M. Hannuksela. 2017. Perceptual quality assessment of HEVC main profile depth map compression for six degrees of freedom virtual reality video. In Proc. of IEEE International Conference on Image Processing (ICIP'17). Beijing, China, 181--185. https://doi.org/10.1109/ICIP.2017.8296267

Digital Library

[36]

BT Series. 2012. Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT (2012), 500--13.

[37]

Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. 2018. AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. In Field and Service Robotics. Springer, 621--635.

[38]

Jakub Szekieda, Adrian Dziembowski, and Dawid Mieloch. 2021. The Influence of Coding Tools on Immersive Video Coding. In Proc. of WSCG International Conference on Computer Graphics, Visualization and Computer Vision. Pilsen (Plzen), Czech Republic. https://doi.org/10.24132/CSRN.2021.3002.21

[39]

Shishun Tian, Lu Zhang, Wenbin Zou, Xia Li, Ting Su, Luce Morin, and Olivier Deforges. 2019. Quality Assessment of DIBR-synthesized views: An Overview. https://doi.org/10.48550/ARXIV.1911.07036

[40]

Xuejin Wang, Feng Shao, Qiuping Jiang, Randi Fu, and Yo-Sung Ho. 2019. Quality Assessment of 3D Synthesized Images via Measuring Local Feature Similarity and Global Sharpness. IEEE Access 7 (2019), 10242--10253. https://doi.org/10. 1109/ACCESS.2019.2891070

[41]

Jiebin Yan, Jing Li, Yuming Fang, Zhaohui Che, Xue Xia, and Yang Liu. 2022. Subjective and Objective Quality of Experience of Free Viewpoint Videos. IEEE Transactions on Image Processing 31 (2022), 3896--3907. https://doi.org/10.1109/ TIP.2022.3177127

Digital Library

[42]

Huiqing Zhang, Donghao Li, Zhifang Xia, Zichen Wang, and Guangchen Wang. 2020. Energy Loss Estimation Based Reference-Free Quality Assessment of DIBRSynthesized Views. In 2020 39th Chinese Control Conf

Cited By

Wei YSun YZheng SHsu HHuang CHsu C(2024)Mitigating Privacy Threats Without Degrading Visual Quality of VR Applications: Using Re-Identification Attack as a Case Study2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR)10.1109/MIPR62202.2024.00040(214-220)Online publication date: 7-Aug-2024
https://doi.org/10.1109/MIPR62202.2024.00040
Tang SSun YHsu CEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)A Blind Streaming System for Multi-client Online 6-DoF View TouringProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612257(9124-9133)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612257

Index Terms

On Objective and Subjective Quality of 6DoF Synthesized Live Immersive Videos
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
    2. Image manipulation
      1. Image-based rendering
2. Information systems
  1. Information systems applications
    1. Multimedia information systems
      1. Multimedia streaming

Recommendations

Towards Viewport-dependent 6DoF 360 Video Tiled Streaming for Virtual Reality Systems
MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Previous studies of 360-degree video streaming with regard to virtual reality allowed users to move their head freely, while their position is fixed according to the camera's location in virtual reality. One of the approaches to overcome the problem is ...
Subjective Evaluation of an Olfaction Enhanced Immersive Virtual Reality Environment
AltMM '17: Proceedings of the 2nd International Workshop on Multimedia Alternate Realities

Recent research efforts have reported findings on user Quality of Experience (QoE) of immersive virtual reality (VR) experiences. Truly immersive multimedia experiences also include multisensory components such as factional, tactile etc., in addition to ...
Subjective quality evaluation of compressed digital compound images

We build a new Compound Image Quality Assessment Database (CIQAD).Subjective test is conducted to get the MOS values of images in CIQAD.Five compression methods are compared with respect to the obtained MOS values.We study the applicability of existing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

QoEVMA '22: Proceedings of the 2nd Workshop on Quality of Experience in Visual Multimedia Applications

October 2022

75 pages

ISBN:9781450394994

DOI:10.1145/3552469

General Chairs:
Jing Li
Alibaba Group, China
,
Xinbo Gao
Xidian University, China
,
Patrick Le Callet
Nantes University, France
,
Zhi Li
Netflix Inc., U.S.
,
Wen Lu
Xidian University, China
,
Jiachen Yang
Tianjin University, China
,
Junle Wang
Tencent, China

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 14, 2022

Lisboa, Portugal

Acceptance Rates

QoEVMA '22 Paper Acceptance Rate 8 of 14 submissions, 57%;

Overall Acceptance Rate 14 of 20 submissions, 70%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
106
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)3

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wei YSun YZheng SHsu HHuang CHsu C(2024)Mitigating Privacy Threats Without Degrading Visual Quality of VR Applications: Using Re-Identification Attack as a Case Study2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR)10.1109/MIPR62202.2024.00040(214-220)Online publication date: 7-Aug-2024
https://doi.org/10.1109/MIPR62202.2024.00040
Tang SSun YHsu CEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)A Blind Streaming System for Multi-client Online 6-DoF View TouringProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612257(9124-9133)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612257

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents