research-article

ST-GREED: Space-Time Generalized Entropic Differences for Frame Rate Dependent Video Quality Prediction

Authors:

Pavan C. Madhusudana,

Balu Adsumilli,

Alan C. BovikAuthors Info & Claims

IEEE Transactions on Image Processing, Volume 30

Pages 7446 - 7457

https://doi.org/10.1109/TIP.2021.3106801

Published: 01 January 2021 Publication History

Abstract

We consider the problem of conducting frame rate dependent video quality assessment (VQA) on videos of diverse frame rates, including high frame rate (HFR) videos. More generally, we study how perceptual quality is affected by frame rate, and how frame rate and compression combine to affect perceived quality. We devise an objective VQA model called Space-Time GeneRalized Entropic Difference (GREED) which analyzes the statistics of spatial and temporal band-pass video coefficients. A generalized Gaussian distribution (GGD) is used to model band-pass responses, while entropy variations between reference and distorted videos under the GGD model are used to capture video quality variations arising from frame rate changes. The entropic differences are calculated across multiple temporal and spatial subbands, and merged using a learned regressor. We show through extensive experiments that GREED achieves state-of-the-art performance on the LIVE-YT-HFR Database when compared with existing VQA models. The features used in GREED are highly generalizable and obtain competitive performance even on standard, non-HFR VQA databases. The implementation of GREED has been made available online: <uri>https://github.com/pavancm/GREED</uri>.

References

[1]

R. M. Nasiri, J. Wang, A. Rehman, S. Wang, and Z. Wang, “Perceptual quality assessment of high frame rate video,” in Proc. IEEE Int. Workshop Multimedia Signal Process. (MMSP), Oct. 2015, pp. 1–6.

[2]

A. Mackin, F. Zhang, and D. R. Bull, “A study of high frame rate video formats,” IEEE Trans. Multimedia, vol. 21, no. 6, pp. 1499–1512, Jun. 2019.

Digital Library

[3]

P. C. Madhusudana, X. Yu, N. Birkbeck, Y. Wang, B. Adsumilli, and A. C. Bovik, “Subjective and objective quality assessment of high frame rate videos,” IEEE Access, vol. 9, pp. 108069–108082, 2021.

[4]

R. Soundararajan and A. C. Bovik, “Video quality assessment by reduced reference spatio-temporal entropic differencing,” IEEE Trans. Circuits Syst. Video Technol., vol. 23, no. 4, pp. 684–694, Apr. 2013.

Digital Library

[5]

Z. Li, A. Aaron, I. Katsavounidis, A. Moorthy, and M. Manohara. Toward a Practical Perceptual Video Quality Metric. Accessed: Nov.1, 2019. [Online]. Available: http://techblog.netflix.com/2016/06/toward-practical-perceptual-video.html

[6]

C. G. Bampis, P. Gupta, R. Soundararajan, and A. C. Bovik, “SpEED-QA: Spatial efficient entropic differencing for image and video quality,” IEEE Signal Process. Lett., vol. 24, no. 9, pp. 1333–1337, Sep. 2017.

[7]

C. G. Bampis, Z. Li, and A. C. Bovik, “Spatiotemporal feature integration and model fusion for full reference video quality assessment,” IEEE Trans. Circuits Syst. Video Technol., vol. 29, no. 8, pp. 2256–2270, Aug. 2019.

[8]

S. Chikkerur, V. Sundaram, M. Reisslein, and L. J. Karam, “Objective video quality assessment methods: A classification, review, and performance comparison,” IEEE Trans. Broadcast., vol. 57, no. 2, pp. 165–182, Jun. 2011.

[9]

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.

Digital Library

[10]

Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in Proc. Asilomar Conf. Signals, Syst. Comput., vol. 2, Nov. 2003, pp. 1398–1402.

[11]

L. Zhang, L. Zhang, X. Mou, and D. Zhang, “FSIM: A feature similarity index for image quality assessment,” IEEE Trans. Image Process., vol. 20, no. 8, pp. 2378–2386, Aug. 2011.

Digital Library

[12]

M. H. Pinson and S. Wolf, “A new standardized method for objectively measuring video quality,” IEEE Trans. Broadcast., vol. 50, no. 3, pp. 312–322, Sep. 2004.

[13]

M. H. Pinson, L. K. Choi, and A. C. Bovik, “Temporal video quality model accounting for variable frame delay distortions,” IEEE Trans. Broadcast., vol. 60, no. 4, pp. 637–649, Dec. 2014.

[14]

A. K. Moorthy, L. K. Choi, A. C. Bovik, and G. de Veciana, “Video quality assessment on mobile devices: Subjective, behavioral and objective studies,” IEEE J. Sel. Topics Signal Process., vol. 6, no. 6, pp. 652–671, Oct. 2012.

[15]

K. Seshadrinathan and A. C. Bovik, “Motion tuned spatio-temporal quality assessment of natural videos,” IEEE Trans. Image Process., vol. 19, no. 2, pp. 335–350, Feb. 2010.

Digital Library

[16]

K. Seshadrinathan and A. C. Bovik, “A structural similarity metric for video based on motion models,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., Apr. 2007, p. 869.

[17]

P. V. Vu, C. T. Vu, and D. M. Chandler, “A spatiotemporal most-apparent-distortion model for video quality assessment,” in Proc. IEEE Int. Conf. Image Process., Sep. 2011, pp. 2505–2508.

[18]

E. C. Larson and D. M. Chandler, “Most apparent distortion: Full-reference image quality assessment and the role of strategy,” J. Electron. Imag., vol. 19, no. 1, 2010, Art. no.

[19]

J. You, T. Ebrahimi, and A. Perkis, “Attention driven foveated video quality assessment,” IEEE Trans. Image Process., vol. 23, no. 1, pp. 200–213, Jan. 2014.

Digital Library

[20]

B. Ortiz-Jaramillo, A. Kumcu, L. Platisa, and W. Philips, “A full reference video quality measure based on motion differences and saliency maps evaluation,” in Proc. Int. Conf. Comput. Vis. Theory Appl., vol. 2, Jan. 2014, pp. 714–722.

[21]

K. Manasa and S. S. Channappayya, “An optical flow-based full reference video quality assessment algorithm,” IEEE Trans. Image Process., vol. 25, no. 6, pp. 2480–2492, Jun. 2016.

Digital Library

[22]

W. Kim, J. Kim, S. Ahn, J. Kim, and S. Lee, “Deep video quality assessor: From spatio-temporal visual sensitivity to a convolutional neural aggregation network,” in Proc. Eur. Conf. Comput. Vis., Sep. 2018, pp. 219–234.

[23]

S. Becker, K.-R. Müller, T. Wiegand, and S. Bosse, “A neural network model of spatial distortion sensitivity for video quality estimation,” in Proc. IEEE Int. Workshop Mach. Learn. Signal Process., Oct. 2019, pp. 1–6.

[24]

M. Xu, J. Chen, H. Wang, S. Liu, G. Li, and Z. Bai, “C3DVQA: Full-reference video quality assessment with 3D convolutional neural network,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., May 2020, pp. 4447–4451.

[25]

R. M. Nasiri and Z. Wang, “Perceptual aliasing factors and the impact of frame rate on video quality,” in Proc. IEEE Int. Conf. Image Process., Sep. 2017, pp. 3475–3479.

[26]

R. M. Nasiri, Z. Duanmu, and Z. Wang, “Temporal motion smoothness and the impact of frame rate variation on video quality,” in Proc. IEEE Int. Conf. Image Process., Oct. 2018, pp. 1418–1422.

[27]

F. Zhang, A. Mackin, and D. R. Bull, “A frame rate dependent video quality metric based on temporal wavelet decomposition and spatiotemporal pooling,” in Proc. IEEE Int. Conf. Image Process., Sep. 2017, pp. 300–304.

[28]

P. C. Madhusudana, N. Birkbeck, Y. Wang, B. Adsumilli, and A. C. Bovik, “Capturing video frame rate variations via entropic differencing,” IEEE Signal Process. Lett., vol. 27, pp. 1809–1813, 2020.

[29]

M. A. Saad, A. C. Bovik, and C. Charrier, “Blind prediction of natural video quality,” IEEE Trans. Image Process., vol. 23, no. 3, pp. 1352–1365, Mar. 2014.

Digital Library

[30]

H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Trans. Image Process., vol. 15, no. 2, pp. 430–444, Feb. 2006.

Digital Library

[31]

R. Soundararajan and A. C. Bovik, “RRED indices: Reduced reference entropic differencing for image quality assessment,” IEEE Trans. Image Process., vol. 21, no. 2, pp. 517–526, Feb. 2012.

Digital Library

[32]

D. L. Ruderman, “The statistics of natural images,” Netw., Comput. Neural Syst., vol. 5, no. 4, pp. 517–548, 1994.

[33]

A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Trans. Image Process., vol. 21, no. 12, pp. 4695–4708, Dec. 2012.

Digital Library

[34]

A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a ‘completely blind’ image quality analyzer,” IEEE Signal Process. Lett., vol. 20, no. 3, pp. 209–212, Mar. 2013.

[35]

S. G. Chang, B. Yu, and M. Vetterli, “Adaptive wavelet thresholding for image denoising and compression,” IEEE Trans. Image Process., vol. 9, no. 9, pp. 1532–1546, Sep. 2000.

Digital Library

[36]

M. N. Do and M. Vetterli, “Wavelet-based texture retrieval using generalized Gaussian density and Kullback–Leibler distance,” IEEE Trans. Image Process., vol. 11, no. 2, pp. 146–158, Feb. 2002.

Digital Library

[37]

Q. Zhao, H.-W. Li, and Y.-T. Shen, “On the sum of generalized Gaussian random signals,” in Proc. IEEE Int. Conf. Signal Process., Aug./Sep. 2004, pp. 50–53.

[38]

H. Soury and M.-S. Alouini, “New results on the sum of two generalized Gaussian random variables,” in Proc. IEEE Global Conf. Signal Inf. Process., Dec. 2015, pp. 1017–1021.

[39]

X. Pan, X. Zhang, and S. Lyu, “Exposing image splicing with inconsistent local noise variances,” in Proc. IEEE Int. Conf. Comput. Photogr. (ICCP), Apr. 2012, pp. 1–10.

[40]

FFmpeg. Encoding for Streaming Sites. Accessed: Nov.1, 2019. [Online]. Available: https://trac.ffmpeg.org/wiki

[41]

B. Schölkopf, A. J. Smola, R. C. Williamson, and P. L. Bartlett, “New support vector algorithms,” Neural Comput., vol. 12, no. 5, pp. 1207–1245, 2000.

Digital Library

[42]

C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vector machines,” ACM Trans. Intell. Syst. Technol., vol. 2, no. 3, pp. 27:1–27:27, May 2011.

Digital Library

[43]

R. R. Coifman and M. V. Wickerhauser, “Entropy-based algorithms for best basis selection,” IEEE Trans. Inf. Theory, vol. 38, no. 2, pp. 713–718, Mar. 1992.

Digital Library

[44]

K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack, “Study of subjective and objective quality assessment of video,” IEEE Trans. Image Process., vol. 19, no. 6, pp. 1427–1441, Jun. 2010.

Digital Library

[45]

“Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment,” VQEG, Tech. Rep., 2000.

[46]

J. G. Robson, “Spatial and temporal contrast-sensitivity functions of the visual system,” J. Opt. Soc. Amer., vol. 56, no. 8, pp. 1141–1142, 1966.

[47]

P. V. Vu and D. M. Chandler, “ViS3: An algorithm for video quality assessment via analysis of spatial and spatiotemporal slices,” J. Electron. Imag., vol. 23, no. 1, 2014, Art. no.

Cited By

Tan XZhang JQuan YLi JWu YBian ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Highly Efficient No-reference 4K Video Quality Assessment with Full-Pixel Covering Sampling and Training StrategyProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680907(9913-9922)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680907
Chen YSaha AChapiro AHäne CBazin JQiu BZanetti SKatsavounidis IBovik A(2024)Subjective and Objective Quality Assessment of Rendered Human Avatar Videos in Virtual RealityIEEE Transactions on Image Processing10.1109/TIP.2024.346888133(5740-5754)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3468881
Ebenezer JShang ZChen YWu YWei HSethuraman SBovik A(2024)HDR or SDR? A Subjective and Objective Study of Scaled and Compressed VideosIEEE Transactions on Image Processing10.1109/TIP.2024.340489033(3606-3619)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3404890
Show More Cited By

Recommendations

Frame Rate and Perceptual Quality for HD Video
Proceedings, Part II, of the 16th Pacific-Rim Conference on Advances in Multimedia Information Processing -- PCM 2015 - Volume 9315

The frame rate FR of a video plays an important role in affecting the perceptual video quality. Most studies about the effect of FR on the video quality mainly focused on low frame rate, e.g. less than 30 frames per second fps, at low resolutions like ...
Quality Enhancement of Frame Rate Up-Converted Video by Adaptive Frame Skip and Reliable Motion Extraction

Frame rate up-conversion is a postprocessing tool to convert the frame rate from a lower number to a higher one. It is a useful technique for a lot of practical applications, such as display format conversion, low bit rate video coding and slow motion ...
Fast frame-rate up-conversion of depth video via video coding
MM '11: Proceedings of the 19th ACM international conference on Multimedia

Recent development of depth sensors has facilitated the progress of 2D-plus-depth methods for 3D video representation, for which frame-rate up-conversion (FRUC) of depth video is a critical step. However, due to the computational cost of state-of-the-...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Image Processing

IEEE Transactions on Image Processing Volume 30, Issue

2021

5053 pages

ISSN:1057-7149

Issue’s Table of Contents

1941-0042 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 January 2021

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tan XZhang JQuan YLi JWu YBian ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Highly Efficient No-reference 4K Video Quality Assessment with Full-Pixel Covering Sampling and Training StrategyProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680907(9913-9922)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680907
Chen YSaha AChapiro AHäne CBazin JQiu BZanetti SKatsavounidis IBovik A(2024)Subjective and Objective Quality Assessment of Rendered Human Avatar Videos in Virtual RealityIEEE Transactions on Image Processing10.1109/TIP.2024.346888133(5740-5754)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3468881
Ebenezer JShang ZChen YWu YWei HSethuraman SBovik A(2024)HDR or SDR? A Subjective and Objective Study of Scaled and Compressed VideosIEEE Transactions on Image Processing10.1109/TIP.2024.340489033(3606-3619)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3404890
Wu XLópez-Tapia SWang XMolina RKatsaggelos A(2024)Real-Time Lightweight Video Super-Resolution With RRED-Based Perceptual ConstraintIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.340582734:10_Part_2(10310-10325)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3405827
Herglotz CKränzler MLudwig RKaup AAfzal SFerlin-Reiter SHoßfeld TSchatz RTimmerer C(2023)Video Decoding Energy Reduction Using Temporal-Domain FilteringProceedings of the First International Workshop on Green Multimedia Systems10.1145/3593908.3593948(22-27)Online publication date: 7-Jun-2023
https://dl.acm.org/doi/10.1145/3593908.3593948
Wu HChen CLiao LHou JSun WYan QGu JLin W(2023)Neighbourhood Representative Sampling for Efficient End-to-End Video Quality AssessmentIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.331933245:12(15185-15202)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TPAMI.2023.3319332
Danier DZhang FBull D(2023)BVI-VFI: A Video Quality Database for Video Frame InterpolationIEEE Transactions on Image Processing10.1109/TIP.2023.332791232(6004-6019)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIP.2023.3327912
Madhusudana PBirkbeck NWang YAdsumilli BBovik A(2023)CONVIQT: Contrastive Video Quality EstimatorIEEE Transactions on Image Processing10.1109/TIP.2023.331034432(5138-5152)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIP.2023.3310344
Saha AChen YDavis CQiu BWang XGowda RKatsavounidis IBovik A(2023)Study of Subjective and Objective Quality Assessment of Mobile Cloud Gaming VideosIEEE Transactions on Image Processing10.1109/TIP.2023.328117032(3295-3310)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIP.2023.3281170
Wu HChen CLiao LHou JSun WYan QLin W(2023)DisCoVQA: Temporal Distortion-Content Transformers for Video Quality AssessmentIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.324974133:9(4840-4854)Online publication date: 28-Feb-2023
https://dl.acm.org/doi/10.1109/TCSVT.2023.3249741
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents