research-article

No-reference Stereoscopic Image Quality Assessment Using Binocular Self-similarity and Deep Neural Network

Authors:

Fen ChenAuthors Info & Claims

Image Communication, Volume 47, Issue C

Pages 346 - 357

https://doi.org/10.1016/j.image.2016.07.003

Published: 01 September 2016 Publication History

Abstract

Quality assessment of three-dimensional (3D) images is more challenging than that of 2D images. The quality of 3D visual experience is one of the most challenging areas of human binocular perception and is affected by multiple factors such as asymmetric stereo image/video compression, depth perception, visual discomfort, and single view quality. In this paper, we propose a new no-reference quality assessment method for stereoscopic images based on Binocular Self-similarity (BS) and Deep Neural Networks (DNN). To be more specific, a BS index is defined and computed according to binocular rivalry and suppression based on the depth image-based rendering technique. Then, a DNN is trained in an opinion unaware way to predict local quality. Binocular integration (BI) index is calculated by using the trained DNN, accounting for binocular integration behaviors. Finally, the final quality score of stereoscopic image is obtained by combining the BS and BI indexes together. Experimental results on four public 3D image quality assessment databases demonstrate that compared with existing methods, the proposed method can achieve high consistency with subjective perception on stereoscopic images with both symmetric and asymmetric distortions. We propose a new no-reference quality assessment method for stereoscopic images.Two indexes, Binocular Self-similarity and binocular integration, are defined.We train a Deep Neural Network in an opinion unaware way to predict local quality.Experimental results show this method is consistent with subjective perception.

References

[1]

W. Lin, C.-C.J. Kuo, Perceptual visual quality metrics: a survey, J. Vis. Commun. Image Represent., 22 (2011) 297-312.

Digital Library

[2]

A.K. Moorthy, A.C. Bovik, A survey on 3D quality of experience and 3D quality assessment, in: Proceedings of the IS&T/SPIE Electronic Imaging International Society for Optics and Photonics, vol. 8651, 2013, pp. 86510M-86510M-11.

[3]

Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process., 13 (2004) 600-612.

Digital Library

[4]

H.R. Sheikh, A.C. Bovik, Image information and visual quality, IEEE Trans. Image Process., 15 (2006) 430-444.

Digital Library

[5]

L. Zhang, L. Zhang, X. Mou, D. Zhang, FSIM: a feature similarity index for image quality assessment, IEEE Trans. Image Process., 20 (2011) 2378-2386.

Digital Library

[6]

W. Xue, L. Zhang, X. Mou, A. Bovik, Gradient magnitude similarity deviation: a highly efficient perceptual image quality index, IEEE Trans. Image Process., 23 (2014) 684-695.

Digital Library

[7]

A.K. Moorthy, A.C. Bovik, Blind image quality assessment: from natural scene statistics to perceptual quality, IEEE Trans. Image Process., 20 (2011) 3350-3364.

Digital Library

[8]

M.A. Saad, A.C. Bovik, C. Charrier, A. DCT, statistics-based blind image quality index, IEEE Signal Process. Lett., 17 (2010) 583-586.

[9]

A. Mittal, A.K. Moorthy, A.C. Bovik, No-reference image quality assessment in the spatial domain, IEEE Trans. Image Process., 21 (2012) 4695-4708.

Digital Library

[10]

H. Tang, N. Joshi, A. Kapoor, Learning a blind measure of perceptual image quality, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2011, pp. 305-312.

Digital Library

[11]

P. Ye, J. Kumar, L. Kang, D. Doermann, Unsupervised feature learning framework for no-reference image quality assessment, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 1098-1105.

Digital Library

[12]

P. Zhang, W. Zhou, L. Wu, H. Li, SOM: semantic obviousness metric for image quality assessment, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 2394-2402.

[13]

A. Mittal, G.S. Muralidhar, A.C. Bovik, Making a completely blind image quality analyzer, IEEE Signal Process. Lett., 20 (2013) 209-212.

[14]

L. Zhang, L. Zhang, A.C. Bovik, A feature-enriched completely blind image quality evaluator, IEEE Trans. Image Process., 24 (2015) 2579-2591.

[15]

W. Xue, L. Zhang, X. Mou, Learning without human scores for blind image quality assessment, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 9, 2013, pp. 995-1002.

Digital Library

[16]

P. Ye, J. Kumar, D. Doermann, Beyond human opinion scores: blind image quality assessment based on synthetic scores, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 4241-4248.

Digital Library

[17]

A. Benoit, P. Le Callet, P. Campisi, R. Cousseau, Using disparity for quality assessment of stereoscopic images, EURASIP J. Image Video Process., 6957 (2008) 389-392.

[18]

J. You, L. Xing, A. Perkis, X. Wang, Perceptual quality assessment for stereoscopic images based on 2D image quality metrics and disparity analysis, in: Proceedings of the International Workshop Video Process, 2010, pp. 61-66.

[19]

S.L.P. Yasakethu, C.T.E.R. Hewage, W.A.C. Fernando, A.M. Kondoz, Quality analysis for 3D video using 2D video quality models, IEEE Trans. Consum. Electron., 54 (2008) 1969-1976.

Digital Library

[20]

X. Wang, S. Kwong, Y. Zhang, Considering binocular spatial sensitivity in stereoscopic image quality assessment, in: Proceedings of the IEEE Vis. Commun. Image Process. 2011, pp. 1-4.

[21]

A. Maalouf, M.C. Larabi, CYCLOP: a stereo color image quality assessment metric, in: Proceedings of the IEEE Int. Conf. Acoust. Speech Signal Process., 2011, pp. 1161-1164.

[22]

L. Jin, A. Boev, A. Gotchev, K. Egiazarian, 3D-DCT based perceptual quality assessment of stereo video, in: Proceedings of the IEEE International Conference on Image Processing, vol. 6626, 2011, pp. 2521-2524.

[23]

S.A. Fezza, M.C. Larabi, K.M. Faraoun, Asymmetric coding of stereoscopic 3D based on perceptual significance, in: Proceedings of the IEEE International Conference on Image Processing, 2014, pp. 5656-5660.

[24]

E. Ekmekcioglu, V.D. Silva, P.T. Pesch, A. Kondoz, Visual attention model aided non-uniform asymmetric coding of stereoscopic video, IEEE J. Sel. Top. Signal Process., 8 (2014) 402-414.

[25]

R. Bensalma, M.-C. Larabi, A perceptual metric for stereoscopic image quality assessment based on the binocular energy, Multidimens. Syst. Signal Process., 24 (2013) 281-316.

Digital Library

[26]

M.-J. Chen, C.-C. Su, D.-K. Kwon, L.K. Cormack, A.C. Bovik, Full-reference quality assessment of stereopairs accounting for binocular rivalry, Signal Process. Image Commun. 28 (9), pp. 1143-1155.

Digital Library

[27]

Y.-H. Lin, J.-L. Wu, Quality assessment of stereoscopic 3D image compression by binocular integration behaviors, IEEE Trans. Image Process., 23 (2014) 1527-1542.

Digital Library

[28]

F. Shao, K. Li, W. Lin, G. Jiang, M. Yu, Q. Dai, Full-reference quality assessment of stereoscopic images by learning binocular receptive field properties, IEEE Trans. Image Process., 24 (2015) 2971-2983.

[29]

K. Lee, S. Lee, 3D perception based quality pooling: stereopsis, binocular rivalry, and binocular suppression, IEEE J. Sel. Top. Signal Process., 9 (2015) 533-545.

[30]

V. De Silva, H.K. Arachchi, E. Ekmekcioglu, A. Kondoz, Toward an impairment metric for stereoscopic video: a full-reference video quality metric to assess compressed stereoscopic video, IEEE Trans. Image Process., 22 (2013) 3392-3404.

Digital Library

[31]

Z.M.P. Sazzad, R. Akhter, J. Baltes, Y. Horita, Objective no-reference stereoscopic image quality prediction based on 2D image features and relative disparity, Adv. Multimed. (2012).

Digital Library

[32]

M.-J. Chen, L.K. Cormack, A.C. Bovik, No-reference quality assessment of natural stereopairs, IEEE Trans. Image Process., 22 (2013) 3379-3391.

Digital Library

[33]

C.-C. Su, L.K. Cormack, A.C. Bovik, Oriented correlation models of distorted natural images with application to natural stereopair quality evaluation, IEEE Trans. Image Process., 24 (2015) 1685-1699.

[34]

S. Ryu, D.H. Kim, K. Sohn, Stereoscopic image quality metric based on binocular perception model, in: Proceedings of International Conference on Image Processing, 2012, pp. 609-612.

[35]

F. Shao, W. Lin, S. Wang, G. Jiang, M. Yu, Blind image quality assessment for stereoscopic images using binocular guided quality lookup and visual codebook, IEEE Trans. Broadcast., 61 (2015) 154-165.

[36]

A.K. Moorthy, A.C. Bovik, A two-step framework for constructing blind image quality indices, IEEE Signal Process. Lett., 17 (2010) 513-516.

[37]

L. Zhang, Z. Gu, X. Liu, H. Li, J. Lu, Training quality-aware filters for no-reference image quality assessment, IEEE Multimed. Mag., 22 (2014) 67-75.

[38]

C. Li, A.C. Bovik, X. Wu, Blind image quality assessment using a general regression neural network, IEEE Trans. Neural Netw., 22 (2011) 793-799.

Digital Library

[39]

G.E. Hinton, S. Osindero, Y.W. Teh, A fast learning algorithm for deep belief nets, Neural Comput., 18 (2006) 1527-1554.

Digital Library

[40]

Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle, Greedy layer-wise training of deep networks, Adv. Neural Inf. Process. Syst., 19 (2007) 153-160.

Digital Library

[41]

Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proc. IEEE, 86 (1998) 2278-2324.

[42]

L. Kang, P. Ye, Y. Li, D. Doermann, Convolutional neural networks for no-reference image quality assessment, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1733-1740.

Digital Library

[43]

H. Tang, N. Joshi, A. Kapoor, Blind image quality assessment using semi-supervised rectifier networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2877-2884.

Digital Library

[44]

W. Hou, X. Gao, D. Tao, X. Li, Blind image quality assessment via deep learning, IEEE Trans. Neural Netw. Learn. Syst., 26 (2014) 1275-1286.

[45]

Y. Lv, G. Jiang, M. Yu, H. Xu, F. Shao, S. Liu, Difference of Gaussian statistical features based blind image quality assessment: a deep learning approach, in: Proceedings of the IEEE International Conference on Image Processing, 2015, pp. 2344-2348.

[46]

Y. Jung, H. Kim, Y. Ro, Critical binocular asymmetry measure for perceptual quality assessment of synthesized stereo 3D images in view synthesis, IEEE Trans. Circuits Syst. Video Technol. (2015).

[47]

C. Fehn, A. 3D-TV approach using depth-image-based rendering (DIBR), in: Proceedings of Picture Coding Symposium, 2004, pp. 482-487.

[48]

D. Sun, S. Roth, M.J. Black, Secrets of optical flow estimation and their principles, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2010, pp. 2432-2439.

[49]

A. Mittal, A.K. Moorthy, A.C. Bovik, Making image quality assessment robust, in: Proceedings of the Asilomar Conference on Circuits, Systems and Computers, 2012, pp.1718-1722.

[50]

Z. Wang, E.P. Simoncelli, and A.C. Bovik, Multiscale structural similarity for image quality assessment, in: Proceedings of the Asilomar Conference on Signals Systems and Computers, 2003, pp. 1398-1402.

[51]

Y. Lecun, Y. Bengio, G.E. Hinton, Deep learning, Nature, 521 (2015) 436-444.

[52]

V. Nair, G.E. Hinton, Rectified linear units improve restricted boltzmann machines, Adv. Neural Inf. Process. Syst., 19 (2007) 153-160.

[53]

G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, R.R. Salakhutdinov, Improving neural networks by preventing co-adaptation of feature detectors, ResearchGate, 3 (2012) 212-223.

[54]

J. Wang, A. Rehman, K. Zeng, S. Wang, Z. Wang, Quality prediction of asymmetrically distorted stereoscopic 3D images, IEEE Trans. Image Process., 24 (2015) 3400-3414.

[55]

X. Wang, M. Yu, Y. Yang, G. Jiang, Research on subjective stereoscopic image quality assessment, in: Proceedings of the IS&T/SPIE Electronic Imaging International Society for Optics and Photonics, 2009, pp. 725509-725509-10.

[56]

Methodology for the Subjective Assessment of the Quality of Television Pictures, ITU-R Standard BT.500-511, 2002.

[57]

J. Zhou, G. Jiang, X. Mao, M. Yu, F. Shao, Z. Peng, Y. Zhang, Subjective quality analyses of stereoscopic images in 3DTV system, in: Proceedings of the IEEE Visual Commun. Image Process., 2011.

[58]

A.K. Moorthy, C.-C. Su, A. Mittal, A.C. Bovik, Subjective evaluation of stereoscopic image quality, Signal Process.: Image Commun., 28 (2013) 870-883.

Cited By

Li CYun LXu S(2022)Blind stereoscopic image quality assessment using 3D saliency selected binocular perception and 3D convolutional neural networkMultimedia Tools and Applications10.1007/s11042-022-12707-481:13(18437-18455)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1007/s11042-022-12707-4
Bayrak HNur Yilmaz G(2019)A depth perception evaluation metric for immersive user experience towards 3D multimedia servicesMultimedia Systems10.1007/s00530-018-00602-825:3(253-261)Online publication date: 10-Oct-2019
https://dl.acm.org/doi/10.1007/s00530-018-00602-8
Jiang QShao FLin WJiang G(2018)Learning a referenceless stereopair quality engine with deep nonnegativity constrained sparse autoencoderPattern Recognition10.1016/j.patcog.2017.11.00176:C(242-255)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.1016/j.patcog.2017.11.001
Show More Cited By

No-reference Stereoscopic Image Quality Assessment Using Binocular Self-similarity and Deep Neural Network
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches

Recommendations

Deep network based stereoscopic image quality assessment via binocular summing and differencing
Abstract
With the development of deep networks in dealing with various visual tasks, the deep network based on binocular vision is expected to tackle the issue of stereoscopic image quality assessment. Here, we present a stereoscopic image ...
No-Reference Stereoscopic Image Quality Assessment Based on Binocular Visual Saliency Weighting
ICIGP '23: Proceedings of the 2023 6th International Conference on Image and Graphics Processing

Deep learning is widely used in the field of image quality assessment. The complex binocular vision mechanism and the multidimensional characteristics make the quality assessment of stereoscopic images more challenging. In this paper, a new no-reference ...
Full-reference stereoscopic image quality assessment accounting for binocular combination and disparity information
2017 IEEE International Conference on Image Processing (ICIP)
One of the most challenging issues in stereoscopic image quality assessment (SIQA) is how to effectively model the binocular behavior of the human visual system (HVS). The latter has a great impact on the perceptual 3D quality. In this paper, we propose a ...

Comments

Information & Contributors

Information

Published In

cover image Image Communication

Image Communication Volume 47, Issue C

September 2016

556 pages

ISSN:0923-5965

Issue’s Table of Contents

Copyright © Elsevier B.V.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 September 2016

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li CYun LXu S(2022)Blind stereoscopic image quality assessment using 3D saliency selected binocular perception and 3D convolutional neural networkMultimedia Tools and Applications10.1007/s11042-022-12707-481:13(18437-18455)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1007/s11042-022-12707-4
Bayrak HNur Yilmaz G(2019)A depth perception evaluation metric for immersive user experience towards 3D multimedia servicesMultimedia Systems10.1007/s00530-018-00602-825:3(253-261)Online publication date: 10-Oct-2019
https://dl.acm.org/doi/10.1007/s00530-018-00602-8
Jiang QShao FLin WJiang G(2018)Learning a referenceless stereopair quality engine with deep nonnegativity constrained sparse autoencoderPattern Recognition10.1016/j.patcog.2017.11.00176:C(242-255)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.1016/j.patcog.2017.11.001
Zhou WChen ZLi W(2018)Stereoscopic Video Quality Prediction Based on End-to-End Dual Stream Deep Neural NetworksAdvances in Multimedia Information Processing – PCM 201810.1007/978-3-030-00764-5_44(482-492)Online publication date: 21-Sep-2018
https://dl.acm.org/doi/10.1007/978-3-030-00764-5_44
Jiang GXu HYu MLuo TZhang Y(2017)Stereoscopic image quality assessment by learning non-negative matrix factorization-based color visual characteristics and considering binocular interactionsJournal of Visual Communication and Image Representation10.1016/j.jvcir.2017.04.01046:C(269-279)Online publication date: 1-Jul-2017
https://dl.acm.org/doi/10.1016/j.jvcir.2017.04.010
Hu BLi LWu JWang STang LQian J(2017)No-reference quality assessment of compressive sensing image recoveryImage Communication10.1016/j.image.2017.08.00358:C(165-174)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1016/j.image.2017.08.003

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents