research-article

Image Super-Resolution Using Deep Convolutional Networks

Authors:

Chen Change Loy,

Xiaoou TangAuthors Info & Claims

IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 38, Issue 2

Pages 295 - 307

https://doi.org/10.1109/TPAMI.2015.2439281

Published: 01 February 2016 Publication History

Abstract

We propose a deep learning method for single image super-resolution (SR). Our method directly learns an end-to-end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art restoration quality, and achieves fast speed for practical on-line usage. We explore different network structures and parameter settings to achieve trade-offs between performance and speed. Moreover, we extend our network to cope with three color channels simultaneously, and show better overall reconstruction quality.

References

[1]

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311– 4322, Nov. 2006.

Digital Library

[2]

M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. A. Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” in Proc. Brit. Mach. Vis. Conf., 2012, pp. 1–10.

[3]

H. C. Burger, C. J. Schuler, and S. Harmeling, “ Image denoising: Can plain neural networks compete with BM3D?” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2392–2399.

[4]

H. Chang, D. Y. Yeung, and Y. Xiong, “ Super-resolution through neighbor embedding,” presented at the IEEE Conf. Comput. Vis. Pattern Recog., Washington, DC, USA, 2004.

[5]

Z. Cui, H. Chang, S. Shan, B. Zhong, and X. Chen, “Deep network cascade for image super-resolution,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 49–64.

[6]

D. Dai, R. Timofte, and L. Van Gool, “Jointly optimized regressors for image super-resolution,” Eurographics, vol. 7, p. 8, 2015.

[7]

S. Dai, M. Han, W. Xu, Y. Wu, Y. Gong, and A. K. Katsaggelos, “ Softcuts: A soft edge smoothness prior for color image super-resolution,” IEEE Trans. Image Process., vol. 18, no. 11, pp. 969–981, May 2009.

[8]

N. Damera-Venkata, T. D. Kite, W. S. Geisler, B. L. Evans, and A. C. Bovik, “Image quality assessment based on a degradation model, ” IEEE Trans. Image Process., vol. 9, no. 11, pp. 636–650, Apr. 2000.

Digital Library

[9]

J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, and L. Fei-Fei, “ ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 248–255.

[10]

E. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, “Exploiting linear structure within convolutional networks for efficient evaluation,” in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 1269–1277.

[11]

C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 184–199.

[12]

D. Eigen, D. Krishnan, and R. Fergus, “Restoring an image taken through a window covered with dirt or rain,” in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 633–640.

[13]

G. Freedman and R. Fattal, “Image and video upscaling from local self-examples,” ACM Trans. Graph., vol. 30, no. 11, p. 12, 2011.

[14]

W. T. Freeman, T. R. Jones, and E. C. Pasztor, “ Example-based super-resolution,” Comput. Graph. Appl., vol. 22, no. 11, pp. 56–65, 2002.

Digital Library

[15]

W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, “ Learning low-level vision,” Int. J. Comput. Vis., vol. 40, no. 11, pp. 25–47, 2000.

Digital Library

[16]

D. Glasner, S. Bagon, and M. Irani, “ Super-resolution from a single image,” in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 349–356.

[17]

K. He and J. Sun, “ Convolutional neural networks at constrained time cost,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3791–3799.

[18]

K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 346–361.

[19]

J. B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 5197–5206.

[20]

M. Irani and S. Peleg, “Improving resolution by image registration,” Graph. Models Image Process., vol. 53, no. 11, pp. 231–239, 1991.

Digital Library

[21]

V. Jain and S. Seung, “Natural image denoising with convolutional networks,” in Proc. Adv. Neural Inf. Process. Syst., 2008, pp. 769–776.

[22]

K. Jia, X. Wang, and X. Tang, “Image transformation based on learning dictionaries across image spaces,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 35, no. 11, pp. 367–380, Feb. 2013.

[23]

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in Proc. ACM Int. Conf. Multimedia, 2014, pp. 675–678.

[24]

K. I. Kim and Y. Kwon, “Single-image super-resolution using sparse regression and natural image prior, ” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 6, pp. 1127–1133, Jun. 2010.

[25]

A. Krizhevsky, I. Sutskever, and G. Hinton, “ImageNet classification with deep convolutional neural networks,” in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1097–1105.

[26]

Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “ Backpropagation applied to handwritten zip code recognition,” Neural Comput., vol. 1, no. 4, pp. 541–551, 1989.

Digital Library

[27]

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov. 1998.

[28]

H. Lee, A. Battle, R. Raina, and A. Y. Ng, “Efficient sparse coding algorithms, ” in Proc. Adv. Neural Inf. Process. Syst., 2006, pp. 801– 808.

[29]

C. Liu, H. Y. Shum, and W. T. Freeman, “Face hallucination: Theory and practice,” Int. J. Comput. Vis., vol. 75, no. 11, pp. 115–134, 2007.

Digital Library

[30]

F. Mamalet and C. Garcia, “Simplifying convNets for fast learning,” in Proc. Int. Conf. Artif. Neural Netw., 2012, pp. 58–65.

[31]

D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics, ” in Proc. IEEE Int. Conf. Comput. Vis., 2001, vol. 2, pp. 416–423.

[32]

V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in Proc. Int. Conf. Mach. Learn., 2010, pp. 807–814.

[33]

W. Ouyang, X. Wang, X. Zeng, S. Qiu, P. Luo, Y. Tian, H. Li, S. Yang, Z. Wang, C.-C. Loy, and X. Tang, “Deepid-net: Deformable deep convolutional neural networks for object detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recogn., 2015, pp. 2403–2412.

[34]

W. Ouyang and X. Wang, “Joint deep learning for pedestrian detection,” in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 2056–2063.

[35]

C. J. Schuler, H. C. Burger, S. Harmeling, and B. Scholkopf, “A machine learning approach for non-blind image deconvolution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 1067–1074.

[36]

S. Schulter, C. Leistner, and H. Bischof, “Fast and accurate image upscaling with super-resolution forests,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3791–3799.

[37]

H. R. Sheikh, A. C. Bovik, and G. De Veciana, “An information fidelity criterion for image quality assessment using natural scene statistics,” IEEE Trans. Image Process., vol. 14, no. 11, pp. 2117 –2128, Dec. 2005.

Digital Library

[38]

Y. Sun, Y. Chen, X. Wang, and X. Tang, “Deep learning face representation by joint identification-verification,” in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 1988–1996.

[39]

R. Timofte, V. De Smet, and L. Van Gool, “Anchored neighborhood regression for fast example-based super-resolution,” in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1920–1927.

[40]

R. Timofte, V. De Smet, and L. Van Gool, “A+: Adjusted anchored neighborhood regression for fast super-resolution,” in Proc. IEEE Asian Conf. Comput. Vis., 2014, pp. 111–126.

[41]

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process. , vol. 13, no. 11, pp. 600–612, Apr. 2004.

Digital Library

[42]

Z. Wang, E. P. Simoncelli, and A. C. Bovik, “ Multiscale structural similarity for image quality assessment,” in Proc. IEEE Conf. Rec. 37th Asilomar Conf. Signals, Syst. Comput., 2003, vol. 2, pp. 1398–1402.

[43]

C. Y. Yang, J. B. Huang, and M. H. Yang, “Exploiting self-similarities for single frame super-resolution,” in Proc. IEEE Asian Conf. Comput. Vis., 2010, pp. 497–510.

[44]

C. Y. Yang, C. Ma, and M. H. Yang, “ Single-image super-resolution: A benchmark,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 372–386.

[45]

J. Yang, Z. Lin, and S. Cohen, “Fast image super-resolution based on in-place example regression,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 1059–1066.

[46]

J. Yang, Z. Wang, Z. Lin, S. Cohen, and T. Huang, “Coupled dictionary training for image super-resolution,” IEEE Trans. Image Process., vol. 21, no. 11, pp. 3467 –3478, Aug. 2012.

Digital Library

[47]

J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution as sparse representation of raw image patches,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2008, pp. 1–8.

[48]

J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process., vol. 19, no. 11, pp. 2861–2873, Nov. 2010.

Digital Library

[49]

R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in Proc. 7th Int. Conf. Curves Surfaces , 2012, pp. 711–730.

Digital Library

[50]

N. Zhang, J. Donahue, R. Girshick, and T. Darrell, “Part-based R-CNNs for fine-grained category detection,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 834–849.

Cited By

Han CGui YCheng PYou Z(2025)Semantically Guided Efficient Attention Transformer for Face Super-Resolution TasksInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.36409721:1(1-24)Online publication date: 3-Jan-2025
https://dl.acm.org/doi/10.4018/IJSWIS.364097
Momen-Tayefeh MMomen-Tayefeh MGhahramani SHemmatyar A(2025)Channel estimation for Massive MIMO systems aided by intelligent reflecting surface using semi-super resolution GANSignal Processing10.1016/j.sigpro.2024.109710227:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.sigpro.2024.109710
Hu ZSun WChen Z(2025)Lightweight image super-resolution with sliding Proxy Attention NetworkSignal Processing10.1016/j.sigpro.2024.109704227:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.sigpro.2024.109704
Show More Cited By

Index Terms

Image Super-Resolution Using Deep Convolutional Networks
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Microscopic image super resolution using deep convolutional neural networks
Abstract
Recently, deep convolutional neural networks (CNNs) have achieved excellent results in single image super resolution (SISR). Owing to the strength of deep CNNs, it gives promising results compared to state-of-the-art learning based models on ...
Convolutional neural networks for wavelet domain super resolution

Proposed a super resolution method with higher reconstruction accuracy than before.Cast super resolution as a problem of estimating sparse wavelet detail coefficients.Estimated sparse wavelet coefficients using a convolutional neural network (CNN)...
Image super-resolution via deep dilated convolutional networks
2017 IEEE International Conference on Image Processing (ICIP)
Deep learning techniques have been successfully applied in single image super-resolution (SR). Recently, researches have shown that increasing the depth of network can significantly improve SR performance. Very deep networks for SR achieved a large ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Pattern Analysis and Machine Intelligence

IEEE Transactions on Pattern Analysis and Machine Intelligence Volume 38, Issue 2

Feb. 2016

208 pages

ISSN:0162-8828

Issue’s Table of Contents

Copyright © 2015.

Publisher

IEEE Computer Society

United States

Publication History

Published: 01 February 2016

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1,509
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Han CGui YCheng PYou Z(2025)Semantically Guided Efficient Attention Transformer for Face Super-Resolution TasksInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.36409721:1(1-24)Online publication date: 3-Jan-2025
https://dl.acm.org/doi/10.4018/IJSWIS.364097
Momen-Tayefeh MMomen-Tayefeh MGhahramani SHemmatyar A(2025)Channel estimation for Massive MIMO systems aided by intelligent reflecting surface using semi-super resolution GANSignal Processing10.1016/j.sigpro.2024.109710227:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.sigpro.2024.109710
Hu ZSun WChen Z(2025)Lightweight image super-resolution with sliding Proxy Attention NetworkSignal Processing10.1016/j.sigpro.2024.109704227:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.sigpro.2024.109704
Su HLi YXu YFu XLiu S(2025)A review of deep-learning-based super-resolutionPattern Recognition10.1016/j.patcog.2024.110935157:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.patcog.2024.110935
Esmaeilzehi AMirzaei MZaredar HHatzinakos DAhmad M(2025)DJUHNetImage Communication10.1016/j.image.2024.117187129:COnline publication date: 7-Jan-2025
https://dl.acm.org/doi/10.1016/j.image.2024.117187
Gopalakrishnan SChoudhury A(2025)A ‘deep’ review of video super-resolutionImage Communication10.1016/j.image.2024.117175129:COnline publication date: 7-Jan-2025
https://dl.acm.org/doi/10.1016/j.image.2024.117175
Du JWang MWang XYang ZLi XWu X(2025)Reference-based image super-resolution with attention extraction and pooling of residualsThe Journal of Supercomputing10.1007/s11227-024-06587-881:1Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s11227-024-06587-8
Shi SQi NLi YZhu Q(2025)Self-supervised Reference-Based Image Super-Resolution with Conditional Diffusion ModelMultiMedia Modeling10.1007/978-981-96-2064-7_32(439-452)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1007/978-981-96-2064-7_32
Hao CGu H(2024)Semantic Web-Based Image Super-Resolution for Enhanced Steel Surface Defect DetectionInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.35845220:1(1-32)Online publication date: 13-Dec-2024
https://dl.acm.org/doi/10.4018/IJSWIS.358452
Tan JZhou WLin LJumahong H(2024)A Review of Semantic Medical Image Segmentation Based on Different ParadigmsInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.34524620:1(1-25)Online publication date: 21-Jun-2024
https://dl.acm.org/doi/10.4018/IJSWIS.345246
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents