research-article

Blind image super-resolution with spatially variant degradations

Authors:

Victor Cornillère,

Abdelaziz Djelouah,

Olga Sorkine-Hornung,

Christopher SchroersAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 38, Issue 6

Article No.: 166, Pages 1 - 13

https://doi.org/10.1145/3355089.3356575

Published: 08 November 2019 Publication History

Abstract

Existing deep learning approaches to single image super-resolution have achieved impressive results but mostly assume a setting with fixed pairs of high resolution and low resolution images. However, to robustly address realistic upscaling scenarios where the relation between high resolution and low resolution images is unknown, blind image super-resolution is required. To this end, we propose a solution that relies on three components: First, we use a degradation aware SR network to synthesize the HR image given a low resolution image and the corresponding blur kernel. Second, we train a kernel discriminator to analyze the generated high resolution image in order to predict errors present due to providing an incorrect blur kernel to the generator. Finally, we present an optimization procedure that is able to recover both the degradation kernel and the high resolution image by minimizing the error predicted by our kernel discriminator. We also show how to extend our approach to spatially variant degradations that typically arise in visual effects pipelines when compositing content from different sources and how to enable both local and global user interaction in the upscaling process.

Supplementary Material

ZIP File (a166-cornillere.zip)

Supplemental files.

Download
1.58 MB

References

[1]

Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. 2010. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence 33, 5 (2010), 898--916.

[2]

Isabelle Begin and FR Ferrie. 2004. Blind super-resolution using a learning-based approach. In ICPR.

[3]

Isabelle Begin and Frank P Ferrie. 2007. PSF recovery from examples for blind super-resolution. In ICIP.

[4]

Siavash Arjomand Bigdeli, Matthias Zwicker, Paolo Favaro, and Meiguang Jin. 2017. Deep mean-shift priors for image restoration. In Advances in Neural Information Processing Systems.

[5]

Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Real-time video super-resolution with spatiotemporal networks and motion compensation. In CVPR.

[6]

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. In ECCV.

[7]

Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Accelerating the super-resolution convolutional neural network. In ECCV.

[8]

Netalee Efrat, Daniel Glasner, Alexander Apartsin, Boaz Nadler, and Anat Levin. 2013. Accurate blur models vs. image priors in single image super-resolution. In ICCV.

[9]

Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. 2019. Blind super-resolution with iterative kernel correction. In CVPR.

[10]

Neel Joshi, Richard Szeliski, and David J Kriegman. 2008. PSF estimation using sharp edge prediction. In CVPR.

[11]

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In CVPR.

[12]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[13]

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR.

[14]

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In CVPR Workshops.

[15]

Tomer Michaeli and Michal Irani. 2013. Nonparametric blind super-resolution. In ICCV.

[16]

Kamal Nasrollahi and Thomas B Moeslund. 2014. Super-resolution: a comprehensive survey. Machine vision and applications (2014).

[17]

Jianping Qiao, Ju Liu, and Caihua Zhao. 2006. A novel SVM-based blind super-resolution algorithm. In The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[18]

JH Rick Chang, Chun-Liang Li, Barnabas Poczos, BVK Vijaya Kumar, and Aswin C Sankaranarayanan. 2017. One Network to Solve Them All-Solving Linear Inverse Problems Using Deep Projection Models. In ICCV.

[19]

Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-recurrent video super-resolution. In CVPR.

[20]

Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR.

[21]

Assaf Shocher, Nadav Cohen, and Michal Irani. 2018. "Zero-Shot" Super-Resolution using Deep Internal Learning. In CVPR.

[22]

Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. 2017. Ntire 2017 challenge on single image super-resolution: Methods and results. In CVPR Workshops.

[23]

Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine-Hornung, Olga Sorkine-Hornung, and Christopher Schroers. 2018. A Fully Progressive Approach to Single-Image Super-Resolution. (2018).

[24]

Chih-Yuan Yang, Chao Ma, and Ming-Hsuan Yang. 2014. Single-image super-resolution: A benchmark. In ECCV.

[25]

Roman Zeyde, Michael Elad, and Matan Protter. 2010. On single image scale-up using sparse-representations. In International conference on curves and surfaces. Springer, 711--730.

[26]

Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017. Learning deep CNN denoiser prior for image restoration. In CVPR.

[27]

Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2018d. Learning a single convolutional super-resolution network for multiple degradations. In CVPR.

[28]

Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2019. Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernels. CVPR (2019).

[29]

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018b. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.

[30]

Xinyi Zhang, Hang Dong, Zhe Hu, Wei-Sheng Lai, Fei Wang, and Ming-Hsuan Yang. 2018a. Gated Fusion Network for Joint Image Deblurring and Super-Resolution. In BMVC.

[31]

Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018c. Image super-resolution using very deep residual channel attention networks. In ECCV.

Cited By

Chen DZhang ZLiang JZhang LCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)SSL: A Self-similarity Loss for Improving Generative Image Super-resolutionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680874(3189-3198)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680874
Neshatavar RYavartanoo MSon SLee K(2024)ICF-SRSR: Invertible scale-Conditional Function for Self-Supervised Real-world Single Image Super-Resolution2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00158(1546-1556)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00158
Xia JYang ZLi SZhang SFu YGündüz DLi X(2024)Blind Super-Resolution via Meta-Learning and Markov Chain Monte Carlo SimulationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.340004146:12(8139-8156)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3400041
Show More Cited By

Index Terms

Blind image super-resolution with spatially variant degradations
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography
  2. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

Efficient Blind Image Super-Resolution
Advances in Computational Intelligence
Abstract
A hybrid method to Single Image Super-resolution is proposed. We used zero-shot super-resolution method to reconstruct high-resolution image from low-resolution one based on the degradation trained on unpaired high-resolution and low-resolution ...
Degradation Reconstruction Loss: A Perceptual-Oriented Super-Resolution Framework for Multi-downsampling Degradations
Image and Graphics
Abstract
Recent years have witnessed the great success of deep learning-based single image super-resolution (SISR) methods. However, most of the existing SR methods assume that low-resolution (LR) images are purely bicubic downsampled from high-resolution (...
A Fast Domain Adaptation Network for Image Super-Resolution
Image and Graphics
Abstract
Most previous super-resolution (SR) methods are based on high-resolution (HR) images and corresponding low-resolution (LR) images obtained artificially through bicubic downsampling. However, in real scenes, LR images are usually obtained with ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 38, Issue 6

December 2019

1292 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3355089

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2019

Published in TOG Volume 38, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

66
Total Citations
View Citations
969
Total Downloads

Downloads (Last 12 months)87
Downloads (Last 6 weeks)11

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen DZhang ZLiang JZhang LCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)SSL: A Self-similarity Loss for Improving Generative Image Super-resolutionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680874(3189-3198)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680874
Neshatavar RYavartanoo MSon SLee K(2024)ICF-SRSR: Invertible scale-Conditional Function for Self-Supervised Real-world Single Image Super-Resolution2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00158(1546-1556)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00158
Xia JYang ZLi SZhang SFu YGündüz DLi X(2024)Blind Super-Resolution via Meta-Learning and Markov Chain Monte Carlo SimulationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.340004146:12(8139-8156)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3400041
Mou CWang XWu YShan YZhang J(2024)Empowering Real-World Image Super-Resolution With Flexible Interactive ModulationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.339117746:11(7317-7330)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3391177
Khan AMicheloni CMartinel N(2024)Lightweight Prompt Learning Implicit Degradation Estimation Network for Blind Super ResolutionIEEE Transactions on Image Processing10.1109/TIP.2024.344261333(4556-4567)Online publication date: 19-Aug-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3442613
Wen WGuo CRen WWang HShao X(2024)Adaptive Blind Super-Resolution Network for Spatial-Specific and Spatial-Agnostic DegradationsIEEE Transactions on Image Processing10.1109/TIP.2024.342516933(4404-4418)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3425169
Zhang YLi YZhang ZZhang CHe J(2024)Blind Image Super-Resolution via Domain Translation and Diffusion Models2024 6th International Conference on Natural Language Processing (ICNLP)10.1109/ICNLP60986.2024.10692727(238-244)Online publication date: 22-Mar-2024
https://doi.org/10.1109/ICNLP60986.2024.10692727
Khan AMicheloni CMartinel N(2024)IDENet: Implicit Degradation Estimation Network for Efficient Blind Super Resolution2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00613(6065-6075)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00613
Asensio Ramos A(2024)Solar multiobject multiframe blind deconvolution with a spatially variant convolution neural emulatorAstronomy & Astrophysics10.1051/0004-6361/202449568688(A88)Online publication date: 8-Aug-2024
https://doi.org/10.1051/0004-6361/202449568
Zha BYang SLei JXu ZYe NFeng B(2024)A dual branch attention network based on practical degradation model for face super resolutionScientific Reports10.1038/s41598-024-79695-314:1Online publication date: 14-Nov-2024
https://doi.org/10.1038/s41598-024-79695-3
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents