Guided Linear Upsampling
Article No.: 100, Pages 1 - 12
Abstract
Guided upsampling is an effective approach for accelerating high-resolution image processing. In this paper, we propose a simple yet effective guided upsampling method. Each pixel in the high-resolution image is represented as a linear interpolation of two low-resolution pixels, whose indices and weights are optimized to minimize the upsampling error. The downsampling can be jointly optimized in order to prevent missing small isolated regions. Our method can be derived from the color line model and local color transformations. Compared to previous methods, our method can better preserve detail effects while suppressing artifacts such as bleeding and blurring. It is efficient, easy to implement, and free of sensitive parameters. We evaluate the proposed method with a wide range of image operators, and show its advantages through quantitative and qualitative analysis. We demonstrate the advantages of our method for both interactive image editing and real-time high-resolution video processing. In particular, for interactive editing, the joint optimization can be precomputed, thus allowing for instant feedback without hardware acceleration.
Supplementary Material
References
[1]
Eirikur Agustsson and Radu Timofte. 2017. NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
[2]
Codruta O Ancuti, Cosmin Ancuti, Radu Timofte, Luc Van Gool, Lei Zhang, and Ming-Hsuan Yang. 2019. Ntire 2019 image dehazing challenge report. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 0--0.
[3]
Mathieu Aubry, Sylvain Paris, Samuel W Hasinoff, Jan Kautz, and Frédo Durand. 2014. Fast local laplacian filters: Theory and applications. ACM Transactions on Graphics (TOG) 33, 5 (2014), 1--14.
[4]
Jonathan T Barron, Andrew Adams, YiChang Shih, and Carlos Hernández. 2015. Fast bilateral-space stereo for synthetic defocus. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4466--4474.
[5]
Adrien Bousseau, Sylvain Paris, and Frédo Durand. 2009. User-assisted intrinsic images. In ACM SIGGRAPH Asia 2009 papers. 1--10.
[6]
Lucy Chai, Michaël Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. 2022. Any-Resolution Training For High-Resolution Image Synthesis. In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part XVI. Springer-Verlag, Berlin, Heidelberg, 170--188.
[7]
Jiawen Chen, Andrew Adams, Neal Wadhwa, and Samuel W Hasinoff. 2016. Bilateral guided upsampling. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1--8.
[8]
Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. 2013. KNN matting. IEEE transactions on pattern analysis and machine intelligence 35, 9 (2013), 2175--2188.
[9]
Yutong Dai, Hao Lu, and Chunhua Shen. 2021. Learning Affinity-Aware Upsampling for Deep Image Matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6841--6850.
[10]
Frédo Durand and Julie Dorsey. 2002. Fast bilateral filtering for the display of high-dynamic-range images. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 257--266.
[11]
Zeev Farbman, Raanan Fattal, Dani Lischinski, and Richard Szeliski. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Transactions on Graphics (TOG) 27, 3 (2008), 1--10.
[12]
Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand. 2017. Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--12.
[13]
Michaël Gharbi, YiChang Shih, Gaurav Chaurasia, Jonathan Ragan-Kelley, Sylvain Paris, and Frédo Durand. 2015. Transform recipes for efficient cloud photo enhancement. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1--12.
[14]
Kaiming He, Jian Sun, and Xiaoou Tang. 2012. Guided image filtering. IEEE transactions on pattern analysis and machine intelligence 35, 6 (2012), 1397--1409.
[15]
Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Transactions on Graphics (ToG) 35, 4 (2016), 1--11.
[16]
Michael Kazhdan and Hugues Hoppe. 2008. Streaming multigrid for gradient-domain operations on large images. ACM Transactions on graphics (TOG) 27, 3 (2008), 1--10.
[17]
Heewon Kim, Myungsub Choi, Bee Lim, and Kyoung Mu Lee. 2018. Task-aware image downscaling. In Proceedings of the European Conference on Computer Vision (ECCV). 399--414.
[18]
Johannes Kopf, Michael F Cohen, Dani Lischinski, and Matt Uyttendaele. 2007. Joint bilateral upsampling. ACM Transactions on Graphics (ToG) 26, 3 (2007), 96--es.
[19]
Johannes Kopf, Ariel Shamir, and Pieter Peers. 2013. Content-adaptive image down-scaling. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1--8.
[20]
Andreas Koschan and Mongi Abidi. 2005. Detection and classification of edges in color images. IEEE Signal Processing Magazine 22, 1 (2005), 64--73.
[21]
Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization using optimization. In ACM SIGGRAPH 2004 Papers. 689--694.
[22]
Anat Levin, Dani Lischinski, and Yair Weiss. 2007. A closed-form solution to natural image matting. IEEE transactions on pattern analysis and machine intelligence 30, 2 (2007), 228--242.
[23]
Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, and Dan Feng. 2017. Aod-net: All-in-one dehazing network. In Proceedings of the IEEE international conference on computer vision. 4770--4778.
[24]
Ping Li, Hanqiu Sun, Jianbing Shen, and Chen Huang. 2012. HDR image rerendering using GPU-based processing. International Journal of Image and Graphics 12, 01 (2012), 1250007.
[25]
Yijun Li, Ming-Yu Liu, Xueting Li, Ming-Hsuan Yang, and Jan Kautz. 2018. A closed-form solution to photorealistic image stylization. In Proceedings of the European Conference on Computer Vision (ECCV). 453--468.
[26]
Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira Kemelmacher-Shlizerman. 2021. Real-time high-resolution background matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8762--8771.
[27]
Hengsheng Liu and Jianbing Shen. 2011. Tone mapping using intensity layer decomposition-based fast trilateral filter. Journal of Computer-Aided Design & Computer Graphics 23, 1 (2011), 85--90.
[28]
Davide Mazzini. 2018. Guided upsampling network for real-time semantic segmentation. arXiv preprint arXiv:1807.07466 (2018).
[29]
Dat Ngo, Seungmin Lee, and Bongsoon Kang. 2020. Nonlinear Unsharp Masking Algorithm. In 2020 International Conference on Electronics, Information, and Communication (ICEIC). IEEE, 1--6.
[30]
A Cengiz Oeztireli and Markus Gross. 2015. Perceptually based downscaling of images. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1--10.
[31]
Jinshan Pan, Jiangxin Dong, Jimmy S Ren, Liang Lin, Jinhui Tang, and Ming-Hsuan Yang. 2019. Spatially variant linear representation models for joint filtering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1702--1711.
[32]
Sylvain Paris and Frédo Durand. 2006. A fast approximation of the bilateral filter using a signal processing approach. In European conference on computer vision. Springer, 568--580.
[33]
Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, and Richard Zhang. 2020. Swapping Autoencoder for Deep Image Manipulation. In Advances in Neural Information Processing Systems.
[34]
Jonathan Ragan-Kelley, Andrew Adams, Sylvain Paris, Marc Levoy, Saman Amarasinghe, and Frédo Durand. 2012. Decoupling algorithms from schedules for easy optimization of image processing pipelines. ACM Transactions on Graphics (TOG) 31, 4 (2012), 1--12.
[35]
Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. 2013. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. Acm Sigplan Notices 48, 6 (2013), 519--530.
[36]
Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott. 2009. A perceptually motivated online benchmark for image matting. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1826--1833.
[37]
Zenglin Shi, Yunlu Chen, Efstratios Gavves, Pascal Mettes, and Cees GM Snoek. 2021. Unsharp Mask Guided Filtering. IEEE Transactions on Image Processing 30 (2021), 7472--7485.
[38]
Wanjie Sun and Zhenzhong Chen. 2020. Learned image downscaling for upscaling using content adaptive resampler. IEEE Transactions on Image Processing 29 (2020), 4027--4040.
[39]
Zachary Teed and Jia Deng. 2020. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision. Springer, 402--419.
[40]
Carlo Tomasi and Roberto Manduchi. 1998. Bilateral filtering for gray and color images. In Sixth international conference on computer vision (IEEE Cat. No. 98CH36271). IEEE, 839--846.
[41]
Tobi Vaudrey and Reinhard Klette. 2009. Fast trilateral filtering. In International Conference on Computer Analysis of Images and Patterns. Springer, 541--548.
[42]
Jue Wang and Michael F Cohen. 2007. Optimized color sampling for robust matting. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1--8.
[43]
Nicolas Weber, Michael Waechter, Sandra C Amend, Stefan Guthe, and Michael Goesele. 2016. Rapid, detail-preserving image downscaling. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1--6.
[44]
Hao Wu and Dan Xu. 2009. Improved Poisson image editing and its implementation on GPU. In 2009 IEEE 10th International Conference on Computer-Aided Industrial Design & Conceptual Design. IEEE, 1044--1048.
[45]
Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. 2018. Fast end-to-end trainable guided filter. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1838--1847.
[46]
Xide Xia, Tianfan Xue, Wei-sheng Lai, Zheng Sun, Abby Chang, Brian Kulis, and Jiawen Chen. 2021. Real-time localized photorealistic video style transfer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1089--1098.
[47]
Xide Xia, Meng Zhang, Tianfan Xue, Zheng Sun, Hui Fang, Brian Kulis, and Jiawen Chen. 2020. Joint bilateral learning for real-time universal photorealistic style transfer. In European Conference on Computer Vision. Springer, 327--342.
[48]
Mingqing Xiao, Shuxin Zheng, Chang Liu, Yaolong Wang, Di He, Guolin Ke, Jiang Bian, Zhouchen Lin, and Tie-Yan Liu. 2020. Invertible image rescaling. In European Conference on Computer Vision. Springer, 126--144.
[49]
Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. 2011. Image smoothing via L 0 gradient minimization. In Proceedings of the 2011 SIGGRAPH Asia conference. 1--12.
[50]
Hui Yin, Yuanhao Gong, and Guoping Qiu. 2019. Side window filtering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8758--8766.
[51]
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkss. In Computer Vision (ICCV), 2017 IEEE International Conference on.
Index Terms
- Guided Linear Upsampling
Index terms have been assigned to the content through auto-classification.
Recommendations
Bilateral guided upsampling
We present an algorithm to accelerate a large class of image processing operators. Given a low-resolution reference input and output pair, we model the operator by fitting local curves that map the input to the output. We can then produce a full-...
RGB-Guided Hyperspectral Image Upsampling
ICCV '15: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)Hyperspectral imaging usually lack of spatial resolution due to limitations of hardware design of imaging sensors. On the contrary, latest imaging sensors capture a RGB image with resolution of multiple times larger than a hyperspectral image. In this ...
Denoising and Guided Upsampling of Monte Carlo Path Traced Low Resolution Renderings
SIGGRAPH '22: ACM SIGGRAPH 2022 PostersMonte Carlo path tracing generates renderings by estimating the rendering equation using the Monte Carlo method. Studies focus on rendering a noisy image at the original resolution with a low sample per pixel count to decrease the rendering time. Image-...
Comments
Information & Contributors
Information
Published In
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 July 2023
Published in TOG Volume 42, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Key R&D Program of China
- National Natural Science Foundation of China
- Center-initiated Research Project of Zhejiang Lab
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 246Total Downloads
- Downloads (Last 12 months)125
- Downloads (Last 6 weeks)21
Reflects downloads up to 14 Jan 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in