research-article

Ensemble denoising for Monte Carlo renderings

Authors:

Ling-Qi YanAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 40, Issue 6

Article No.: 274, Pages 1 - 17

https://doi.org/10.1145/3478513.3480510

Published: 10 December 2021 Publication History

Editorial Notes

The authors have requested minor, non-substantive changes to the Version of Record and, in accordance with ACM policies, a Corrected Version of Record (CVoR) was published on August 16, 2022. The authors provided an incorrect funder ID number in the original published version of record. The CVoR corrects this error. For reference purposes, the VoR may still be accessed via the Supplemental Material section on this page.

Abstract

Various denoising methods have been proposed to clean up the noise in Monte Carlo (MC) renderings, each having different advantages, disadvantages, and applicable scenarios. In this paper, we present Ensemble Denoising, an optimization-based technique that combines multiple individual MC denoisers. The combined image is modeled as a per-pixel weighted sum of output images from the individual denoisers. Computation of the optimal weights is formulated as a constrained quadratic programming problem, where we apply a dual-buffer strategy to estimate the overall MSE. We further propose an iterative solver to overcome practical issues involved in the optimization. Besides nice theoretical properties, our ensemble denoiser is demonstrated to be effective and robust, and outperforms any individual denoiser across dozens of scenes and different levels of sample rates. We also perform a comprehensive analysis on the selection of individual denoisers to be combined, providing important and practical guides for users.

Supplementary Material

ZIP File (a274-zheng.zip)

Supplemental files.

Download
476.30 MB

3480510-vor (3480510-vor.pdf)

Version of Record for "Ensemble denoising for Monte Carlo renderings" by Zheng et al., ACM Transactions on Graphics, Volume 40, Issue 6 (TOG 40:6).

Download
72.52 MB

References

[1]

Sameer Agarwal and Keir Mierle. 2010. Ceres Solver. http://ceres-solver.org.

[2]

Jonghee Back, Binh-Son Hua, Toshiya Hachisuka, and Bochang Moon. 2020. Deep combiner for independent and correlated pixel estimates. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1--12.

Digital Library

[3]

Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017) 36, 4, Article 97 (2017), 97:1--97:14 pages.

Digital Library

[4]

Pablo Bauszat, Martin Eisemann, Elmar Eisemann, and Marcus Magnor. 2015. General and Robust Error Estimation and Reconstruction for Monte Carlo Rendering. Computer Graphics Forum 34, 2 (2015), 597--608. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.12587

Digital Library

[5]

Benedikt Bitterli. 2014. Tungsten Renderer. https://benedikt-bitterli.me/tungsten.html.

[6]

Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.

[7]

Benedikt Bitterli and Wojciech Jarosz. 2019. Selectively Metropolised Monte Carlo Light Transport Simulation. ACM Trans. Graph. 38, 6, Article 153 (Nov. 2019), 10 pages.

Digital Library

[8]

Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, José A Iglesias-Guitián, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novák. 2016. Nonlinearly weighted first-order regression for denoising Monte Carlo renderings. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 107--117.

Digital Library

[9]

Antoni Buades, Bartomeu Coll, and J-M Morel. 2005. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), Vol. 2. IEEE, 60--65.

Digital Library

[10]

Chakravarty R Alla Chaitanya, Anton S Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--12.

Digital Library

[11]

Sean Cooper, Remi Achard, Alex Fry, Robert Molholm, Lars Borg, Lucien Fostier, Anders Langlands, Thomas Mansencal, Steve Agland, Brian Karis, et al. 2017. Retrospective and Enhancements. (2017).

[12]

Kevin Egan, Florian Hecht, Frédo Durand, and Ravi Ramamoorthi. 2011. Frequency analysis and sheared filtering for shadow light fields of complex occluders. ACM Transactions on Graphics (TOG) 30, 2 (2011), 1--13.

Digital Library

[13]

Eduardo S. L. Gastal and Manuel M. Oliveira. 2012. Adaptive Manifolds for Real-Time High-Dimensional Filtering. ACM TOG 31, 4, Article 33 (2012), 33:1--33:13 pages. Proceedings of SIGGRAPH 2012.

Digital Library

[14]

Iliyan Georgiev, Jaroslav Křivánek, Tomáš Davidovič, and Philipp Slusallek. 2012. Light Transport Simulation with Vertex Connection and Merging. ACM Trans. Graph. 31, 6, Article 192 (Nov. 2012), 10 pages.

Digital Library

[15]

Michaël Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Frédo Durand. 2019. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.

Digital Library

[16]

Pascal Grittmann, Iliyan Georgiev, Philipp Slusallek, and Jaroslav Křivánek. 2019. Variance-Aware Multiple Importance Sampling. ACM Trans. Graph. 38, 6, Article 152 (Nov. 2019), 9 pages.

Digital Library

[17]

Gaël Guennebaud, Benoît Jacob, et al. 2010. Eigen v3. http://eigen.tuxfamily.org.

[18]

Jie Guo, Mengtian Li, Quewei Li, Yuting Qiang, Bingyang Hu, Yanwen Guo, and Ling-Qi Yan. 2019. GradNet: unsupervised deep screened poisson reconstruction for gradient-domain rendering. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--13.

Digital Library

[19]

Yuchi Huo and Sung-eui Yoon. 2021. A survey on deep learning-based Monte Carlo denoising. Computational Visual Media 7, 2 (2021), 169--185.

[20]

Intel. 2019. Open Image Denoise. https://www.openimagedenoise.org/.

[21]

Nima Khademi Kalantari and Pradeep Sen. 2013. Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms. Computer Graphics Forum 32, 2pt1 (2013), 93--102.

[22]

Anton S. Kaplanyan and Carsten Dachsbacher. 2013. Adaptive Progressive Photon Mapping. ACM Trans. Graph. 32, 2, Article 16 (April 2013), 13 pages.

Digital Library

[23]

Timothy Keller and Ingram Olkin. 2004. Combining Correlated Unbiased Estimators of the Mean of a Normal Distribution. Lecture Notes-Monograph Series 45 (2004), 218--227. http://www.jstor.org/stable/4356311

[24]

Markus Kettunen, Marco Manzi, Miika Aittala, Jaakko Lehtinen, Frédo Durand, and Matthias Zwicker. 2015. Gradient-Domain Path Tracing. ACM Trans. Graph. 34, 4 (2015).

Digital Library

[25]

Ivo Kondapaneni, Petr Vevoda, Pascal Grittmann, Tomáš Skřivan, Philipp Slusallek, and Jaroslav Křivánek. 2019. Optimal Multiple Importance Sampling. ACM Trans. Graph. 38, 4, Article 37 (July 2019), 14 pages.

Digital Library

[26]

Johannes Kopf, Michael F Cohen, Dani Lischinski, and Matt Uyttendaele. 2007. Joint bilateral upsampling. ACM Transactions on Graphics (ToG) 26, 3 (2007), 96--es.

Digital Library

[27]

Frédéric Lavancier and Paul Rochet. 2016. A general procedure to combine estimators. Computational Statistics & Data Analysis 94 (2016), 175--192.

Digital Library

[28]

Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012. SURE-Based Optimization for Adaptive Sampling and Reconstruction. ACM Trans. Graph. 31, 6, Article 194 (Nov. 2012), 9 pages.

Digital Library

[29]

Dominic Liao-McPherson and Ilya Kolmanovsky. 2020. FBstab: A proximally stabilized semismooth algorithm for convex quadratic programming. Automatica 113 (2020), 108801.

Digital Library

[30]

Weiheng Lin, Beibei Wang, Lu Wang, and Nicolas Holzschuch. 2020a. A detail preserving neural network model for Monte Carlo denoising. Computational Visual Media 6, 2 (2020), 157--168.

[31]

Weiheng Lin, Beibei Wang, Jian Yang, Lu Wang, and Ling-Qi Yan. 2020b. Path-based Monte Carlo Denoising Using a Three-Scale Neural Network. In Computer Graphics Forum. Wiley Online Library.

[32]

Shuai Liu, Ruipeng Gang, Chenghua Li, and Ruixia Song. 2019. Adaptive deep residual network for single image super-resolution. Computational Visual Media 5, 4 (2019), 391--401.

[33]

Michael D. McCool. 1999. Anisotropic Diffusion for Monte Carlo Noise Reduction. ACM Trans. Graph. 18, 2 (April 1999), 171--194.

Digital Library

[34]

Soham Uday Mehta, Brandon Wang, Ravi Ramamoorthi, and Fredo Durand. 2013. Axis-aligned filtering for interactive physically-based diffuse indirect lighting. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1--12.

Digital Library

[35]

Bochang Moon, Jong Yun Jun, JongHyeob Lee, Kunho Kim, Toshiya Hachisuka, and Sung-Eui Yoon. 2013. Robust image denoising using a virtual flash image for Monte Carlo ray tracing. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 139--151.

[36]

Bochang Moon, Steven McDonagh, Kenny Mitchell, and Markus Gross. 2016. Adaptive polynomial rendering. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--10.

Digital Library

[37]

Hisanari Otsu, Shinichi Kinuwaki, and Toshiya Hachisuka. 2018. Supervised Learning of How to Blend Light Transport Simulations. In Monte Carlo and Quasi-Monte Carlo Methods, Art B. Owen and Peter W. Glynn (Eds.). Springer International Publishing, Cham, 409--427.

[38]

Robi Polikar. 2006. Ensemble based systems in decision making. IEEE Circuits and systems magazine 6, 3 (2006), 21--45.

[39]

Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive Rendering with Non-Local Means Filtering. ACM Trans. Graph. 31, 6, Article 195 (Nov. 2012), 11 pages.

Digital Library

[40]

Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust denoising using feature and color information. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 121--130.

[41]

David Saad. 1998. Online algorithms and stochastic approximations. Online Learning 5 (1998), 6--3.

[42]

Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. 2017. Spatiotemporal variance-guided filtering: real-time reconstruction for path-traced global illumination. In Proceedings of High Performance Graphics. 1--12.

Digital Library

[43]

Bartolomeo Stellato, Goran Banjac, Paul Goulart, Alberto Bemporad, and Stephen Boyd. 2020. OSQP: an operator splitting solver for quadratic programs. Mathematical Programming Computation 12, 4 (2020), 637--672.

[44]

Eric Veach. 1997. Robust Monte Carlo methods for light transport simulation. Vol. 1610. Stanford University PhD thesis.

Digital Library

[45]

Eric Veach and Leonidas J Guibas. 1995. Optimally combining sampling techniques for Monte Carlo rendering. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 419--428.

Digital Library

[46]

Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--15.

Digital Library

[47]

Kin-Ming Wong and Tien-Tsin Wong. 2019. Deep residual learning for denoising Monte Carlo renderings. Computational Visual Media 5, 3 (2019), 239--255.

[48]

Stephen J Wright. 2015. Coordinate descent algorithms. Mathematical Programming 151, 1 (2015), 3--34.

Digital Library

[49]

Bing Xu, Junfei Zhang, Rui Wang, Kun Xu, Yong-Liang Yang, Chuan Li, and Rui Tang. 2019. Adversarial Monte Carlo Denoising with Conditioned Auxiliary Feature. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2019) 38, 6 (2019), 224:1--224:12.

Digital Library

[50]

Ling-Qi Yan, Soham Uday Mehta, Ravi Ramamoorthi, and Fredo Durand. 2015. Fast 4D sheared filtering for interactive rendering of distribution effects. ACM Transactions on Graphics (TOG) 35, 1 (2015), 1--13.

Digital Library

[51]

Zheng Zeng, Lu Wang, Bei-Bei Wang, Chun-Meng Kang, and Yan-Ning Xu. 2020. Denoising Stochastic Progressive Photon Mapping Renderings Using a Multi-Residual Network. Journal of Computer Science and Technology 35, 3 (2020), 506--521.

Digital Library

[52]

Theingi Zin, Yusuke Nakahara, Takuro Yamaguchi, and Masaaki Ikehara. 2021. Improved image denoising via RAISR with fewer filters. Computational Visual Media (2021).

[53]

Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and S-E Yoon. 2015. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. In Computer graphics forum, Vol. 34. Wiley Online Library, 667--681.

Digital Library

Cited By

Qiao PWang QHuo YZhai SXie ZHua WBao HLiu T(2024)Neural Kernel Regression for Consistent Monte Carlo DenoisingACM Transactions on Graphics10.1145/368794943:6(1-14)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687949
Tong XHachisuka T(2024)Efficient Image-Space Shape Splatting for Monte Carlo RenderingACM Transactions on Graphics10.1145/368794343:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687943
Sakai HFreude CAuzinger THahn DWimmer M(2024)A Statistical Approach to Monte Carlo DenoisingSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687591(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687591
Show More Cited By

Index Terms

Ensemble denoising for Monte Carlo renderings
1. Computing methodologies
  1. Computer graphics
    1. Rendering
      1. Ray tracing

Recommendations

Kernel-predicting convolutional networks for denoising Monte Carlo renderings

Regression-based algorithms have shown to be good at denoising Monte Carlo (MC) renderings by leveraging its inexpensive by-products (e.g., feature buffers). However, when using higher-order models to handle complex cases, these techniques often overfit ...
Robust deep residual denoising for Monte Carlo rendering
SA '18: SIGGRAPH Asia 2018 Technical Briefs

We propose a Deep Residual Learning based method that consistently outperforms both the state-of-the-art handcrafted denoisers and learning-based methods for single-image Monte Carlo denoising. Unlike the indirect nature of existing learning-based ...
Neural Partitioning Pyramids for Denoising Monte Carlo Renderings
SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings

Recent advancements in hardware-accelerated raytracing made it possible to achieve interactive framerates even for algorithms previously considered offline, such as path tracing. Interactive path tracing pipelines rely heavily on spatiotemporal ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 40, Issue 6

December 2021

1351 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3478513

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 December 2021

Published in TOG Volume 40, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

The National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
431
Total Downloads

Downloads (Last 12 months)101
Downloads (Last 6 weeks)10

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Qiao PWang QHuo YZhai SXie ZHua WBao HLiu T(2024)Neural Kernel Regression for Consistent Monte Carlo DenoisingACM Transactions on Graphics10.1145/368794943:6(1-14)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687949
Tong XHachisuka T(2024)Efficient Image-Space Shape Splatting for Monte Carlo RenderingACM Transactions on Graphics10.1145/368794343:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687943
Sakai HFreude CAuzinger THahn DWimmer M(2024)A Statistical Approach to Monte Carlo DenoisingSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687591(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687591
Lee JLee SYoon MSong B(2024)Real-Time Monte Carlo Denoising With Adaptive Fusion NetworkIEEE Access10.1109/ACCESS.2024.336958812(29154-29165)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3369588
Oh GMoon B(2024)Joint self-attention for denoising Monte Carlo renderingThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-024-03446-840:7(4623-4634)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s00371-024-03446-8
Back JHua BHachisuka TMoon B(2023)Input-Dependent Uncorrelated Weighting for Monte Carlo DenoisingSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618177(1-10)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.1145/3610548.3618177
Balint MWolski KMyszkowski KSeidel HMantiuk R(2023)Neural Partitioning Pyramids for Denoising Monte Carlo RenderingsACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591562(1-11)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591562
Firmino AFrisvad JJensen H(2023)Denoising-Aware Adaptive Sampling for Monte Carlo Ray TracingACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591537(1-11)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591537
Fan JWang BHasan MYang JYan L(2023)Neural Biplane Representation for BTF Rendering and AcquisitionACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591505(1-11)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591505
Han KOdenthal OKim WYoon S(2023)Pixel-wise Guidance for Utilizing Auxiliary Features in Monte Carlo DenoisingProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/35855056:1(1-19)Online publication date: 16-May-2023
https://dl.acm.org/doi/10.1145/3585505
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents