article

Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing

Authors:

Ashok Veeraraghavan,

Jack TumblinAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 26, Issue 3

Pages 69 - es

https://doi.org/10.1145/1276377.1276463

Published: 29 July 2007 Publication History

Abstract

We describe a theoretical framework for reversibly modulating 4D light fields using an attenuating mask in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera image without any additional refractive elements as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the rays on the 2D sensor. Our mask-equipped camera focuses just as a traditional camera to capture conventional 2D photos at full sensor resolution, but the raw pixel values also hold a modulated 4D light field. The light field can be recovered by rearranging the tiles of the 2D Fourier transform of sensor values into 4D planes, and computing the inverse Fourier transform. In addition, one can also recover the full resolution image information for the in-focus parts of the scene.

We also show how a broadband mask placed at the lens enables us to compute refocused images at full sensor resolution for layered Lambertian scenes. This partial encoding of 4D ray-space data enables editing of image contents by depth, yet does not require computational recovery of the complete 4D light field.

Supplementary Material

JPG File (pps069.jpg)

Download
15.58 KB

MP4 File (pps069.mp4)

Download
55.87 MB

References

[1]

Accorsi, R., Gasparini, F., and Lanza, R. C. 2001. Optimal coded aperture patterns for improved SNR in nuclear medicine imaging. Nuclear Instruments and Methods in Physics Research A 474 (Dec.), 273--284.

[2]

Adelson, T., and Wang, J. 1992. Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Machine Intell. 14, 99--106.

Digital Library

[3]

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graph. 23, 3, 294--302.

Digital Library

[4]

Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization using graph cuts. IEEE Trans. Pattern Anal. Machine Intell. 23, 1222--1239.

Digital Library

[5]

Chaudhuri, S., and Rajagopalan, A. 1999. Depth from Defocus: A Real Aperture Imaging Approach. Springer.

[6]

Dowski, E. R., and Cathey, W. 1995. Extended depth of field through wavefront coding. Appl. Optics 34, 11 (Apr.), 1859--1866.

[7]

Dowski, E. R., and Johnson, G. E. 1999. Wavefront coding: A modern method of achieving high performance and/or low cost imaging systems. In SPIE Annual Meeting.

[8]

Farid, H., and Simoncelli, E. 1998. Range estimation by optical differentiation. J. Opt. Soc. of America A 15, 1, 1777--1786.

[9]

Fergus, R., Torralba, A., and Freeman, W. 2006. Random lens imaging. Tech. rep., MIT.

[10]

Fessenden, R. 1908. Wireless telephony. Trans. American Institute of Electrical Engineers 27, 553--629.

[11]

Field, D. 1994. What is the goal of sensory coding? Neural Comput. 6, 559--601.

Digital Library

[12]

Georgiev, T., Zheng, C., Nayar, S., Curless, B., Salasin, D., and Intwala, C. 2006. Spatio-angular resolution trade-offs in integral photography. In Eurographics Symposium on Rendering, 263--272.

[13]

Gortler, S., Grzeszczuk, R., Szeliski, R., and Cohen, M. 1996. The lumigraph. In SIGGRAPH, 43--54.

Digital Library

[14]

Gottesman, S. R., and Fenimore, E. E. 1989. New family of binary arrays for coded aperture imaging. Appl. Optics 28, 20 (Oct), 4344--4352.

[15]

Haeberli, P., 1994. A multifocus method for controlling depth of field. GraficaObscura.

[16]

Hiura, S., and Matsuyama, T. 1998. Depth measurement by the multi-focus camera. In Proc. Conf. Computer Vision and Pattern Recognition, 953--961.

Digital Library

[17]

Imatest. Image quality evaluation software. http://www.imatest.com/.

[18]

Isaksen, A., McMillan, L., and Gortler, S. 2000. Dynamically reparameterized light fields. In SIGGRAPH, 297--306.

Digital Library

[19]

Ives, H. 1928. Camera for making parallax panoramagrams. J. Opt. Soc. Amer. 17, 435--439.

[20]

Javidi, B., and Okano, F., Eds. 2002. Three-Dimensional Television, Video and Display Technologies. Springer-Verlag.

Digital Library

[21]

Johnson, G. E., Dowski, E. R., and Cathey, W. T. 2000. Passive ranging through wave-front coding: Information and application. Applied Optics 39, 1700--1710.

[22]

Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH 96, 31--42.

Digital Library

[23]

Levoy, M., Chen, B., Vaish, V., Horowitz, M., McDowall, M., and Bolas, M. 2004. Synthetic aperture confocal imaging. ACM Trans. Graph. 23, 825--834.

Digital Library

[24]

Lippmann, G. 1908. Epreuves reversible donnant la sensation du relief. J. Phys 7, 821--825.

[25]

Lucy, L. 1974. An iterative technique for the rectification of observed distributions. J. Astronomy 79, 745--754.

[26]

Martnez-Corral, M., Javidi, B., Martnez-Cuenca, R., and Saavedra, G. 2004. Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays. Applied Optics 43, 5806--5813.

[27]

Nayar, S., and Mitsunaga, T. 2000. High dynamic range imaging: spatially varying pixel exposures. In Proc. Conf. Computer Vision and Pattern Recognition, vol. 1, 472--479.

[28]

Nayar, S. K., Branzoi, V., and Boult, T. E. 2006. Programmable imaging: Towards a flexible camera. Int'l J. Computer Vision 70, 1, 7--22.

Digital Library

[29]

Ng, R., Levoy, M., Brdif, M., Duval, G., Horowitz, M., and Hanrahan, P. 2005. Light field photography with a hand-held plenoptic camera. Tech. rep., Stanford Univ.

[30]

Ng, R. 2005. Fourier slice photography. ACM Trans. Graph. 24, 735--744.

Digital Library

[31]

Okano, F., Hoshino, H., and Yuyama, A. 1997. Real-time pickup method for a three-dimensional image based on integral photography. Applied Optics 36, 15981603.

[32]

Okano, F., Arai, J., Hoshino, H., and Yuyama, I. 1999. Three dimensional video system based on integral photography. Optical Engineering 38, 1072--1077.

[33]

Oppenheim, A. V., Schafer, R. W., and Buck, J. R. 1999. Discrete-Time Signal Processing. Prentice-Hall.

Digital Library

[34]

Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph. 25, 3, 795--804.

Digital Library

[35]

Richardson, W. 1972. Bayesian-based iterative method of image restoration. J. Opt. Soc. of Am. 62, 1, 55--59.

[36]

Skinner, G. K. 1988. X-Ray Imaging with Coded Masks. Scientific American 259 (Aug.), 84.

[37]

Sun, W., and Barbastathis, G. 2005. Rainbow volume holographic imaging. Opt. Lett. 30, 976--978.

[38]

Vaish, V., Wilburn, B., Joshi, N., and Levoy, M. 2004. Using plane + parallax for calibrating dense camera arrays. In Proc. Conf. Computer Vision and Pattern Recognition, 2--9.

[39]

van der Gracht, J., Dowski, E., Taylor, M., and Deaver, D. 1996. Broadband behavior of an optical-digital focus-invariant system. Optics Letters 21, 13 (July), 919--921.

[40]

Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., and Tumblin, J. 2007. Non-refractive modulators for coding and capturing scene appearance. Tech. Rep. UMIACS-TR-2007-21, Univ. of Maryland.

[41]

Wang, S., and Heidrich, W. 2004. The design of an inexpensive very high resolution scan camera system. Eurographics 23, 441--450.

[42]

Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. 24, 3, 765--776.

Digital Library

[43]

Yang, J. C. 2000. A Light Field Camera For Image Based Rendering. Master's thesis, Massachussettes Institute of Technology.

[44]

Zomet, A., and Nayar, S. 2006. Lensless imaging with a controllable aperture. In Proc. Conf. Computer Vision and Pattern Recognition, 339--346.

Digital Library

Cited By

Guindy MKara P(2024)Lessons Learned from Implementing Light Field Camera Animation: Implications, Limitations, Potentials, and Future Research EffortsMultimodal Technologies and Interaction10.3390/mti80800688:8(68)Online publication date: 1-Aug-2024
https://doi.org/10.3390/mti8080068
Singh CHe BFermüller CMetzler CAloimonos Y(2024)Minimal perception: enabling autonomy in resource-constrained robotsFrontiers in Robotics and AI10.3389/frobt.2024.143182611Online publication date: 18-Sep-2024
https://doi.org/10.3389/frobt.2024.1431826
Mizuno RTakahashi KYoshida MTsutake CFujii TNagahara H(2024)[Paper] Compressive Acquisition of Light Field Video Using Aperture-Exposure-Coded CameraITE Transactions on Media Technology and Applications10.3169/mta.12.2212:1(22-35)Online publication date: 2024
https://doi.org/10.3169/mta.12.22
Show More Cited By

Index Terms

Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Scene understanding
      2. Image and video acquisition
2. Theory of computation
  1. Randomness, geometry and discrete structures
    1. Computational geometry

Recommendations

Programmable aperture photography: multiplexed light field acquisition

In this paper, we present a system including a novel component called programmable aperture and two associated post-processing algorithms for high-quality light field acquisition. The shape of the programmable aperture can be adjusted and used to ...
Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing
SIGGRAPH '07: ACM SIGGRAPH 2007 papers

We describe a theoretical framework for reversibly modulating 4D light fields using an attenuating mask in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera ...
Dual photography
SIGGRAPH '05: ACM SIGGRAPH 2005 Papers

We present a novel photographic technique called dual photography, which exploits Helmholtz reciprocity to interchange the lights and cameras in a scene. With a video projector providing structured illumination, reciprocity permits us to generate ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 26, Issue 3

July 2007

976 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1276377

Issue’s Table of Contents

Copyright © 2007 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 July 2007

Published in TOG Volume 26, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

423
Total Citations
View Citations
2,448
Total Downloads

Downloads (Last 12 months)62
Downloads (Last 6 weeks)9

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Guindy MKara P(2024)Lessons Learned from Implementing Light Field Camera Animation: Implications, Limitations, Potentials, and Future Research EffortsMultimodal Technologies and Interaction10.3390/mti80800688:8(68)Online publication date: 1-Aug-2024
https://doi.org/10.3390/mti8080068
Singh CHe BFermüller CMetzler CAloimonos Y(2024)Minimal perception: enabling autonomy in resource-constrained robotsFrontiers in Robotics and AI10.3389/frobt.2024.143182611Online publication date: 18-Sep-2024
https://doi.org/10.3389/frobt.2024.1431826
Mizuno RTakahashi KYoshida MTsutake CFujii TNagahara H(2024)[Paper] Compressive Acquisition of Light Field Video Using Aperture-Exposure-Coded CameraITE Transactions on Media Technology and Applications10.3169/mta.12.2212:1(22-35)Online publication date: 2024
https://doi.org/10.3169/mta.12.22
MINAMATA THAMASAKI HKAWASAKI HNAGAHARA HONO S(2024)A Coded Aperture as a Key for Information Hiding Designed by Physics-in-the-Loop OptimizationIEICE Transactions on Information and Systems10.1587/transinf.2023MUP0003E107.D:1(29-38)Online publication date: 1-Jan-2024
https://doi.org/10.1587/transinf.2023MUP0003
Saragadam VHan ZBoominathan VHuang LTan SFröch JBöhringer KBaraniuk RMajumdar AVeeraraghavan A(2024)Foveated thermal computational imaging prototype using all-silicon meta-opticsOptica10.1364/OPTICA.50285711:1(18)Online publication date: 3-Jan-2024
https://doi.org/10.1364/OPTICA.502857
Peng HHuang SZuo CLiu XCai Z(2024)Noise sensitivity analysis of focal scanning light field imagingOptics Express10.1364/OE.53947432:21(37542)Online publication date: 2-Oct-2024
https://doi.org/10.1364/OE.539474
Mo CLiu XTong JXi JYu YCai Z(2024)Lensless light-field imaging using LMIOptics Express10.1364/OE.53902132:22(38112)Online publication date: 7-Oct-2024
https://doi.org/10.1364/OE.539021
Huang SYang QDeng ZYao MZhang ZLiu XLi JPeng JZhong J(2024)Light-field photography using differential high-speed aperture codingApplied Optics10.1364/AO.52033863:11(2939)Online publication date: 4-Apr-2024
https://doi.org/10.1364/AO.520338
Rao CZhong LGuo YLi MZhang LWei K(2024)Astronomical adaptive optics: a reviewPhotoniX10.1186/s43074-024-00118-75:1Online publication date: 1-May-2024
https://doi.org/10.1186/s43074-024-00118-7
Ma JLiu ZXu YYu JWang YLiu Y(2024)A Robust Large-scale Dataset for Assessing Light Field Image Quality2024 9th International Conference on Signal and Image Processing (ICSIP)10.1109/ICSIP61881.2024.10671428(752-756)Online publication date: 12-Jul-2024
https://doi.org/10.1109/ICSIP61881.2024.10671428
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents