research-article

Computational imaging with multi-camera time-of-flight systems

Authors:

Shikhar Shrestha,

Wolfgang Heidrich,

Gordon WetzsteinAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 35, Issue 4

Article No.: 33, Pages 1 - 11

https://doi.org/10.1145/2897824.2925928

Published: 11 July 2016 Publication History

Abstract

Depth cameras are a ubiquitous technology used in a wide range of applications, including robotic and machine vision, human-computer interaction, autonomous vehicles as well as augmented and virtual reality. In this paper, we explore the design and applications of phased multi-camera time-of-flight (ToF) systems. We develop a reproducible hardware system that allows for the exposure times and waveforms of up to three cameras to be synchronized. Using this system, we analyze waveform interference between multiple light sources in ToF applications and propose simple solutions to this problem. Building on the concept of orthogonal frequency design, we demonstrate state-of-the-art results for instantaneous radial velocity capture via Doppler time-of-flight imaging and we explore new directions for optically probing global illumination, for example by de-scattering dynamic scenes and by non-line-of-sight motion detection via frequency gating.

Supplementary Material

MP4 File (a33.mp4)

Download
555.55 MB

References

[1]

Bamji, C., O'Connor, P., Elkhatib, T., Mehta, S., Thompson, B., Prather, L., Snow, D., Akkaya, O., Daniel, A., Payne, A., amd M. Fenton, T. P., and Chan, V. 2015. A 0.13 um CMOS System-on-Chip for a 512 x 424 Time-of-Flight Image Sensor With Multi-Frequency Photo-Demodulation up to 130 MHz and 2 GS/s ADC. IEEE Journal of Solid-State Circuits 50, 1, 303--319.

[2]

Bhandari, A., Kadambi, A., Whyte, R., Barsi, C., Feigin, M., Dorrington, A., and Raskar, R. 2014. Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization. Optics Letters 39, 1705--1708.

[3]

Buades, A., Coll, B., and Morel, J.-M. 2005. A non-local algorithm for image denoising. In Proc. IEEE CVPR, vol. 2.

Digital Library

[4]

Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. In Proc. SIGGRAPH, 425--432.

Digital Library

[5]

Butler, D., Izadi, S., Hilliges, O., Molyneaux, D., Hodges, S., and Kim, D. 2012. Shake'n'sense: reducing interference for overlapping structured light depth cameras. In Proc. ACM UIST, 1933--1936.

Digital Library

[6]

Büttgen, B., and Seitz, P. 2008. Robust optical time-of-flight range imaging based on smart pixel structures. IEEE Trans. Circuits and Systems 55, 6, 1512--1525.

[7]

Carranza, J., Theobalt, C., Magnor, M. A., and Seidel, H.-P. 2003. Free-viewpoint video of human actors. ACM Trans. Graph. (SIGGRAPH) 22, 3, 569--577.

Digital Library

[8]

Castaneda, V., Mateus, D., and Navab, N. 2014. Stereo time-of-flight with constructive interference. IEEE Trans. PAMI 36, 7, 1402--1413.

Digital Library

[9]

Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. In Proc. SIGGRAPH, 145--156.

Digital Library

[10]

Dong, S., Horstmeyer, R., Shiradkar, R., Guo, K., Ou, X., Bian, Z., Xin, H., and Zheng, G. 2014. Aperture-scanning fourier ptychography for 3d refocusing and super-resolution macroscopic imaging. Optics Express 22, 11, 13586--99.

[11]

Dorrington, A., Godbaz, J., Cree, M., Payne, A., and Streeter, L. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In Proc. Electronic Imaging.

[12]

Freedman, D., Krupka, E., Smolin, Y., Leichter, I., and Schmidt, M. 2014. Sra: fast removal of general multipath for tof sensors. In Proc. ECCV.

[13]

Fuchs, S. 2010. Multipath interference compensation in time-of-flight camera images. In Proc. ICPR.

Digital Library

[14]

Gall, J., Ho, H., Izadi, S., Kohli, P., Ren, X., and Yang, R. 2014. Towards solving real-world vision problems with rgb-d cameras. In CVPR Tutorial.

[15]

Gariepy, G., Tonolini, F., and Jonathan Leach, R. H., and Faccio, D. 2016. Detection and tracking of moving objects hidden from view. Nature Photonics Letters 10, 23--26.

[16]

Gokturk, S., Yalcin, H., and Bamji, C. 2004. A time-of-flight depth sensor - system description, issues and solutions. In Proc. CVPR, 35--35.

Digital Library

[17]

Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The lumigraph. In Proc. SIGGRAPH.

Digital Library

[18]

Hansard, M., Lee, S., Choi, O., and Horaud, R. 2012. Time of Flight Cameras: Principles, Methods, and Applications. Springer.

Digital Library

[19]

Heide, F., Hullin, M. B., Gregson, J., and Heidrich, W. 2013. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph. (SIGGRAPH) 32, 4, 45:1--45:10.

Digital Library

[20]

Heide, F., Xiao, L., Heidrich, W., and Hullin, M. B. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proc. CVPR.

Digital Library

[21]

Heide, F., Xiao, L., Kolb, A., Hullin, M. B., and Heidrich, W. 2014. Imaging in scattering media using correlation image sensors and sparse convolutional coding. Optics Express 22, 21, 26338--26350.

[22]

Heide, F., Heidrich, W., Hullin, M., and Wetzstein, G. 2015. Doppler Time-of-Flight Imaging. ACM Trans. Graph. (SIGGRAPH), 4.

Digital Library

[23]

Holloway, J., Salman Asif, M., Sharma, M. K., Matsuda, N., Horstmeyer, R., Cossairt, O., and Veeraraghavan, A. 2015. Toward Long Distance, Sub-diffraction Imaging Using Coherent Camera Arrays. ArXiv 1510.08470.

[24]

Jayasuriya, S., Pediredla, A., Sivaramakrishnan, S., Molnar, A., and Veeraraghavan, A. 2015. Depth fields: Extending light field techniques to time-of-flight imaging. In Proc. 3DV, 1--9.

Digital Library

[25]

Jimenez, D., Pizarro, D., Mazo, M., and Palazuelos, S. 2012. Modelling and correction of multipath interference in time of flight cameras. In Proc. CVPR.

Digital Library

[26]

Kadambi, A., Whyte, R., Bhandari, A., Streeter, L., Barsi, C., Dorrington, A., and Raskar, R. 2013. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. (SIGGRAPH Asia) 32, 6.

Digital Library

[27]

Kadambi, A., Bhandari, A., Whyte, R., Dorrington, A., and Raskar, R. 2014. Demultiplexing Illumination via Low Cost Sensing and Nanosecond Coding. In Proc. ICCP.

[28]

Kim, S. K., Kang, B., Heo, J., Jung, S.-W., and Choi, O. 2014. Photometric stereo-based single time-of-flight camera. Optics Letters 39, 1, 166--169.

[29]

Kirmani, A., Hutchison, T., Davis, J., and Raskar, R. 2009. Looking around the corner using transient imaging. In Proc. ICCV, 159--166.

[30]

Lange, R., and Seitz, P. 2001. Solid-state time-of-flight range camera. IEEE J. Quantum Electronics 37, 3, 390--397.

[31]

Levoy, M., and Hanrahan, P. 1996. Light field rendering. In Proc. SIGGRAPH, 31--42.

Digital Library

[32]

Li, L., Xiang, S., Yang, Y., and Yu, L. 2015. Multi-camera interference cancellation of time-of-flight (tof) cameras. In Proc. IEEE ICIP, 556--560.

[33]

Maimone, A., and Fuchs, H. 2012. Reducing interference between multiple structured light depth sensors using motion. In Proc. VR.

Digital Library

[34]

Matusik, W., Buehler, C., Raskar, R., Gortler, S. J., and McMillan, L. 2000. Image-based visual hulls. In Proc. SIGGRAPH, 369--374.

Digital Library

[35]

McCandless, S. W., and Jackson, C. R. 2004. Principles of synthetic aperture radar. In AR Marine Users Manual, J. Fagerberg, D. C. Mowery, and R. R. Nelson, Eds. NOAA, ch. 1, 11.

[36]

Naik, N., Zhao, S., Velten, A., Raskar, R., and Bala, K. 2011. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH Asia) 30, 6, 171:1--171:10.

Digital Library

[37]

Naik, N., Kadambi, A., Rhemann, C., Izadi, S., Raskar, R., and Kang, S. 2015. A light transport model for mitigating multipath interference in tof sensors. In Proc. CVPR.

[38]

O'Toole, M., Heide, F., Xiao, L., Hullin, M. B., Heidrich, W., and Kutulakos, K. N. 2014. Temporal frequency probing for 5d transient analysis of global light transport. ACM Trans. Graph. (SIGGRAPH) 33, 4, 87:1--87:11.

Digital Library

[39]

Payne, A., Jongenelen, A., Dorrington, A., Cree, M., and Carnegie, D. 2009. Multiple Frequency Range Imaging to Remove Measurement Ambiguity. In Proc. Optical 3-D measurement techniques IX.

[40]

Peters, C., Klein, J., Hullin, M. B., and Klein, R. 2015. Solving trigonometric moment problems for fast transient imaging. ACM Trans. Graph. (SIGGRAPH Asia) 34, 6.

Digital Library

[41]

Rander, P., Narayanan, P. J., and Kanade, T. 1997. Virtualized reality: Constructing time-varying virtual worlds from real events. In Proc. IEEE Visualization, 277--283.

Digital Library

[42]

Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. 2011. Real-time human pose recognition in parts from single depth images. In Proc. CVPR.

Digital Library

[43]

Tadano, R., Pediredla, A. K., and Veeraraghavan, A. 2015. Depth selective camera: A direct, on-chip, programmable technique for depth selectivity in photography. In Proc. IEEE ICCV.

Digital Library

[44]

Ti, C., Yang, R., Davis, J., and Pan, Z. 2015. Simultaneous Time-of-Flight Sensing and Photometric Stereo With a Single ToF Sensor. In Proc. CVPR.

[45]

Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M., and Raskar, R. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat Commun 745, 3.

[46]

Velten, A., Wu, D., Jarabo, A., Masia, B., Barsi, C., Joshi, C., Lawson, E., Bawendi, M., Gutierrez, D., and Raskar, R. 2013. Femto-photography: Capturing and visualizing the propagation of light. ACM Trans. Graph. (SIGGRAPH) 32, 4, 44:1--44:8.

Digital Library

[47]

Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. (SIGGRAPH) 24, 3, 765--776.

Digital Library

[48]

Woodham, R. J. 1980. Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1.

[49]

Wu, D., Wetzstein, G., Barsi, C., Willwacher, T., O'Toole, M., Naik, N., Dai, Q., Kutulakos, K., and Raskar, R. 2012. Frequency analysis of transient light transport with applications in bare sensor imaging. In Proc. ECCV.

Digital Library

Cited By

Wang JXie ZMao PSun MGuo J(2024)Fruit modeling and application based on 3D imaging technology: a reviewJournal of Food Measurement and Characterization10.1007/s11694-024-02480-318:6(4120-4136)Online publication date: 12-Mar-2024
https://doi.org/10.1007/s11694-024-02480-3
Rodriguez BZhang XRajan D(2023)Probabilistic Modeling of Multicamera Interference for Time-of-Flight SensorsSensors10.3390/s2319804723:19(8047)Online publication date: 23-Sep-2023
https://doi.org/10.3390/s23198047
Baek SWalsh NChugunov IShi ZHeide F(2023)Centimeter-wave Free-space Neural Time-of-Flight ImagingACM Transactions on Graphics10.1145/352267142:1(1-18)Online publication date: 3-Mar-2023
https://dl.acm.org/doi/10.1145/3522671
Show More Cited By

Index Terms

Computational imaging with multi-camera time-of-flight systems
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        3D imaging
        Computational photography

Recommendations

Doppler time-of-flight imaging

Over the last few years, depth cameras have become increasingly popular for a range of applications, including human-computer interaction and gaming, augmented reality, machine vision, and medical imaging. Many of the commercially-available devices use ...
Panorama light-field imaging
SIGGRAPH '12: ACM SIGGRAPH 2012 Posters

We present a first approach towards panorama light-field imaging. By converting overlapping sub-light-fields into individual focal stacks, computing a panoramic focal stack from them, and converting the panoramic focal stack back into a panoramic light ...
Panorama light-field imaging
SIGGRAPH '12: ACM SIGGRAPH 2012 Talks

We present a first approach towards panorama light-field imaging. By converting overlapping sub-light-fields into individual focal stacks, computing a panoramic focal stack from them, and converting the panoramic focal stack back into a panoramic light ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 35, Issue 4

July 2016

1396 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2897824

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2016

Published in TOG Volume 35, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Intuitive Surgical
KAUST Office of Sponsored Research
National Science Foundation
Intel Partnership on Visual and Experiential Computing

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

63
Total Citations
View Citations
1,688
Total Downloads

Downloads (Last 12 months)91
Downloads (Last 6 weeks)6

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang JXie ZMao PSun MGuo J(2024)Fruit modeling and application based on 3D imaging technology: a reviewJournal of Food Measurement and Characterization10.1007/s11694-024-02480-318:6(4120-4136)Online publication date: 12-Mar-2024
https://doi.org/10.1007/s11694-024-02480-3
Rodriguez BZhang XRajan D(2023)Probabilistic Modeling of Multicamera Interference for Time-of-Flight SensorsSensors10.3390/s2319804723:19(8047)Online publication date: 23-Sep-2023
https://doi.org/10.3390/s23198047
Baek SWalsh NChugunov IShi ZHeide F(2023)Centimeter-wave Free-space Neural Time-of-Flight ImagingACM Transactions on Graphics10.1145/352267142:1(1-18)Online publication date: 3-Mar-2023
https://dl.acm.org/doi/10.1145/3522671
Lee JGupta M(2023)Mitigating AC and DC Interference in Multi-ToF-Camera EnvironmentsIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.330756445:12(15005-15017)Online publication date: Dec-2023
https://doi.org/10.1109/TPAMI.2023.3307564
Qiao XGe CDeng PWei HPoggi MMattoccia S(2023)Depth Restoration in Under-Display Time-of-Flight ImagingIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.320990545:5(5668-5683)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TPAMI.2022.3209905
Suo JZhang WGong JYuan XBrady DDai Q(2023)Computational Imaging and Artificial Intelligence: The Next Revolution of Mobile VisionProceedings of the IEEE10.1109/JPROC.2023.3338272111:12(1607-1639)Online publication date: Dec-2023
https://doi.org/10.1109/JPROC.2023.3338272
Wu DYang ZZhang PWang RYang BMa X(2023)Virtual-Reality Interpromotion Technology for Metaverse: A SurveyIEEE Internet of Things Journal10.1109/JIOT.2023.326584810:18(15788-15809)Online publication date: 15-Sep-2023
https://doi.org/10.1109/JIOT.2023.3265848
Tong YChen JLeng ZLiu BWang Y(2023) -ToF: A feature-alignment and frequency-division time-of-flight data denoise network Computer Communications10.1016/j.comcom.2023.04.033207(66-76)Online publication date: Jul-2023
https://doi.org/10.1016/j.comcom.2023.04.033
Züfle AGao SAnderson T(2022)Introduction to the Special Issue on Understanding the Spread of COVID-19, Part 2ACM Transactions on Spatial Algorithms and Systems10.1145/35686698:4(1-5)Online publication date: 26-Nov-2022
https://dl.acm.org/doi/10.1145/3568669
Bernardin ACoevoet EKry PAndrews SDuriez CMarchal M(2022)Constraint-based Simulation of Passive Suction CupsACM Transactions on Graphics10.1145/355188942:1(1-14)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1145/3551889
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents