research-article

Differential Frequency Heterodyne Time-of-Flight Imaging for Instantaneous Depth and Velocity Estimation

Authors:

Masatoshi IshikawaAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 42, Issue 1

Article No.: 9, Pages 1 - 13

https://doi.org/10.1145/3546939

Published: 14 September 2022 Publication History

Abstract

In this study, we discuss the imaging of depth and velocity using heterodyne-mode time-of-flight (ToF) cameras. In particular, Doppler ToF (D-ToF) imaging utilizes heterodyne modulation to measure the velocity from the Doppler frequency shift, which uniquely facilitates the instantaneous radial velocity estimation. However, theoretical discussion on D-ToF is limited to orthogonal frequency and sinusoidal waveform modulation. This study extends the formulation of the D-ToF imaging, and proposes an arbitrary-frequency, arbitrary-waveform framework considering a phase-compensated, symmetrical two-dimensional correlation map. With the proposed framework, the optimal heterodyne frequency for frequency decoding is found. A differential frequency sampling and decoding method is then proposed, which computes the frequency and phase from as few as four simultaneously captured images. With an experiment platform we built, it is confirmed that the minimum velocity sensing error is half that of the orthogonal frequency method, and the sensible phase range is approximately 2.5 times larger. The conclusions in this study allow the ToF velocity imaging to be applied at the optimal sample frequencies for a wide range of ToF sensors. This pushes one step further to the practical use of ToF velocity imaging.

References

[1]

2008. Guides to the Expression of Uncertainty in Measurement. Retrieved December 28th, 2021 https://www.iso.org/sites/JCGM/GUM-introduction.htm.

[2]

P. J. Besl and Neil D. McKay. 1992. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 2 (1992), 239–256. DOI:

Digital Library

[3]

Ayush Bhandari, Christopher Barsi, and Ramesh Raskar. 2015. Blind and reference-free fluorescence lifetime estimation via consumer time-of-flight sensors. Optica 2, 11 (Nov. 2015), 965–973. DOI:

[4]

Ayush Bhandari, Achuta Kadambi, Refael Whyte, Christopher Barsi, Micha Feigin, Adrian Dorrington, and Ramesh Raskar. 2014. Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization. Optics Letter 39, 6 (Mar. 2014), 1705–1708. DOI:

[5]

Ayush Bhandari and Ramesh Raskar. 2016. Signal processing for time-of-flight imaging sensors: An introduction to inverse problems in computational 3-D imaging. IEEE Signal Processing Magazine 33, 5 (2016), 45–58. DOI:

[6]

Bernhard Buttgen, M’Hamed-Ali El Mechat, Felix Lustenberger, and Peter Seitz. 2007. Pseudonoise optical modulation for real-time 3-D imaging with minimum interference. IEEE Transactions on Circuits and Systems I: Regular Papers 54, 10 (2007), 2109–2119. DOI:

[7]

Richard M. Conroy, Adrian A. Dorrington, Rainer Künnemeyer, and Michael J. Cree. 2009. Range imager performance comparison in homodyne and heterodyne operating modes. Three-Dimensional Imaging Metrology 7239, (Jan. 2009), 723905. DOI:

[8]

Richard Ferriere, Johann Cussey, and John M. Dudley. 2008. Time-of-flight range detection using low-frequency intensity modulation of a cw laser diode: Application to fiber length measurement. Optical Engineering 47, 9 (2008), 093602.

[9]

R. Grootjans, W. Van der Tempel, D. Van Nieuwenhove, C. de Tandt, and M. Kuijk. 2006. Improved modulation techniques for time-of-flight ranging cameras using pseudo random binary sequences. In 11th Annual Symposium of the IEEE/LEOS Benelux Chapter. 217.

[10]

Mohit Gupta, Andreas Velten, Shree K. Nayar, and Eric Breitbach. 2018. What are optimal coding functions for time-of-flight imaging? ACM Transactions on Graphics 37, 2 (Feb. 2018), Article 13, 18 pages. DOI:

Digital Library

[11]

Felix Heide, Wolfgang Heidrich, Matthias Hullin, and Gordon Wetzstein. 2015. Doppler time-of-flight imaging. ACM Transactions on Graphics. 34, 4 (July 2015), Article 36, 11 pages. DOI:

Digital Library

[12]

Felix Heide, Lei Xiao, Wolfgang Heidrich, and Matthias B. Hullin. 2014a. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14).

Digital Library

[13]

Felix Heide, Lei Xiao, Andreas Kolb, Matthias B. Hullin, and Wolfgang Heidrich. 2014b. Imaging in scattering media using correlation image sensors and sparse convolutional coding. Optics Express 22, 21 (2014), 26338–26350.

[14]

Berthold K. P. Horn and Brian G. Schunck. 1981. Determining optical flow. Artificial Intelligence 17, 1-3 (1981), 185–203.

Digital Library

[15]

Yunpu Hu, Leo Miyashita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2017. Robust 6-DOF motion sensing for an arbitrary rigid body by multi-view laser Doppler measurements. Optics Express 25, 24 (Nov. 2017), 30371–30387. DOI:

[16]

Kensei Jo, Mohit Gupta, and Shree K. Nayar. 2015. SpeDo: 6 DOF ego-motion sensor using speckle defocus imaging. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15).

Digital Library

[17]

Achuta Kadambi, Refael Whyte, Ayush Bhandari, Lee Streeter, Christopher Barsi, Adrian Dorrington, and Ramesh Raskar. 2013. Coded time of flight cameras: Sparse deconvolution to address multipath interference and recover time profiles. ACM Transactions on Graphics (ToG) 32, 6 (2013), 1–10.

Digital Library

[18]

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar. 2009. Looking around the corner using transient imaging. In 2009 IEEE 12th International Conference on Computer Vision. 159–166. DOI:

[19]

Fengqiang Li, Florian Willomitzer, Prasanna Rangarajan, Mohit Gupta, Andreas Velten, and Oliver Cossairt. 2018. SH-ToF: Micro resolution time-of-flight imaging with superheterodyne interferometry. In 2018 IEEE International Conference on Computational Photography (ICCP’18). 1–10. DOI:

[20]

C. A. Lickfold, L. Streeter, M. J. Cree, and J. B. Scott. 2019. Frequency based radial velocity estimation in time-of-flight range imaging. In 2019 International Conference on Image and Vision Computing New Zealand (IVCNZ’19), 1–6. DOI:

[21]

Ce Liu, Jenny Yuen, Antonio Torralba, Josef Sivic, and William T. Freeman. 2008. Sift flow: Dense correspondence across different scenes. In European Conference on Computer Vision. Springer, 28–42.

Digital Library

[22]

Leo Miyashita, Ryota Yonezawa, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2015. 3D motion sensing of any object without prior knowledge. ACM Transactions on Graphics 34, 6 (Oct. 2015), Article 218, 11 pages. DOI:

Digital Library

[23]

Nikhil Naik, Shuang Zhao, Andreas Velten, Ramesh Raskar, and Kavita Bala. 2011. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. In Proceedings of the 2011 SIGGRAPH Asia Conference (SA’11). Association for Computing Machinery, New York, NY, Article 171, 10 pages. DOI:

Digital Library

[24]

Shikhar Shrestha, Felix Heide, Wolfgang Heidrich, and Gordon Wetzstein. 2016. Computational imaging with multi-camera time-of-flight systems. ACM Transactions on Graphics 35, 4 (July 2016), Article 33, 11 pages. DOI:

Digital Library

[25]

Lee Streeter. 2017a. Methods for linear radial motion estimation in time-of-flight range imaging. In Videometrics, Range Imaging, and Applications XIV, Fabio Remondino and Mark R. Shortis (Eds.), Vol. 10332. International Society for Optics and Photonics, SPIE, 70–77. DOI:

[26]

Lee Streeter. 2017b. Stochastic calculus analysis of optical time-of-flight range imaging and estimation of radial motion. Journal of the Optical Society of America A 34, 7 (2017), 1063–1072.

[27]

Lee Streeter. 2020. Frequency estimation from limited samples: Nonlinearizing time-of-flight radial velocity estimation. IEEE Sensors Letters 4, 10 (2020), 1–4. DOI:

[28]

Lee Streeter and Adrian A. Dorrington. 2014. Coded exposure correction of transverse motion in full-field range imaging. Optical Engineering 53, 10 (2014), 102109.

[29]

Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G. Bawendi, and Ramesh Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications 3, 1 (2012), 1–8.

[30]

Refael Whyte, Lee Streeter, Michael J. Cree, and Adrian A. Dorrington. 2015. Application of lidar techniques to time-of-flight range imaging. Applied Optics 54, 33 (Nov. 2015), 9654–9664. DOI:

Cited By

Wang GWu XTu XLiu ZYan J(2024)Unsupervised Video Moment Retrieval with Knowledge-Based Pseudo-Supervision ConstructionACM Transactions on Information Systems10.1145/370122943:1(1-26)Online publication date: 19-Oct-2024
https://dl.acm.org/doi/10.1145/3701229
Wang KLiu HJie LLi ZHu YNie LCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Explicit Granularity and Implicit Scale Correspondence Learning for Point-Supervised Video Moment LocalizationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680774(9214-9223)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680774
Friday SShi YCherivirala YSaragadam VPediredla A(2024)Snapshot Lidar: Fourier Embedding of Amplitude and Phase for Single-Image Depth Reconstruction2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02381(25203-25212)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02381
Show More Cited By

Index Terms

Differential Frequency Heterodyne Time-of-Flight Imaging for Instantaneous Depth and Velocity Estimation
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        3D imaging
  2. Computer graphics
    1. Image manipulation
      1. Computational photography

Recommendations

Antipodal pilot tones in orthogonal frequency division multiplexing systems for carrier frequency offset estimation

The main factor to degrade the orthogonality of OFDM subcarriers is the carrier frequency offset (CFO) linked to the intercarrier interference problem. Hence, it is necessary to precisely estimate and suitably compensate the CFO. In this study, two pilot ...
A time-frequency decision-feedback loop for carrier frequency offset tracking in OFDM systems

Acquisition and tracking are two crucial stages necessary to the carrier frequency synchronization in orthogonal frequency division multiplexing (OFDM) systems. In this letter, by employing the rotation property of OFDM data subcarriers, a simple time-...
Noise insensitive sampling frequency offset estimation methods for DVB-T/H systems
ICUIMC '09: Proceedings of the 3rd International Conference on Ubiquitous Information Management and Communication

A precise estimation and compensation of SFO (Sampling Frequency Offset) is an important issue in OFDM (Orthogonal Frequency Division Multiplexing) system, because sampling frequency mismatch between the transmitter and the receiver dramatically ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 42, Issue 1

February 2023

211 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3555791

Editor:
Carol O'Sullivan
Trinity College Dublin, Ireland

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 September 2022

Online AM: 11 July 2022

Accepted: 25 June 2022

Revised: 04 June 2022

Received: 08 October 2021

Published in TOG Volume 42, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
635
Total Downloads

Downloads (Last 12 months)221
Downloads (Last 6 weeks)26

Reflects downloads up to 07 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang GWu XTu XLiu ZYan J(2024)Unsupervised Video Moment Retrieval with Knowledge-Based Pseudo-Supervision ConstructionACM Transactions on Information Systems10.1145/370122943:1(1-26)Online publication date: 19-Oct-2024
https://dl.acm.org/doi/10.1145/3701229
Wang KLiu HJie LLi ZHu YNie LCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Explicit Granularity and Implicit Scale Correspondence Learning for Point-Supervised Video Moment LocalizationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680774(9214-9223)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680774
Friday SShi YCherivirala YSaragadam VPediredla A(2024)Snapshot Lidar: Fourier Embedding of Amplitude and Phase for Single-Image Depth Reconstruction2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02381(25203-25212)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02381
Guan CZhang ZSong JZhao Y(2024)A novel GM-APD array-based heterodyne imaging detection system supports large array multi-pixel parallel target motion state image detectionOptics and Lasers in Engineering10.1016/j.optlaseng.2024.108430181(108430)Online publication date: Oct-2024
https://doi.org/10.1016/j.optlaseng.2024.108430
Lee JSuess RGupta M(2024)Light-in-Flight for a World-in-MotionComputer Vision – ECCV 202410.1007/978-3-031-72754-2_12(204-220)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72754-2_12
Kim JJarosz WGkioulekas IPediredla A(2023)Doppler Time-of-Flight RenderingACM Transactions on Graphics10.1145/361833542:6(1-18)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618335

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents