research-article

iDFusion: Globally Consistent Dense 3D Reconstruction from RGB-D and Inertial Measurements

Authors:

Lu FangAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 962 - 970

https://doi.org/10.1145/3343031.3351085

Published: 15 October 2019 Publication History

Abstract

We present a practical fast, globally consistent and robust dense 3D reconstruction system, iDFusion, by exploring the joint benefit of both the visual (RGB-D) solution and inertial measurement unit (IMU). A global optimization considering all the previous states is adopted to maintain high localization accuracy and global consistency, yet its complexity of being linear to the number of all previous camera/IMU observations seriously impedes real-time implementation. We show that the global optimization can be solved efficiently at the complexity linear to the number of keyframes, and further realize a real-time dense 3D reconstruction system given the estimated camera states. Meanwhile, for the sake of robustness, we propose a novel loop-validity detector based on the estimated bias of the IMU state. By checking the consistency of camera movements, a false loop closure constraint introduces manifest inconsistency between the camera movements and IMU measurements. Experiments reveal that iDFusion owns superior reconstruction performance running in 25 fps on CPU computing of portable devices, under challenging yet practical scenarios including texture-less, motion blur, and repetitive contents.

References

[1]

Andrew J. Davison, “Futuremapping: The computational structure of spatial AI systems,” CoRR, vol. abs/1803.11288, 2018.

[2]

Angela Dai, Matthias Nießner, Michael Zollöfer, Shahram Izadi, and Christian Theobalt, “Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface re-integration,” ACM Transactions on Graphics 2017 (TOG), 2017.

[3]

Sungjoon Choi, Qian-Yi Zhou, and Vladlen Koltun, “Robust reconstruction of indoor scenes,” in Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, 2015, pp. 5556--5565.

[4]

Thomas Whelan, Renato F Salas-Moreno, Ben Glocker, Andrew J Davison, and Stefan Leutenegger, “Elasticfusion: Real-time dense slam and light source estimation,” The International Journal of Robotics Research, vol. 35, no. 14, pp. 1697--1716, 2016.

Digital Library

[5]

Lei Han and Lu Fang, “Flashfusion: Real-time globally consistent dense 3d reconstruction using cpu computing,” RSS, 2018.

[6]

Lei Han, Lan Xu, Dmytro Bobkov, Eckehard Steinbach, and Lu Fang, “Real-time global registration for globally consistent rgb-d slam,” IEEE Transactions on Robotics, 2019.

[7]

Christian Forster, Luca Carlone, Frank Dellaert, and Davide Scaramuzza, “Imu preintegration on manifold for efficient visual-inertial maximum-a-posteriori estimation,” Georgia Institute of Technology, 2015.

[8]

Lei Han and Lu Fang, “Mild: Multi-index hashing for appearance based loop closure detection,” in Multimedia and Expo (ICME), 2017 IEEE International Conference on. IEEE, 2017, pp. 139--144.

[9]

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun, “Fast global registration,” in European Conference on Computer Vision. Springer, 2016, pp. 766--782.

[10]

Brian Curless and Marc Levoy, “A volumetric method for building complex models from range images,” 1996.

Digital Library

[11]

M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger, “Real-time 3d reconstruction at scale using voxel hashing,” ACM Transactions on Graphics (TOG), 2013.

[12]

Raul Mur-Artal and Juan D Tardós, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,” IEEE Transactions on Robotics, vol. 33, no. 5, pp. 1255--1262, 2017.

Digital Library

[13]

Jonathan Ventura, Clemens Arth, Gerhard Reitmayr, and Dieter Schmalstieg, “Global localization from monocular slam on a mobile phone,” IEEE transactions on visualization and computer graphics, vol. 20, no. 4, pp. 531--539, 2014.

[14]

Sebastian Thrun, Wolfram Burgard, and Dieter Fox, Probabilistic robotics, MIT press, 2005.

[15]

Anastasios I Mourikis and Stergios I Roumeliotis, “A multi-state constraint kalman filter for vision-aided inertial navigation,” in Proceedings 2007 IEEE International Conference on Robotics and Automation. IEEE, 2007, pp. 3565--3572.

[16]

Mingyang Li, Byung Hyung Kim, and Anastasios I Mourikis, “Real-time motion tracking on a cellphone using inertial sensing and a rolling-shutter camera,” in 2013 IEEE International Conference on Robotics and Automation. IEEE, 2013, pp. 4712--4719.

[17]

Joel A. Hesch, Dimitrios G. Kottas, Sean L. Bowman, and Stergios I. Roumeliotis, “Observability-constrained vision-aided inertial navigation,” 2012.

[18]

Martin Brossard, Silvere Bonnabel, and Axel Barrau, “Invariant kalman filtering for visual inertial slam,” 2018 21st International Conference on Information Fusion (FUSION), pp. 2021--2028, 2018.

[19]

Tong Qin, Peiliang Li, and Shaojie Shen, “Vins-mono: A robust and versatile monocular visual-inertial state estimator,” IEEE Transactions on Robotics, vol. 34, no. 4, pp. 1004--1020, 2018.

Digital Library

[20]

Stefan Leutenegger, Simon Lynen, Michael Bosse, Roland Siegwart, and Paul Furgale, “Keyframe-based visual--inertial odometry using nonlinear optimization,” The International Journal of Robotics Research, vol. 34, no. 3, pp. 314--334, 2015.

Digital Library

[21]

R. Mur-Artal and J. D. Tardós, “Visual-inertial monocular slam with map reuse,” IEEE Robotics and Automation Letters, vol. 2, no. 2, pp. 796--803, April 2017.

[22]

Ming Hsiao, Eric Westman, and Michael Kaess, “Dense planar-inertial slam with structural constraints,” 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6521--6528, 2018.

[23]

T. Laidlow, M. Bloesch, W. Li, and S. Leutenegger, “Dense rgb-d-inertial slam with map deformations,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Sep. 2017, pp. 6741--6748.

[24]

Zunjie Zhu and Feng Xu, “Real-time indoor scene reconstruction with rgbd and inertia input,” 12 2018.

[25]

O. Kähler, V. Adrian Prisacariu, C. Yuheng Ren, X. Sun, P. Torr, and D. Murray, “Very high frame rate volumetric integration of depth images on mobile devices,” IEEE Transactions on Visualization and Computer Graphics, vol. 21, no. 11, pp. 1241--1250, Nov 2015.

Digital Library

[26]

Matthew Klingensmith, Ivan Dryanovski, Siddhartha Srinivasa, and Jizhong Xiao, “Chisel: Real time large scale 3d reconstruction onboard a mobile device using spatially hashed signed distance fields,” 07 2015.

[27]

C. Feng, Y. Taguchi, and V. R. Kamat, “Fast plane extraction in organized point clouds using agglomerative hierarchical clustering,” in 2014 IEEE International Conference on Robotics and Automation (ICRA), May 2014, pp. 6218--6225.

[28]

Jose Luis Blanco, “A tutorial on se(3) transformation parameterizations and on-manifold optimization,” 09 2010.

[29]

Lingni Ma, Christian Kerl, Jörg Stückler, and Daniel Cremers, “Cpa-slam: Consistent plane-model alignment for direct rgb-d slam,” in Robotics and Automation (ICRA), 2016 IEEE International Conference on. IEEE, 2016, pp. 1285--1291.

Digital Library

[30]

F. Endres, J. Hess, N. Engelhard, J. Sturm, D. Cremers, and W. Burgard, “An evaluation of the rgb-d slam system,” in 2012 IEEE International Conference on Robotics and Automation, May 2012, pp. 1691--1696.

[31]

Chen Wang, Minh-Chung Hoang, L. Xie, and Junsong Yuan, “Non-iterative rgb-d-inertial odometry,” 2017.

[32]

Chen Wang, Junsong Yuan, and Lihua Xie, “Non-iterative slam,” 2017 18th International Conference on Advanced Robotics (ICAR), pp. 83--90, 2017.

[33]

P. Furgale, J. Rehder, and R. Siegwart, “Unified temporal and spatial calibration for multi-sensor systems,” in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Nov 2013, pp. 1280--1286.

[34]

T. Qin, P. Li, and S. Shen, “Vins-mono: A robust and versatile monocular visual-inertial state estimator,” IEEE Transactions on Robotics, vol. 34, no. 4, pp. 1004--1020, Aug 2018.

Digital Library

[35]

J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of rgb-d slam systems,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct 2012, pp. 573--580.

[36]

A. Handa, T. Whelan, J. McDonald, and A. J. Davison, “A benchmark for rgb-d visual odometry, 3d reconstruction and slam,” in 2014 IEEE International Conference on Robotics and Automation (ICRA), May 2014, pp. 1524--1531.

[37]

Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers, “A benchmark for the evaluation of rgb-d slam systems,” in Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, 2012, pp. 573--580.

Cited By

Liu YXiaoqun W(2024)Survey of texture optimization algorithms for 3D reconstructed scenesJournal of Image and Graphics10.11834/jig.23047829:8(2303-2318)Online publication date: 2024
https://doi.org/10.11834/jig.230478
Li XDing YLi RTang ZLi K(2024)MoNeRF: Deformable Neural Rendering for Talking Heads via Latent Motion NavigationComputer Graphics Forum10.1111/cgf.15274Online publication date: 20-Dec-2024
https://doi.org/10.1111/cgf.15274
Fang LFang L(2024)Plenoptic ReconstructionPlenoptic Imaging and Processing10.1007/978-981-97-6915-5_4(75-189)Online publication date: 16-Oct-2024
https://doi.org/10.1007/978-981-97-6915-5_4
Show More Cited By

Index Terms

iDFusion: Globally Consistent Dense 3D Reconstruction from RGB-D and Inertial Measurements
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality

Recommendations

Dense 3D reconstruction combining depth and RGB information

Dense 3D reconstruction has important applications in many fields. The existing depth information based methods are typically constrained in their effective camera-object distance which should be from 0.4m to 4m. We present a novel method that can ...
Depth-map completion for large indoor scene reconstruction
Highlights
- Propose a new depth completion algorithm for MVS depth-maps.
- Use occlusion ...
Abstract
Traditional Multi View Stereo (MVS) algorithms are often difficult to deal with large-scale indoor scene reconstruction, due to the photo-consistency measurement errors in weak textured regions, which are commonly exist in indoor ...
High Quality Photometric Reconstruction Using a Depth Camera
CVPR '14: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition

In this paper we present a depth-guided photometric 3D reconstruction method that works solely with a depth camera like the Kinect. Existing methods that fuse depth with normal estimates use an external RGB camera to obtain photometric information and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
379
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)3

Reflects downloads up to 26 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu YXiaoqun W(2024)Survey of texture optimization algorithms for 3D reconstructed scenesJournal of Image and Graphics10.11834/jig.23047829:8(2303-2318)Online publication date: 2024
https://doi.org/10.11834/jig.230478
Li XDing YLi RTang ZLi K(2024)MoNeRF: Deformable Neural Rendering for Talking Heads via Latent Motion NavigationComputer Graphics Forum10.1111/cgf.15274Online publication date: 20-Dec-2024
https://doi.org/10.1111/cgf.15274
Fang LFang L(2024)Plenoptic ReconstructionPlenoptic Imaging and Processing10.1007/978-981-97-6915-5_4(75-189)Online publication date: 16-Oct-2024
https://doi.org/10.1007/978-981-97-6915-5_4
Cao YShi YCheng ZLi H(2023)End-to-End Point Cloud Registration via Rotation Equivariant Descriptors2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS55552.2023.10342154(5804-5811)Online publication date: 1-Oct-2023
https://doi.org/10.1109/IROS55552.2023.10342154
Han LZhong DLi LZheng KFang L(2022)Learning Residual Color for Novel View SynthesisIEEE Transactions on Image Processing10.1109/TIP.2022.315424231(2257-2267)Online publication date: 2022
https://doi.org/10.1109/TIP.2022.3154242
Gou RChen GPu XZuo T(2022)DR-Fusion: Dynamic SLAM 3D Reconstruction Method of Production Workshop2022 5th International Conference on Pattern Recognition and Artificial Intelligence (PRAI)10.1109/PRAI55851.2022.9904280(556-561)Online publication date: 19-Aug-2022
https://doi.org/10.1109/PRAI55851.2022.9904280

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents