Visual Self-localization via Inferring View-to-Map Correspondences

This paper poses self-localization of a moving camera as a view-to-map correspondence problem, utilizing a large set of ground-view maps rendered by existing mapping tool, like Google Earth. To address the viewpoint and appearance differences between the rendered maps and camera views, we present a unified solution that comprises of three components. Firstly, we represent each rendered map with a set of view-dependent feature patches and discriminatively train an appearance model for each patch. With these models we cast the view-to-map correspondence task as a ranking task. Secondly, we introduce a programming based method to discover feature correspondences over consecutive frames which are used to estimate visual odometry of camera. The programming formula is regularized with both flow type constraints and spatial smoothness constraints to account for scene noises. Thirdly, we present a joint probabilistic formula to integrate both visual odometry and view-to-map correspondences for reasoning with uncertainties. Evaluations with comparisons over challenging monocular videos demonstrated that our method clearly outperforms the alternative methods. In particular, our method is capable of localizing a moving camera with sub-meter accuracies in the scenario of about 13,000 square meters.