Efficient & effective image-based localization

The problem of image-based localization is the problem of accurately determining the position and orientation from which a novel photo was taken relative to a 3D representation of the scene. It is encountered in many interesting applications such as pedestrian or robot navigation, Augmented Reality, or Structure-from-Motion, creating a strong need for algorithms solving the image-based localization problem. In this thesis, we therefore present solutions to this problem that are both effective and efficient, i. e., we propose methods that can localize novel query images taken under a wide range of viewing conditions while requiring only a small amount of processing time. We assume that the 3D scene representation is obtained by using Structure-from- Motion techniques to reconstruct the environment from a set of photos. As a result, we can associate each 3D point with multiple image descriptors modeling the local appearance of the scene around this point. We can then obtain 2D-3D correspondences between 2D feature points in the query image and 3D scene points in the model by solving a descriptor matching problem. These 2D-3D matches can in turn be used to estimate the camera position of the query image, i. e., the position and orientation from which it was taken. The main difficulty of descriptor matching lies in the sheer size of the problem, since our models contain millions of 3D points while thousands of features are found in our query images. As a major contribution, we show that the resulting descriptor matching problem can still be solved very efficiently using prioritized search. We propose a prioritization scheme that is easy to implement, yet can be expected to perform close to optimal in practice. By combining our prioritization with a novel active search step that is able to discover additional matches, we are able to derive an imagebasedlocalizationapproachthatachievesorsurpassesstate-of-the-arteffectivenesswhile offering the fastest run-times published so far. Analyzing such direct matching methods, we demonstrate that their major advantage, namely their ability to identify a set of high-quality matches, also prevents their scalability to larger datasets. Consequently, we also consider image retrieval methods for image-based localization since they are inherently more scalable. As a second major contribution, we identify the algorithmic factors preventing image retrieval methods to achieve the same effectiveness as our original system and propose a modification that is able to close the gap in effectiveness without sacrificing scalability.