Large Scale Joint Semantic Re-Localisation and Scene Understanding via\n Globally Unique Instance Coordinate Regression

摘要

In this work we present a novel approach to joint semantic localisation and\nscene understanding. Our work is motivated by the need for localisation\nalgorithms which not only predict 6-DoF camera pose but also simultaneously\nrecognise surrounding objects and estimate 3D geometry. Such capabilities are\ncrucial for computer vision guided systems which interact with the environment:\nautonomous driving, augmented reality and robotics. In particular, we propose a\ntwo step procedure. During the first step we train a convolutional neural\nnetwork to jointly predict per-pixel globally unique instance labels and\ncorresponding local coordinates for each instance of a static object (e.g. a\nbuilding). During the second step we obtain scene coordinates by combining\nobject center coordinates and local coordinates and use them to perform 6-DoF\ncamera pose estimation. We evaluate our approach on real world (CamVid-360) and\nartificial (SceneCity) autonomous driving datasets. We obtain smaller mean\ndistance and angular errors than state-of-the-art 6-DoF pose estimation\nalgorithms based on direct pose regression and pose estimation from scene\ncoordinates on all datasets. Our contributions include: (i) a novel formulation\nof scene coordinate regression as two separate tasks of object instance\nrecognition and local coordinate regression and a demonstration that our\nproposed solution allows to predict accurate 3D geometry of static objects and\nestimate 6-DoF pose of camera on (ii) maps larger by several orders of\nmagnitude than previously attempted by scene coordinate regression methods, as\nwell as on (iii) lightweight, approximate 3D maps built from 3D primitives such\nas building-aligned cuboids.\n