DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion\n Frames

Abstract

We present Decentralized Distributed Proximal Policy Optimization (DD-PPO), a\nmethod for distributed reinforcement learning in resource-intensive simulated\nenvironments. DD-PPO is distributed (uses multiple machines), decentralized\n(lacks a centralized server), and synchronous (no computation is ever stale),\nmaking it conceptually simple and easy to implement. In our experiments on\ntraining virtual robots to navigate in Habitat-Sim, DD-PPO exhibits near-linear\nscaling -- achieving a speedup of 107x on 128 GPUs over a serial\nimplementation. We leverage this scaling to train an agent for 2.5 Billion\nsteps of experience (the equivalent of 80 years of human experience) -- over 6\nmonths of GPU-time training in under 3 days of wall-clock time with 64 GPUs.\n This massive-scale training not only sets the state of art on Habitat\nAutonomous Navigation Challenge 2019, but essentially solves the task\n--near-perfect autonomous navigation in an unseen environment without access to\na map, directly from an RGB-D camera and a GPS+Compass sensor. Fortuitously,\nerror vs computation exhibits a power-law-like distribution; thus, 90% of peak\nperformance is obtained relatively early (at 100 million steps) and relatively\ncheaply (under 1 day with 8 GPUs). Finally, we show that the scene\nunderstanding and navigation policies learned can be transferred to other\nnavigation tasks -- the analog of ImageNet pre-training + task-specific\nfine-tuning for embodied AI. Our model outperforms ImageNet pre-trained CNNs on\nthese transfer tasks and can serve as a universal resource (all models and code\nare publicly available).\n