Robots Exclusion Protocol

摘要

This document standardizes and extends the Robots Exclusion Protocol method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.