OTHER
Robots Exclusion Protocol
M.B.M. de Koster, G. Illyes, H. Zeller, L. Sassman
- Year
- 2022
- Citations
- 51
Abstract
This document standardizes and extends the Robots Exclusion Protocol method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.
Keywords
Protocol (science)Computer scienceMedicine
Related papers
OTHER
📊 26,957 cites
Statistical Learning Theory
Yuhai Wu, Vladimir Vapnik
1999
PERCEPTION
📊 22,245 cites
Artificial intelligence: a modern approach
1995
OTHER
Open access📊 20,501 cites
Fractional Differential Equations
Igor Podlubný
2025
OTHER
📊 18,993 cites
Applied Nonlinear Control
Jean-Jacques Slotine, Weiping Li
1991