A classification framework for web robots

Abstract

The behavior of modern web robots varies widely when they crawl for different purposes. In this article, we present a framework to classify these web robots from two orthogonal perspectives, namely, their functionality and the types of resources they consume. Applying the classification framework to a year‐long access log from the UConn SoE web server, we present trends that point to significant differences in their crawling behavior.