Analysis of Robot Detection approaches for ethical and unethical robots on Web server log

Abstract

:Due to proliferation of Web robots, it is becoming important to detect robots on commercial and educational websites. Web robots not only consumes a significant proportion of network bandwidth but also a threat for privacy and security. There are various techniques to detect robot from server log i.e. checking robots.txt access, matching keywords in User agent field, calculating browsing speed etc. In this paper, we have implemented and compared theabove three robot detection approaches on real Web server log. Moreover, we have also applied combined approach to detect robots and found that combined approach works better than other three approaches.