Santanu Kolay
Papers
1
Total Citations
17
H-Index
1
About
Santanu Kolay is a researcher whose work sits at the critical intersection of web governance and search engine behavior. His most-cited study, "A larger scale study of robots.txt" (2008, 17 citations), provides one of the earliest systematic analyses of how websites use the Robots Exclusion Protocol to control crawler access. Kolay’s research reveals the subtle biases embedded in robots.txt files—showing how site owners can strategically allow or disallow content to specific crawlers, thereby shaping what gets indexed and how search results are prioritized. This work is foundational for understanding the power dynamics between content publishers and search engines, offering key insights into fairness, transparency, and the hidden rules that govern the web. By illuminating how technical protocols can introduce favorable or unfavorable biases, Kolay’s contributions remain highly relevant for researchers studying web crawling, information retrieval, and the ethics of algorithmic access. His work continues to inform debates on open web standards and the regulation of search engine behavior.
Research Focus
Key Achievements
Top Papers
- 1A larger scale study of robots.txt17 citations · 2008