Web archiving strategies by using Web mining techniques

Abstract

For preserving huge volume of born-digital information in the Internet, national diet library in Japan has been developing a experimental web archiving system, WARP (http://warp.ndl.go.jp/). However, in order to handle monotonously increasing digital information, we consider many difficult problems of long life data preservation from various technical aspects. In this paper, we try to apply web mining techniques to web archiving strategies. Our strategies are based on the experiences of our Mondou web search engine and web robots, which are based on text/web mining technologies.