On updating in very short time by distributed search engines

Abstract

Conventional search engines employ a centralized architecture. However, such engines are not suitable for fresh information retrieval because they take a long time to collect Web pages using a Web robot (or crawler). There are also distributed search engines such as Harvest etc., but they cannot update in a very short time, e.g. a few minutes. In this paper, we propose a distributed information retrieval system, called a cooperative search engine (CSE), in which multiple distributed local search engines communicate with each other in order to realize a whole global search engine. A local search engine has own index database and can update quickly. In addition, they cooperate with each other in order to reduce their workload. In this paper, we describe the structure and behavior of CSE, and its efficiency.