Conventional information retrieval methods cannot simply be applied as is, to effectively access the increasingly diversified types of files and documents that are now available. This paper offers an overview on how to conduct a productive search for specific information on the Web, and also provides the mechanics of TITAN (Total Information Traverse AgeNt), a powerful Internet search system based on the robot mechanism that gives users access to the diverse resources of the Web.
TITAN features cross-lingual information retrieval capability that enables users to quickly and efficiently access information in Japanese, English and some other languages using Japanese or English search requests. It also has an interface that supports browsing based on VRML (Virtual Reality Modelling Language). This browsing mechanism helps clarify the retrieval results and allows users to access the value of information.
In simple terms, a robot is furnished with a list of URLs (Uniform Resource Locators) and it gathers information starting from that list. The robot extracts hyperlinks from Web sites which it then uses to recursively travel to those URLs. Information gathering is complete when all the links have been followed up. Currently, TITAN extracts over 300,000 URLs from the Web. Though this method may not be practical for collecting all the information on the Internet, it does allow the collection of a good proportion of the information that is available. The robot then analyses HTML (Hyper Text Markup Language) documents on the pages that it visits. The analysis consists of extracting those parts which are identified by HTML tags as being especially important. The important parts in this context designates titles, headlines, paragraph-breaks, and hyperlink annotations (generally known as 'anchors').
After being subjected to the above processing, TITAN takes the information and sorts it based on the type of HTML document. There are various kinds of documents on the Web. For example, there are indices such as Yahoo that contain an immense number of links to other Web sites, and there are pages that contain a great deal of information in and of themselves.
Taking this into account, TITAN employs two sorting criteria: information content and links. The former measures the degree of unique informational content contained in a document and the latter determines to what extent a document is linked to other Web sites.
Figure 1 shows the TITAN architecture for information gathering, text retrieval and page browsing. Before it gathers Web pages from the Internet, it indexes the information and makes a catalogue database; then it builds up the index database. Texts are retrieved from the index database according to user requests. For the text retrieval engine, TITAN uses the WAIS-sf and for CGI (Common Gateway Interface) script which makes exchanges between the retrieval engine and users it uses the SF-gate. If the users make retrieval requests in Japanese or Web pages are described by multi-byte code, it is necessary to use Juman which is a Japanese tagger.
Figure 2 is a first screen and an example of a search result that TITAN outputs.
Users can refer to the retrieval results by calling the VRML browser (WebSpace, etc.) from the WWW browser (Netscape, Mosaic, etc.) or SF-gate directly. This VRML browser communicates with link database, which is made from index database, and obtains the topologies of hyperlinks so the system describes these topologies on the VRML browser.
Firstly, phrases (individual words, phrases, and whole sentences are permitted) undergo tagger, and after grammatical articles and other extraneous elements are removed, the phrase is converted to English tokens by a dictionary lookup procedure based on the longest string-matching method.
Multiple variants commonly exist, but these are eliminated using the corpus derived from the information collected from the Web. Finally, the words in original phrases and the words in the translation are separately combined using the 'OR' operator and used in the retrieval.
To support the retrieval of Web resources that are written in other languages and character codes, TITAN also has the ability to automatically identify those character codes and languages. Thus, users can specify fancy describing languages in the first screen when they retrieve and in the output. The basic idea is to use a statistic language model to select the correctly decoded string as well as to determine the language. The strategy of language identification consists of the following steps: (see Figure 3).
The statistic-based language models are effective to identify. These models are based on text, which is regarded as a list of tokens where a token is a word in European languages, or a character in East-Asian languages. The likelihood of a list of tokens with regard to a language is the product of unigram probabilities for the class of every token in the list.
To train the statistic model of some languages, we gathered 100,000 Web pages and classified into target languages by humans. As a result, we gained 700 valid Web pages in the following languages.
| (1)Document's Title | (2)Translated Title | (3)Score |
| (4)Country of Origin | (5)Information Format | (6)Type of Information |
| (7)Describing Language |
Numbers 1 and 2 are WWW titles. If the title is in English, TITAN translates it into Japanese. Number 3 is the score which is assigned by the retrieval engine to show the degree of matching between the request and the actual in formation in WWW pages. Number 4, country of origin, designates the country where the information was created and is indicated by a symbol of a national flag. Number 5, the information format, specifies whether the information is an HTML (Hyper Text Markup Language) document, a graphic file, or some other kind of file and is designated by a unique icon that is associated with each type of information. Number 6, information type, designates to what extent a document contains information in and of itself (information content) and to what extent a document provides links to other WWW sites (link). This information is conveyed with an oval-shaped icon. The information content aspect is indicated by the upper blue part of the icon, and the link aspect is shown by the lower red portion of the icon. Number 7, describing languages, specifies what languages the WWW page is written. The presentation is 'name of language + degree'. Languages are specified in ISO CD 639/2 Draft and includes jpn (Japanese), eng (English), bg3 (Chinese), and so on. TITAN can distinguish Chinese, Japanese, Korean, Danish, Dutch, English, Finnish, French, German, Italian, Norwegian, Portuguese, Spanish, and Swedish.
For instance, the Internet robot outputs results suited to the user request. Although the results are output according to the score set by the retrieval engine, some pages are similar in content and this fact should be made known to users. If they are informed in advance, they can access these pages continuously. If users can make use of VRML browser, TITAN displays retrieval results on the 3-dimensional space based on the following information.
TITAN has the link database which has already been built from the index database. This database consists of about 2,000,000 pairs of URL. Using this information, the topology of hyperlinks and the class information of WWW pages are visualised. Once the user specifies a URL among the search results, TITAN displays the link topology with this URL as the starting point; that is, it extracts URLs which the URL points to and is pointed from. Finally, a portion of the topology is restored as a solid tree in the VRML space. Furthermore, TITAN decides the class information from this link information. To do this, we focus on the construction of URL, which are built in with following form.
scheme://host.domain [:port] / path [#anchor] [?key]
Scheme shows what protocol the URL uses, host and domain show the server that has the information, port shows the number of the server resource, path shows where the data exist, and anchor and key are used to request for CGI script, etc. It becomes clear by referring to the host and domain whether the information is on the same server, and whether to refer to the path information to clarify the class information between URLs. This browsing mechanism with VRML helps clarify the Internet hyperlink topology and allows users to access the value of information.
TITAN has proven to be a very useful tool due to its cross-lingual access capability and clear navigation interface. This cross-lingual access feature makes it much easier for non-native English speakers to access English web pages.
Return to Conference Proceedings