Information Retrieval Project

PART B

Project Description

This is the part B of the project for CSE 494/598.

TASK 3: Implement the Authorities/Hub computation as discussed in the class. Specifically, for a given query Q,
- Construct the "root set' by identifying the "top K" answers using vector space search developed in Part A ( experiment with values of K, Tip: larger K equals more processing time, start with K=10) .
- Then grow these pages into a "base set" of pages. To make the adjacency matrix for the base set of pages use the "LinkExtract.java" and the file "HashedLinks" provided as part of the code (see details below).
- Next do authority/hub computation using the adjacency matrix and return the "top N" authorities and "top N" hubs. (N=10 would be a good bet).
- Compare and analyze the results given by Vector Space for query Q, with those obtained from the authority/hub computation.

SUBMIT:

Hardcopy showing the Top N authorities and hubs using the A/H computation for the sample queries given below.

A report comparing and analyzing the A/H results with those given by pure Vector Space based search. How does varying the size of "root set" affect results of A/H computation? Which results are more relevant: Authorities or Hubs? Comments?

Hardcopy of your code with comments.

TASK 4: Compute PageRank for all the files from the ASU crawl given as part of the code. (Note: You are not required to process the crawled files. The LinkExtract.java code together with the file HashedLinks can be used to identify the link structure necessary to determine PageRank). Then given a query Q,

Return the "Top 10" results for the query by combining PageRank and Vector Space similarity values for the results using the formula
w * (PageRank) + (1-w)* (Vector Space Similarity), where 0 < w < 1.

Do make provisions for varying the value of 'w' at query time. (Hint: For ease of combining with Vector Space rank, normalize the PageRank such that it lies between 0 and 1)

SUBMIT:

Hardcopy showing the results for sample queries derived using a combination of PageRank and Vector Space ranking.

A report comparing and analyzing the A/H results with those given by PageRank+VectorSpace. Comment on the effects of varying 'w' between 0 and 1. Comment on the effects of varying the value of "c" (see formula for PageRank computation in class notes). Does PageRank computation converge ?

Hardcopy of your code with comments.

Extra Credit:

Implement a GUI for your search engine. Make provisions for selecting Vector Space, A/H, PageRank + Vector Space model for ranking your answers.

The GUI should preferably be a stand alone application or applet. The GUI can be servlet based only if you have access to a personal Web server that can be accessed by the TA.

You are encouraged to make the output as "Googlish" as you can i.e. simple interface, links open source documents, anything else you fancy.

additional extra credit tasks are certainly possible; consult the instructor and or TAs.

Note: A end-of-semester Demo of all tasks will be required. A GUI could come in handy at that time. Hence you are highly encouraged to implement a GUI.

Example Queries

Campus Tour
Medical Care
Transcripts
Admissions
Employee Benefits
Languages
Hayden Library
Parking Decal
SRC

Sample Output: Click here for a sample output for A/H and PageRank + Vector Space search.

Resource Description & Documentation

Knowledge of Java is a pre-requisite for the course. If you need to brush up your knowledge read the Java Tutorial. The Java API documentation which describes the class structure for various standard packages is available here.
The following files ares provided as part of the project:
- Project_TaskB.zip (750KB) contains :
  - source, class, and documentation files for classes CSE494pgRank.LinkGen and CSE494pgRank.LinkExtract.
  - Hashedlinks (described below).
- You will need to use class files from part A of the project, including the Lucene package, the CSE494 package, and your implementation files. You also need to use the index from part A.
- LinkGen.java and LinkExtract.java are provided to help in computing PageRank and Authority/Hub for the crawled pages. LinkGen generates the Link Matrix from the files crawled and saves it to the file Hashedlinks (you are given Hashedlinks in Project_TaskB.zip). LinkExtract uses Hashedlinks and provides methods to get information about pages pointed to by a given page and pages pointing to a given page. Since you are given Hashedlinks file, you can see how LinkExtract works.
  - You can use the command
    java CSE494pgRank.LinkExtract crawledpages Hashedlinks
    
    to see a demo of the program.
- Note: Because of the size of the crawl, the PageRank task could be memory intensive (i.e. the JVM may require more heap size than usual). If your program returns a OutOfMemoryException, then try to increase the runtime JVM heap size by using the command -Xmx###m when running your program, where ### is the size of memory you want to allocate to the heap. For instance: 'java -Xmx512m programname' will increase the heap to 512MB.

Deadline : Due date for part B: 03/20/2007.

Last Modified: 02.24.2004