Project Description (Part B)

This is the part B of the project for CSE 494/598.
• TASK 3:   Implement the Authorities/Hub computation as discussed in the class. Specifically, for a given query Q, you first find top K most similar links using vector space model. Next, you grow these pages into a base set of pages. You make the adjacency matrix for these base set pages using the link extraction module provided below. You will then do the authority/hub computation on the adjancency matrix, and return the top m authorities and top m hubs. You will compare the results from pure vector space to the results from authorities/hub computation for a set of queries.
  • NOTE: The new .jar file that contains a crawl of asu.edu domain (around 3000 pages), as well as its link matrix. You need to use this data and not the data from part A for this as well as the later tasks.
  • SUBMIT:
    • Hardcopy showing the top 10 authorities and top 10 hubs using Authority Hub computation for the 10 example queries below
    • Comparison of these results to the results in TASK 1 and TASK 2 in your own words
    • Hardcopy of your code

• TASK 4:   Do the pagerank calculation for the entire ASU crawl. Then given a query, obtain the top 10 documents by combining the pagerank and vector space similarity using w*(pagerank)+(1-w)*(Vector space similarity) where w varies between 0 and 1. You may want to experiment with different weights. (Hint: Make sure to normalize he page rank such that it, like vector similarity, is between 0 and 1.)
  • SUBMIT:
    • Hardcopy showing the results for the below queries when combinining pagerank with vector space model
    • A report on your observations as w varies from 0 to 1; and also comparison with A/H method.
    • Hardcopy of your code
  TASK 5:   Cluster the query results before presenting them to user. Given a query, obtain the top 50 or 100 documents using vector space model (or vector space with page-rank model) and use a clustering algorithm (bisecting k-means is preferred) to cluster the top documents into k clusters. You can experiment with different k values between 3 and 10.
  • SUBMIT:
    • Hardcopy showing the clusters for the sample queries below
    • Analyze the effect of number of clusters, and also your own evaluation of whether the clusters seem to correspond to any natural categories.
    • Hardcopy of your code

• TASK 6:   Demonstration of Tasks 1 through 5. The Demo would be for around 10-15 minutes. You will be asked to run the code on some sample queries (in the week of December 9th).

  Extra credit possibilities: You can consider the following for extra credit:

  1. A nice GUI for your search engine
  2. Use of query expansion techniques (such as scalar clusters) on the top 50 search results.
  3. Trying out other clustering techniques--such as Agglomerative or scalar clustering to cluster search results.
  4. Doing LSI analysis before doing clustering (in task 5) and seeing if the clustering improves if we retain only top few dimensions in LSI
  5. Anything else that you can think of that makes your search engine cooler..