CSE 494/598 Information Retrieval, Mining and Integration on the Internet

This course is geared towards exposing students to some of the core technologies for controlling and using the content on the Internet. The following are some of the questions we will consider:

1. How do search engines work? Why are some pp better than others?
2. Can we think of the web as a big database/knoweldge base and support efficient database style query processing?
3. Can we find useful pearls and patterns in the mass of accessible data on the Internet?

This course will be breadth-oriented introduction to the issues involved in answering these questions.

Prerequisites: CSE 310 required. Other courses that will help include CSE 471 (AI) CSE 412 (Databases) and CSE 450 (Algorithms). I am hoping that students have had at least one of these 4-level courses already, but won't insist on them. Students planning to register for this course are encouraged to talk to the instructor (via email at rao wholivesat asu dot edu).

Grading: The grading will be based on class participation, exams and projects.

Textbooks: There is no prescribed textbook. We will read papers (see the reading list.)

Overview: The best overview is the list of topics and lecture notes from the previous offering (shown below).

Additional pointers:

• Check out what students say about the last offering of this course in interactive review and in CEAS student evaluations.
• Check out what earlier students did in the last offering of the course
• Check out what some of the earlier students are doing now.

Lecture Notes from Fall 2005 (slides in ppt; lectures in .wav)

1. Introduction (8/22;)
   Audio of the lecture [Aug 22, 2005]

2. Text retrieval; vectorspace ranking
   1. Audio of the lecture [Aug 24, 2005]
   2. Audio of the lecture of [Aug 29, 2005]
   3. Audio of the lecture of [Aug 31, 2005]

3. Indexing/Retrieval issues

4. Correlation analysis & Latent Semantic Indexing
   • Audio of the lecture of [Sep 7, 2005]

5. Search engine technology
   • Audio of the lecture of [Sep 12, 2005] (LSI & Websearch start)
   • Audio of the lecture of [Sep 14, 2005] (Google bombs; Authorities/hubs; page rank begin)
   • Audio of the lecture of [Sep 19, 2005] (page rank and variations & beyond; crawling)

6. Anatomy of Google etc
   • Audio of the lecture of [Sep 26, 2005] (Google, Search directions. Note that the class is discussion oriented. Students qns may not have gotten recorded well)

7. Clustering
   • Audio of the lecture of [Sep 21, 2005] (General background on clustering; K-means)
   • Audio of the lecture of [Sep 28, 2005] (Many variations of K-means; Hierarchical clustering; Hybrids.)

8. Text Classification
   • Audio of the lecture of [Oct 3, 2005] (Text Categorization/Classification)
   • Audio of the lecture of [Oct 5, 2005] (Text Categorization/Classification contd NBC/ Feature Selection/ Spam mail detection etc; Start of Filtering)

9. Filtering/Personalization
   • Audio of the lecture of [Oct 10, 2005] (Feature selection vs. LSI/LDA; Collab and Content based filtering, content-boosted collab as well as content-based filtering, co-training)

10. Web & Databases: Why do we even care?
    • Audio of the lecture of [Oct 12, 2005] Web and Databases: Why do we even care..

11. XML and handling semi-structured data
    • Audio of the lecture of [Oct 19, 2005] XML

12. Semantic web and its standards (RDF/RDF-S/OWL...)
    • Audio of the lecture of [Oct 24, 2005] Xquery; Semantic web; RDF/RDFS; folksonomies

13. Information Extraction
    • Audio of the lecture of [Oct 26, 2005] Folksonomies; Information Extraction begin
    • Audio of the lecture of [Oct 31, 2005] Information Extraction & Automated Annotation (+ 3 big ideas)

14. Data/Information Integration/aggregation
    • Audio of the lecture of [Nov 2, 2005] SemTag/Seeker; Information Integration. Warehousing vs. virtual approach
    • Audio of the lecture of [Nov 7, 2005] Overview of data integration; special cases of data integration; dimensions of data integration; GAV vs. LAV start
    • Audio of the lecture of [Nov 9, 2005] LAV/GAV; Discussion on data integration; Start of query processing in data integration.

15. Query Processing in Data Integration: Gathering and Using Source Statistics
    • Audio of the lecture of [Nov 14, 2005] (Adaptive) Query processing and statistics learning in data aggregation
    • Audio of the lecture of [Nov 16, 2005] Discussion of Coverage/overlap statistics learning; latency statistics learning; multi-objective query optimization.

16. Bridging Information Retrieval and Databases
    • Audio of the lecture of [Nov 21, 2005] Bridging DB/IR approaches; motivation; discussion of text-joins and keyword search on DB.
    • Audio of the lecture of [Nov 23, 2005]Discussion of supporting imprecise queries; discussion of collection selection/meta-search

17. Social Networks
    • Audio of the lecture of [Nov 28, 2005] (Social networks; measures of centrality and prestige; small world phenomena; uniform random networks; scalefree networks and powerlaw distributions)
    • Audio of the lecture of [Nov 30, 2005] (Powerlaw distributions; brittleness of scale-free networks to attacks; Searchability in smallworld networks; Applications of social networks in Web; Trust analysis and its applications to pagerank.)

18. Interactive Review + a (corny) ending (Here are the notes by the TA of the student review comments)
    • Summary review Comments by all the students of the class [Dec 5, 2005] (The student speaking always had the microphone; so any comments made by Rao are not recorded well).

11/21; 11/23 (DB & IR); Collection Selection; Webservices
11/28; 11/30: Social network Analysis (Kevin bacon game; Erdos number;
Trust propagation etc)
12/5: Interactive Review
12/17: 

DB+IR --> Imprecise queries (1 week)
Collection Selection (1 class)
web services (1 class)
Social Network Analysis (1-2 classes)