INLS 890:
Experimental Informaton Retrieval

Description: Information Retrieval (IR) is a broad field, encompassing a wide-range of information-seeking tasks, such as web search, patent search, medical record search, and micro-blog search. While the underlying goal is the same (i.e., to retrieve relevant or useful content in response to an information request), different tasks require different solutions and methods of evaluation.

This course takes an in-depth look at experimental IR systems evaluated in annual community-wide evaluation forums such as TREC. Through weekly readings and in-class discussions, students will gain an understanding of different search problems and their best-performing solutions. Through a semester-long project, students will gain practical experience in putting together and evaluating an information retrieval system that addresses a particular information-seeking task.

Student groups will be strongly encouraged to put together a system that can participate in TREC 2014. However, this is not a requirement to do well in the course.

Prerequisites: INLS 509, Informaton Retrieval or consent from the instructor
In-Class Discussions: This is an individual assignment. Each student will be assigned a search task (or track) from either TREC 2013 and will lead two back-to-back in-class discussions (1 hour and 15 minutes each).

These two sessions will be divided in two parts. In the first part, the student will present a historical overview of the track and a survey of the the best-performing systems at TREC 2013. In the second part, the student will lead a brainstorming session on new experimental solutions that might be competitive with the best-performing systems. This will account for 30% of the total grade. See discussion leadership guidelines for helpful tips on being a good presenter and discussion moderator.

Term Projects: Each term project will focus on a particular information-seeking task and will use data (documents + relevance judgements) provided by TREC or INEX (2012 or earlier). The goal of each project will be to investigate and evaluate at least one "special sauce" component that might improve a baseline system's performance. Each project should be associated with a hypothesis of the form: System A + "special sauce" will outperform System A without "special sauce". It is not crucial for the "special sauce" to work in order for the project to be successful. It is more important to determine why it does or doesn't work.

Students must work in groups of two or three. Projects with three students will be expected to be more ambitious than projects with one student.

Time & Location: M 12:30pm-3:15pm, Manning 214
Instructor: Jaime Arguello (email, web)
Office Hours: T, Th 10:00-11:00 am, Manning 305
Recommended Textbook: Search Engines - Information Retrieval in Practice, W. B. Croft, D. Metzler, and T. Strohman. Cambridge University Press. 2009. Available at the bookstore.
Additional Resources: Foundations of Statistical Natural Language Processing. C. Manning and H Schutze. 1999.

Introduction to Information Retrieval. C. Manning, P. Raghavan and H. Schutze. 2008.
Course Policies: Laptops, Attendance, Participation, Collaboration, Plagiarism & Cheating, Late Policy
Grading: 20% class participation
30% in-class discussion (15% survey presentation, 15% brainstorming discussion leadership) See the Track Overview and Brainstorming Session Guidelines
50% final project (10% project proposal, 30% project report, 10% project presentation)
Grade Assignments: Undergraduate grading scale: A+ 97-100%, A 94-96%, A- 90-93%, B+ 87-89%, B 84-86, B- 80-83%, C+ 77-79%, C 74-76%, C- 70-73%, D+ 67-69%, D 64-66%, D- 60-63%, F 0-59%

Graduate grading scale: H 95-100%, P 80-94%, L 60-79%, and F 0-59%.

All homework, exams, and the literature review will be graded on a curve.
Acknowledgement The structure of this course is inspired by Jamie Callan's Experimental Information Retrieval course at Carnegie Mellon University, which I took as a PhD student.
Schedule: Subject to change!
Lecture Date Events Topic
1 Mon. 1/13  

Course Overview and History of TREC

2 Mon. 1/20 MLK Day (No Class)  
3 Mon. 1/27  

Overview of TREC 2013

4 Mon. 2/3  

Web Track (Jaime)

5 Mon. 2/10  

Contextual Suggestion Track (Heejun)

6 Mon. 2/17 Project Proposal Due

Federated Web Search Track (Jeff)

7 Mon. 2/24  

Using the Killdevil Computer Cluster

8 Mon. 3/3  

Knowledge Base Acceleration Track (Patrick)

9 Mon. 3/10 Spring Break (No Class)  
10 Mon. 3/17  

Session Track (Christopher)

11 Mon. 3/24  

Microblog Track (Ravi)

12 Mon. 3/31  

Temporal Summarization Track (Brittany)

13 Mon. 4/7  

Crowdsourcing Track (Jaime)

14 Mon. 4/14 No Class  
15 Mon. 4/21   Student Presentations
16 Fri. 4/25 Project Report Due