INLS 609:
Experimental Information Retrieval

Description: Information Retrieval (IR) is a broad field, encompassing a wide-range of information-seeking tasks, such as web search, patent search, medical record search, and micro-blog search. While the underlying goal is the same (i.e., to retrieve relevant or useful content in response to an information request), different tasks require different solutions and methods of evaluation.

This course takes an in-depth look at experimental IR systems evaluated in annual community-wide evaluation forums such as TREC. Through weekly readings and in-class discussions, students will gain an understanding of different search problems and their best-performing solutions. Through a semester-long project, students will gain practical experience in putting together and evaluating an information retrieval system that addresses a particular information-seeking task.

Student groups will be strongly encouraged to put together a system that can participate in TREC 2016. However, this is not a requirement to do well in the course.

Prerequisites: INLS 509, Informaton Retrieval or consent from the instructor
In-Class Discussions: This is an individual assignment. Each student will be assigned a search task (or track) from TREC 2015 and will lead two back-to-back in-class discussions of 1 hour and 15 minutes each.

These two sessions will be divided into two parts. In the first part, the student will present a historical overview of the track and a survey of the the best-performing systems from TREC 2015. In the second part, the student will lead a brainstorming session on new experimental solutions that might be competitive with the best-performing systems. This will account for 30% of the total grade. See discussion leadership guidelines for helpful tips on being a good presenter and discussion moderator.

Term Projects: Each term project will focus on a particular information-seeking task and will use data (documents + relevance judgements) provided by TREC or INEX (2015 or earlier). The goal of each project will be to investigate and evaluate at least one "special sauce" component that might improve a baseline system's performance. Each project should be associated with a hypothesis of the form: System A + "special sauce" will outperform System A without "special sauce". It is not crucial for the "special sauce" to work in order for the project to be successful. It is more important to determine why it does or doesn't work.

Students must work in groups of two or three. Projects with three students will be expected to be more ambitious than projects with two students.

Time & Location: M 10:10am-11:25am, Manning 307
Instructor: Jaime Arguello (email, web)
Office Hours: T, Th 11:00am-12:00pm, Manning 305
Recommended Textbook: Search Engines - Information Retrieval in Practice, W. B. Croft, D. Metzler, and T. Strohman. Cambridge University Press. 2009. Available on-line.
Additional Resources: Foundations of Statistical Natural Language Processing. C. Manning and H Schutze. 1999.

Introduction to Information Retrieval. C. Manning, P. Raghavan and H. Schutze. 2008.

Clinical Decision Support Track
Contextual Suggestion Track
Dynamic Domain Track
Live Question-Answering Track
Microblog Track
Tasks Track
Temporal Summarization Track
Total Recall Track
Course Policies: Laptops, Attendance, Participation, Collaboration, Plagiarism & Cheating, Late Policy
Grading: 20% class participation
30% in-class discussion (15% survey presentation, 15% brainstorming discussion leadership) See the Track Overview and Brainstorming Session Guidelines
50% final project (10% project proposal, 30% project report, 10% project presentation)
Grade Assignments: Undergraduate grading scale: A+ 97-100%, A 94-96%, A- 90-93%, B+ 87-89%, B 84-86, B- 80-83%, C+ 77-79%, C 74-76%, C- 70-73%, D+ 67-69%, D 64-66%, D- 60-63%, F 0-59%

Graduate grading scale: H 95-100%, P 80-94%, L 60-79%, and F 0-59%.

All assignments, exams, and the literature review will be graded on a curve.
Acknowledgement The structure of this course is inspired by Jamie Callan's Experimental Information Retrieval course at Carnegie Mellon University, which I took as a PhD student.
Schedule: Subject to change!
Date Events Topic
Mon. 1/11   Course Overview
Wed. 1/13  

History of TREC

Mon. 1/18 MLK Day (No class)  
Wed. 1/20  

Overview of TREC 2015

Mon. 1/25 Class Cancelled (Snow!)  
Wed. 1/27  

Experimentation Review

Mon. 2/1  

Using the Killdevil Computer Cluster

Wed. 2/3 Project Discussion  
Mon. 2/8  

Microblog Track (Jaime)

Wed. 2/10   Microblog Track (Jaime)
Mon. 2/15  

Clinical Decision Support Track (Jaime)

Wed. 2/17   Clinical Decision Support Track (Jaime)
Mon. 2/22 Project Proposal Due

Contextual Suggestion Track (TBD)

Wed. 2/24   Contextual Suggestion Track (TBD)
Mon. 2/29  

Dynamic Domain Track (Ryan)

Wed. 3/2   Dynamic Domain Track (Ryan)
Mon. 3/7  

Live Question Answering Track (Ying)

Wed. 3/9   Live Question Answering Track (Ying)
Mon. 3/14 Spring Break (No class)  
Wed. 3/16 Spring Break (No class)  
Mon. 3/21 No Class  
Wed. 3/23 No Class  
Mon. 3/28  

Tasks Track (Yongsu)

Wed. 3/30   Tasks Track (Yongsu)
Mon. 4/4  

Temporal Summarization Track (Justin)

Wed. 4/6   Temporal Summarization Track (Justin)
Mon. 4/11  

Total Recall Track (Jaime)

Wed. 4/13   Total Recall Track (Jaime)
Mon. 4/18   Project Discussion
Wed. 4/20  

Query Performance Prediction (Jaime)

Mon. 4/25   Student Presentations
Wed. 4/27   Student Presentations
Fri. 4/29 Final Project Due