INLS 890:
Experimental Informaton Retrieval

Description: Information Retrieval (IR) is a broad field, encompassing a wide-range of information-seeking tasks, such as web search, patent search, medical record search, and micro-blog search. While the underlying goal is the same (i.e., to retrieve relevant or useful content in response to an information request), different tasks require different solutions and methods of evaluation.

This course takes an in-depth look at experimental IR systems evaluated in annual community-wide evaluation forums such as TREC. Through weekly readings and in-class discussions, students will gain an understanding of different search problems and their best-performing solutions. Through a semester-long project, students will gain practical experience in putting together and evaluating an information retrieval system that addresses a particular information-seeking task.

Student groups will be strongly encouraged to put together a system that can participate in TREC 2013. However, this is not a requirement to do well in the course.

Prerequisites: INLS 509, Informaton Retrieval or consent from the instructor
In-Class Discussions: This is an individual assignment. Each student will be assigned a search task (or track) from either TREC 2012 or TREC 2013 and will lead two back-to-back in-class discussions (1.25 hours each, 2.5 hours total).

These two sessions will be divided in two parts. In the first part, the student will present a historical overview of the track and a survey of the the best-performing systems. In the second part, the student will lead a brainstorming session on new experimental solutions that might be competitive with the best-performing systems. This will account for 30% of the total grade. See discussion leadership guidelines for helpful tips on being a good presenter and discussion moderator.

If a student is interested in reviewing a track that is running in 2013 for the first time, then he/she will be asked to review papers outside of TREC that they think are relevant to the task.

Term Projects: Each term project will focus on a particular information-seeking task and will use data (documents + relevance judgements) provided by TREC or INEX (2012 or earlier). The goal of each project will be to investigate and evaluate at least one "special sauce" component that might improve a baseline system's performance. Each project should be associated with a hypothesis of the form: System A + "special sauce" will outperform System A without "special sauce". It is not crucial for the "special sauce" to work in order for the project to be successful. It is more important to determine why it does or doesn't work.

Students must work in groups of two or three. Projects with three students will be expected to be more ambitious than projects with one student.

Time & Location: M, W 11am-12:15pm, Manning 304
Instructor: Jaime Arguello (email, web)
Office Hours: T, Th 9:30-10:30 am, Manning 305
Recommended Textbook: Search Engines - Information Retrieval in Practice, W. B. Croft, D. Metzler, and T. Strohman. Cambridge University Press. 2009. Available at the bookstore.
Additional Resources: Foundations of Statistical Natural Language Processing. C. Manning and H Schutze. 1999.

Introduction to Information Retrieval. C. Manning, P. Raghavan and H. Schutze. 2008.
2013 TREC Tracks:

Knowledge Base Acceleration

Contextual Suggestion (2012)

Crowdsourcing (2012)


Microblog (2012)


Temporal Summarization

Federated Search

Course Policies: Laptops, Attendance, Participation, Collaboration, Plagiarism & Cheating, Late Policy
Grading: 20% class participation
30% in-class discussion (15% survey presentation, 15% brainstorming discussion leadership) See the Track Overview and Brainstorming Session Guidelines
50% final project (10% project proposal, 30% project report, 10% project presentation)
Grade Assignments: Letter grades will be assigned using the following scale: H 95-100%, P 80-94%, L 60-79%, and F 0-59%. All homework, exams, and the literature review will be graded on a curve.
Acknowledgement The structure of this course is inspired by Jamie Callan's Experimental Information Retrieval course at Carnegie Mellon University, which I took as a PhD student.
Schedule: Subject to change!
Lecture Date Events Topic
1 Wed. 1/9

Course Overview

2 Mon. 1/14

History of TREC

3 Wed. 1/16

Overview of TREC 2012

  • PDF fileDRAFT Overview of TREC 2012
    E.M. Voorhees, NIST

  • PDF fileOn the History of Evaluation in IR
    Stephen Robertson, Microsoft Research

4 Mon. 1/21 Martin Luther King Day (No Class) No Class
5 Wed. 1/23

Overview of TREC 2013

6 Mon. 1/28

Federated Search Track

  • PDF file Sources of Evidence for Vertical Selection
    J. Arguello, F. Diaz, J.-F. Crespo, J. Callan

  • PDF file Learning to Aggregate Vertical Results into Web Search Results
    J. Arguello, F. Diaz, J. Callan

  • PDF file Snippet-based Relevance Predictions for Federated Web Search
    T. Demeester, D. Nguyen, D. Trieschnigg, C. Develder, D. Hiemstra

7 Wed. 1/30 Federated Web Search Track (cont'd)
8 Mon. 2/4

Real-Time (Twitter) Track (Chris and Mary)

  • PDF file Overview of the TREC 2012 Microblog Track (Notebook version)
    I. Soboroff, I. Ounis, C. Macdonald, J. Lin

  • PDF file Overcoming Vocabulary Limitations in Twitter Microblogs
    Y. Kim, R. Yeniterzi, J. Callan

  • PDF file The University of Illinois' Graduate School of Library and Information Science at TREC 2012
    M. Efron, J. Deisner, P. Organisciak, G. Sherman, A. Lucic

  • PDF file Incorporating Temporal Information in Microblog Retrieval
    C. Willis, R. Medlin, J. Arguello

9 Wed. 2/6 Real-Time (Twitter) Track (cont'd)
10 Mon. 2/11

Contextual Suggestion Track (John and Sandeep)

  • PDF file Overview of the TREC 2012 Contextual Suggestion Track
    A. Dean-Hall, C. L. A. Clarke, J. Kamps, P. Thomas, E. M. Voorhees

  • PDF file UDel (Carterette) at TREC 2012
    DRAFT Notebook Version

    A. Bah Rabiou, P. Chandar, N. Kumar, A. Rao, D. Zhu, B. Carterette

  • PDF file (Not Too) Personalized Learning to Rank for Contextual Suggestion
    A. Yates, D. DeBoer, H. Yang, N. Goharian, S. Kunath, O. Frieder

  • PDF file An Exploraton of Ranking-Based Strategy for Contextual Suggestion
    P. Yang, H. Fang

  • PDF file Recommending Personalized Touristic Sights Using Google Places
    M. Sappelli, W. Kraaij, S. Verberne

  • PDF file University of Amsterdam at the TREC 2012 Contextual Suggestion Track:
    Exploiting Community-Based Suggestions From Wikitravel

    M. Koolen, J. Kamps, H. Huurdeman

11 Wed. 2/13 Contextual Suggestion Track (cont'd)
12 Mon. 2/18 Term Project Proposal Due

Session Track (Hunter)

  • PDF file Overview of the TREC 2012 Session TrackDRAFT Notebook Version
    E. Kanoulas, B. Carterette, M. Hall, P. Clough, M. Sanderson

  • PDF file PITT at TREC 2012 Session Track:
    Adaptive Browsing Novelty in a Search Session

    J. Jiang, D. He, S. Han

  • PDF file Effective Structured Query Formulation for Session Search
    D. Guan, H. Yang, N. Goharian

  • PDF file CWI at TREC 2012, KBA Track and Session Track
    S. Araujo, G. Gebremeskel, J. He, C. Bosscarino, A. de Vries

  • PDF file Webis at the TREC 2012 Session Track
    Extended Abstract for the Conference Notebook

    M. Hagen, M. Potthast, M. Busse, J. Gomoll, J. Harder, B. Stein

  • PDF file Rutgers at the TREC 2012 Session Track
    C. Liu, M. Cole, E. Baik, N. J. Belkin

13 Wed. 2/20 Session Track (cont'd)
14 Mon. 2/25

Parameter Tuning, Cross-validation, and Signficance Tests

  • PDF file A Comparison of Statistical Significance Tests for Information Retrieval Evaluation
    M. D. Smucker, J. Allan, and B. Carterette

  • Cross-validation on Wikipedia
    Cross-validation is a technique used to estimate a particular solution's generalization performance (i.e., it's performance on previosly unseen data). This Wikipedia article presents an overview of the different methods for cross-validation.

  • Clever methods of over-fitting
    Over-fitting happens when your estimate of generalization performance is inflated due to error in the experimental design. This blog post presents a nice overview of different ways in which over-fitting can happen.

15 Wed. 2/27

Using the Killdevil Computer Cluster

16 Mon. 3/4

Temporal Summarization Track (Casey, Mrudula)

  • PDF file Temporal summaries of new topics
    J. Allan, R. Gupta, V. Khandelwal

  • PDF file Evolutionary timeline summarization: a balanced optimization framework via iterative substitution
    R. Yan, X. Wan, J. Otterbacher, L. Kong, X. Li, Y. Zhang

  • PDF file On the value of temporal information in information retrieval
    O. Alonso, M. Gertz, R. Baeza-Yates

  • PDF file An evaluation corpus for temporal summarization
    V. Khandelwal, R. Gupta, J. Allan

17 Wed. 3/6 Temporal Summarization Track (cont'd)
18 Mon. 3/11 Spring Break (No Class) No Class
19 Wed. 3/13 Spring Break (No Class) No Class
20 Mon. 3/18

Crowdsourcing Track (Ashlee and Claudia)

  • PDF file Overview of the TREC 2012 Crowdsourcing Track (Notebook)
    M. D. Smucker, G. Kazai, M. Lease

  • PDF file Skierarchy: Extending the Power of Crowdsourcing Using a Hierarchy of Domain Experts, Crowd and Machine Learning
    R. Nellapati, S. Peerreddy, P. Singhal

  • PDF file Using Hybrid Methods to Reduce Crowd Judgments in TREC '12 Crowd
    C. Harris, P. Srinivasan

  • PDF file Using a Bayesian Model to Combine LDA Features with Crowdsourced Responses
    E. Simpson, S. Reece, A. Penta, S. D. Ramchurn

21 Wed. 3/20 Crowdsourcing Track (cont'd)
22 Mon. 3/25

Web Track (Diversity Ranking Task)

  • PDF file The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries
    J. Carbonell, J. Goldstein
    (Background Reading on Diversity Ranking)

  • PDF file Overview of the TREC 2012 Web Track (notebook draft)
    C. L. A. Clarke, N. Craswell, E. M. Voorhees

  • PDF file IRRA at TREC 2012: Divergence From Independence (DFI)
    B. Taner Dincer

  • PDF file LIA at TREC 2012 Web Track:
    Unsupervised Search Concepts Identification from General Sources of Information

    R. Deveaud, E. SanJuan, P. Bellot

  • PDF file University of Glasgow at TREC 2012:
    Experiments with Terrier in Medical Records, Microblog, and Web Tracks

    N. Limsopatham, R. McCreadie, M-Dyaa Albakour, C. Macdonald, R. L. T. Santos, I. Ounis

23 Wed. 3/27 Project Meeting
24 Mon. 4/1

Knowledgebase Acceleration

  • Paper 1

  • Paper 2

  • Paper 3

25 Wed. 4/3 Knowledgebase Acceleration (cont'd)
26 Mon. 4/8 Project Meeting
27 Wed. 4/10 Project Meeting
28 Mon. 4/15 Project Meeting
29 Wed. 4/17 Student Presentations
30 Mon. 4/22 Student Presentations
31 Wed. 4/24 Student Presentations
32 TBD Final Reports Due