INLS 890:
Experimental Informaton Retrieval

Description:

Information Retrieval (IR) is a broad field, encompassing a wide-range of information-seeking tasks, such as web search, patent search, medical record search, and micro-blog search. While the underlying goal is the same (i.e., to retrieve relevant or useful content in response to an information request), different tasks require different solutions and methods of evaluation.

This course takes an in-depth look at experimental IR systems evaluated in annual community-wide evaluation forums such as TREC. Through weekly readings and in-class discussions, students will gain an understanding of different search problems and their best-performing solutions. Through a semester-long project, students will gain practical experience in putting together and evaluating an information retrieval system that addresses a particular information-seeking task.

Student groups will be strongly encouraged to put together a system that can participate in TREC 2013. However, this is not a requirement to do well in the course.

Prerequisites:

INLS 509, Informaton Retrieval or consent from the instructor

In-Class Discussions:

This is an individual assignment. Each student will be assigned a search task (or track) from either TREC 2012 or TREC 2013 and will lead two back-to-back in-class discussions (1.25 hours each, 2.5 hours total).

These two sessions will be divided in two parts. In the first part, the student will present a historical overview of the track and a survey of the the best-performing systems. In the second part, the student will lead a brainstorming session on new experimental solutions that might be competitive with the best-performing systems. This will account for 30% of the total grade. See discussion leadership guidelines for helpful tips on being a good presenter and discussion moderator.

If a student is interested in reviewing a track that is running in 2013 for the first time, then he/she will be asked to review papers outside of TREC that they think are relevant to the task.

Term Projects:

Each term project will focus on a particular information-seeking task and will use data (documents + relevance judgements) provided by TREC or INEX (2012 or earlier). The goal of each project will be to investigate and evaluate at least one "special sauce" component that might improve a baseline system's performance. Each project should be associated with a hypothesis of the form: System A + "special sauce" will outperform System A without "special sauce". It is not crucial for the "special sauce" to work in order for the project to be successful. It is more important to determine why it does or doesn't work.

Students must work in groups of two or three. Projects with three students will be expected to be more ambitious than projects with one student.

Time & Location:

M, W 11am-12:15pm, Manning 304

Instructor:

Jaime Arguello (email, web)

Office Hours:

T, Th 9:30-10:30 am, Manning 305

Recommended Textbook:

Search Engines - Information Retrieval in Practice, W. B. Croft, D. Metzler, and T. Strohman. Cambridge University Press. 2009. Available at the bookstore.

Additional Resources:

Foundations of Statistical Natural Language Processing. C. Manning and H Schutze. 1999.

Introduction to Information Retrieval. C. Manning, P. Raghavan and H. Schutze. 2008.

2013 TREC Tracks:

Knowledge Base Acceleration

Contextual Suggestion (2012)

Temporal Summarization

Federated Search

Course Policies:

Laptops, Attendance, Participation, Collaboration, Plagiarism & Cheating, Late Policy

Grading:

20% class participation
30% in-class discussion (15% survey presentation, 15% brainstorming discussion leadership) See the Track Overview and Brainstorming Session Guidelines
50% final project (10% project proposal, 30% project report, 10% project presentation)

Grade Assignments:

Letter grades will be assigned using the following scale: H 95-100%, P 80-94%, L 60-79%, and F 0-59%. All homework, exams, and the literature review will be graded on a curve.

Acknowledgement

The structure of this course is inspired by Jamie Callan's Experimental Information Retrieval course at Carnegie Mellon University, which I took as a PhD student.

Schedule:

Subject to change!

Lecture	Date	Events	Topic
1	Wed. 1/9		Course Overview
2	Mon. 1/14		History of TREC Readings: Economic Impact Assessment of NIST’s Text REtrieval Conference (pg 1-59)
3	Wed. 1/16		Overview of TREC 2012 Readings: DRAFT Overview of TREC 2012 E.M. Voorhees, NIST On the History of Evaluation in IR Stephen Robertson, Microsoft Research
4	Mon. 1/21	Martin Luther King Day (No Class)	No Class
5	Wed. 1/23		Overview of TREC 2013 Readings:
6	Mon. 1/28		Federated Search Track Readings: Sources of Evidence for Vertical Selection J. Arguello, F. Diaz, J.-F. Crespo, J. Callan Learning to Aggregate Vertical Results into Web Search Results J. Arguello, F. Diaz, J. Callan Snippet-based Relevance Predictions for Federated Web Search T. Demeester, D. Nguyen, D. Trieschnigg, C. Develder, D. Hiemstra
7	Wed. 1/30		Federated Web Search Track (cont'd)
8	Mon. 2/4		Real-Time (Twitter) Track (Chris and Mary) Readings: Overview of the TREC 2012 Microblog Track (Notebook version) I. Soboroff, I. Ounis, C. Macdonald, J. Lin Overcoming Vocabulary Limitations in Twitter Microblogs Y. Kim, R. Yeniterzi, J. Callan The University of Illinois' Graduate School of Library and Information Science at TREC 2012 M. Efron, J. Deisner, P. Organisciak, G. Sherman, A. Lucic Incorporating Temporal Information in Microblog Retrieval C. Willis, R. Medlin, J. Arguello
9	Wed. 2/6		Real-Time (Twitter) Track (cont'd)
10	Mon. 2/11		Contextual Suggestion Track (John and Sandeep) Readings: Overview of the TREC 2012 Contextual Suggestion Track A. Dean-Hall, C. L. A. Clarke, J. Kamps, P. Thomas, E. M. Voorhees UDel (Carterette) at TREC 2012 DRAFT Notebook Version A. Bah Rabiou, P. Chandar, N. Kumar, A. Rao, D. Zhu, B. Carterette (Not Too) Personalized Learning to Rank for Contextual Suggestion A. Yates, D. DeBoer, H. Yang, N. Goharian, S. Kunath, O. Frieder An Exploraton of Ranking-Based Strategy for Contextual Suggestion P. Yang, H. Fang Recommending Personalized Touristic Sights Using Google Places M. Sappelli, W. Kraaij, S. Verberne University of Amsterdam at the TREC 2012 Contextual Suggestion Track: Exploiting Community-Based Suggestions From Wikitravel M. Koolen, J. Kamps, H. Huurdeman
11	Wed. 2/13		Contextual Suggestion Track (cont'd)
12	Mon. 2/18	Term Project Proposal Due	Session Track (Hunter) Readings: Overview of the TREC 2012 Session TrackDRAFT Notebook Version E. Kanoulas, B. Carterette, M. Hall, P. Clough, M. Sanderson PITT at TREC 2012 Session Track: Adaptive Browsing Novelty in a Search Session J. Jiang, D. He, S. Han Effective Structured Query Formulation for Session Search D. Guan, H. Yang, N. Goharian CWI at TREC 2012, KBA Track and Session Track S. Araujo, G. Gebremeskel, J. He, C. Bosscarino, A. de Vries Webis at the TREC 2012 Session Track Extended Abstract for the Conference Notebook M. Hagen, M. Potthast, M. Busse, J. Gomoll, J. Harder, B. Stein Rutgers at the TREC 2012 Session Track C. Liu, M. Cole, E. Baik, N. J. Belkin
13	Wed. 2/20		Session Track (cont'd)
14	Mon. 2/25		Parameter Tuning, Cross-validation, and Signficance Tests Readings: A Comparison of Statistical Significance Tests for Information Retrieval Evaluation M. D. Smucker, J. Allan, and B. Carterette Cross-validation on Wikipedia Cross-validation is a technique used to estimate a particular solution's generalization performance (i.e., it's performance on previosly unseen data). This Wikipedia article presents an overview of the different methods for cross-validation. Clever methods of over-fitting Over-fitting happens when your estimate of generalization performance is inflated due to error in the experimental design. This blog post presents a nice overview of different ways in which over-fitting can happen.
15	Wed. 2/27		Using the Killdevil Computer Cluster Readings: Trevor Strohman's Indri Hints. Indri Step-by-Step LSF Frequently Asked Questions
16	Mon. 3/4		Temporal Summarization Track (Casey, Mrudula) Readings: Temporal summaries of new topics J. Allan, R. Gupta, V. Khandelwal Evolutionary timeline summarization: a balanced optimization framework via iterative substitution R. Yan, X. Wan, J. Otterbacher, L. Kong, X. Li, Y. Zhang On the value of temporal information in information retrieval O. Alonso, M. Gertz, R. Baeza-Yates An evaluation corpus for temporal summarization V. Khandelwal, R. Gupta, J. Allan
17	Wed. 3/6		Temporal Summarization Track (cont'd)
18	Mon. 3/11	Spring Break (No Class)	No Class
19	Wed. 3/13	Spring Break (No Class)	No Class
20	Mon. 3/18		Crowdsourcing Track (Ashlee and Claudia) Readings: Overview of the TREC 2012 Crowdsourcing Track (Notebook) M. D. Smucker, G. Kazai, M. Lease Skierarchy: Extending the Power of Crowdsourcing Using a Hierarchy of Domain Experts, Crowd and Machine Learning R. Nellapati, S. Peerreddy, P. Singhal Using Hybrid Methods to Reduce Crowd Judgments in TREC '12 Crowd C. Harris, P. Srinivasan Using a Bayesian Model to Combine LDA Features with Crowdsourced Responses E. Simpson, S. Reece, A. Penta, S. D. Ramchurn
21	Wed. 3/20		Crowdsourcing Track (cont'd)
22	Mon. 3/25		Web Track (Diversity Ranking Task) Readings: The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries J. Carbonell, J. Goldstein (Background Reading on Diversity Ranking) Overview of the TREC 2012 Web Track (notebook draft) C. L. A. Clarke, N. Craswell, E. M. Voorhees IRRA at TREC 2012: Divergence From Independence (DFI) B. Taner Dincer LIA at TREC 2012 Web Track: Unsupervised Search Concepts Identification from General Sources of Information R. Deveaud, E. SanJuan, P. Bellot University of Glasgow at TREC 2012: Experiments with Terrier in Medical Records, Microblog, and Web Tracks N. Limsopatham, R. McCreadie, M-Dyaa Albakour, C. Macdonald, R. L. T. Santos, I. Ounis
23	Wed. 3/27		Project Meeting
24	Mon. 4/1		Knowledgebase Acceleration Readings: Paper 1 Paper 2 Paper 3
25	Wed. 4/3		Knowledgebase Acceleration (cont'd)
26	Mon. 4/8		Project Meeting
27	Wed. 4/10		Project Meeting
28	Mon. 4/15		Project Meeting
29	Wed. 4/17		Student Presentations
30	Mon. 4/22		Student Presentations
31	Wed. 4/24		Student Presentations
32	TBD	Final Reports Due

INLS 890: Experimental Informaton Retrieval

INLS 890:
Experimental Informaton Retrieval