Measuring Search Engine Quality
and Query Difficulty:
Ranking with Target and Freestyle

Robert M. Losee
School of Information and Library Science
Manning Hall, CB#3360
U. of North Carolina-Chapel Hill
Chapel Hill, NC 27599-3360
losee@ils.unc.edu

and

Lee Anne H. Paris
Oklahoma Christian University Library
Oklahoma Christian University
P.O. Box 11000
Oklahoma City, OK 73136
leeanne.paris@oc.edu


Journal of the
American Society for Information Science
50 (10) 1999, 882-889

(pdf of full article)

Abstract:

Instead of using traditional performance measures such as precision and recall, information retrieval performance may be measured by considering the probability that the search engine is optimal and the difficulty associated with retrieving documents with a given query or on a given topic. These measures of desirable characteristics are more easily and more directly interpretable than are traditional measures. The performance of the Target and Freestyle search engines is examined and is very good. Each query in the CF database is assigned a difficulty number, and these numbers are found to strongly correlate with other measures of retrieval performance such as an E or F value. The query difficulty correlates weakly with query length.