next up previous
Next: Commercial Retrieval Systems and Up: Measuring Search Engine Quality Previous: Measuring Search Engine Quality

Introduction

Journal articles and conference presentations often describe how one retrieval or search engine was found to be better than another. These comparisons often consist of the presentation of a particular retrieval measure computed for a set of queries that are part of one or more experimental databases. In this study, we introduce a method that can isolate two important factors in the comparison of retrieval results. Below we suggest a method for isolating the intrinsic difficulty associated with using a query and retrieving documents up to a certain point in the search. We also suggest a method for computing the quality of a retrieval or search engine that is independent, in many senses, of the experimental databases used. The two measures used here differ from those commonly used to evaluate retrieval systems [HH97]. While these measures are computed using some time-consuming techniques that would be virtually impossible to compute with paper and pencil, the results are very easily interpretable: one being a direct measure of query difficulty and the other being a measure of the probability that the search engine will produce optimal ranking. The values of the measures are more easily understood and more directly relevant to the needs of those evaluating retrieval systems than most traditional retrieval performance measures such as precision and recall.
next up previous
Next: Commercial Retrieval Systems and Up: Measuring Search Engine Quality Previous: Measuring Search Engine Quality
Bob Losee
1999-07-29