Next: Query Difficulty and Correlates
Up: Measuring Search Engine Quality
Previous: Comparing Retrieval or Search
Retrieval performance may be measured at a single point, as was done above, or performance might be measured over a range of values for a variable.
For example, examining the performance characteristics over a range of cutoffs from 2 to N, the number of documents in the database, can show how a given search engine performs at different points in the search process.
A search engine optimized for high-precision lower-recall searches, for example, might have a higher Q at cutoff 10 than at cutoff N.
These variations may be computed experimentally, computing performance at individual cutoff points, or analytically, showing through proof methods that one retrieval engine is superior to another over a range of cutoffs or other values.
Figure 1:
Q values at varying cutoffs for different retrieval engines.
|
Figure 1 shows the Q values computed for 4 retrieval
engines. There is clearly noise in the figures for low cutoffs, with trends
only appearing with higher cutoffs, where larger amounts of data are
available for computation.
Figure 2:
Q values at varying cutoffs for different retrieval engines.
|
Figure 2 shows in more detail the Q values for low
cutoffs.
The Target 2 retrieval engine, which performed very well for larger cutoffs,
shows a lower level of performance for small cutoffs, suggesting that it is
probably better for higher recall searches.
Next: Query Difficulty and Correlates
Up: Measuring Search Engine Quality
Previous: Comparing Retrieval or Search
Bob Losee
1999-07-29