Performance Superiority over a Range of Situations

Retrieval performance may be measured at a single point, as was done above, or performance might be measured over a range of values for a variable. For example, examining the performance characteristics over a range of cutoffs from 2 to N, the number of documents in the database, can show how a given search engine performs at different points in the search process. A search engine optimized for high-precision lower-recall searches, for example, might have a higher Q at cutoff 10 than at cutoff N. These variations may be computed experimentally, computing performance at individual cutoff points, or analytically, showing through proof methods that one retrieval engine is superior to another over a range of cutoffs or other values.

  

Figure 1: Q values at varying cutoffs for different retrieval engines.

Figure 1 shows the Q values computed for 4 retrieval engines. There is clearly noise in the figures for low cutoffs, with trends only appearing with higher cutoffs, where larger amounts of data are available for computation.

  

Figure 2: Q values at varying cutoffs for different retrieval engines.

Figure 2 shows in more detail the Q values for low cutoffs. The Target 2 retrieval engine, which performed very well for larger cutoffs, shows a lower level of performance for small cutoffs, suggesting that it is probably better for higher recall searches.