keyword
retrieval performance
Retrieval performance is the measure of how effectively an information retrieval system identifies, selects, and presents relevant documents or data in response to a user query. It reflects both the capability to locate pertinent material within a collection and the accuracy with which that material is ranked for presentation. While basic evaluations assess retrieval effectiveness using binary criteria such as precision and recall to distinguish relevant items from non-relevant ones, more comprehensive assessments incorporate graded relevance judgments and rank-sensitive metrics like cumulative gain. These advanced measures evaluate how well a system prioritizes the most highly relevant documents near the top of the result list, directly reflecting user satisfaction and utility.
2 items
IR evaluation methods for retrieving highly relevant documents
Kalervo Järvelin, Jaana Kekäläinen
Why you should read this
Introduces discounted cumulative gain (DCG) and cumulative gain metrics to evaluate information retrieval systems using graded, non-binary relevance judgments based on how effectively they prioritize highly relevant documents for users.
This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modern large IR enviroments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (InQuery¹) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.
Added
2026-09-25
Cumulated gain-based evaluation of IR techniques
Kalervo Järvelin, Jaana Kekäläinen
Why you should read this
Establishes the Normalized Discounted Cumulative Gain metric to accurately evaluate search results based on multiple levels of relevance and positional decay.
Modern large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation. In order to develop IR techniques in this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, that is, recall and precision based on binary relevance judgments, to graded relevance judgments. Alternatively, novel measures based on graded relevance judgments may be developed. This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor to the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield. These novel measures are defined and discussed and their use is demonstrated in a case study using TREC data: sample system run results for 20 queries in TREC-7. As a relevance base we used novel graded relevance judgments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, for example, from the user point of view.
Added
2026-05-08