Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

non-dichotomous relevance

Non-dichotomous relevance is an approach in information retrieval where the relationship between a retrieved document and a user query is evaluated along a multi-level or continuous scale rather than as a simple binary choice between relevant and irrelevant. Under this concept, documents are assigned varying degrees of utility or topical match, such as highly relevant, partially relevant, marginally relevant, or irrelevant. This multi-tiered perspective better reflects real-world search behavior and user satisfaction, allowing evaluation frameworks to distinguish between minimally helpful results and exceptionally valuable content. Consequently, it forms the foundation for graded evaluation measures, such as cumulative gain and discounted cumulative gain, which specifically reward retrieval algorithms for ranking the most relevant items at the top of the result list.

1 item

IR evaluation methods for retrieving highly relevant documents

IR evaluation methods for retrieving highly relevant documents

Kalervo Järvelin, Jaana Kekäläinen

OrganizationsUniversity of Tampere

Why you should read this

Introduces discounted cumulative gain (DCG) and cumulative gain metrics to evaluate information retrieval systems using graded, non-binary relevance judgments based on how effectively they prioritize highly relevant documents for users.

This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modern large IR enviroments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (InQuery¹) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.

Added

2026-09-25