Back

The use of MMR, diversity-based reranking for reordering documents and producing summaries

Jaime CarbonellJade Goldstein

article1998SIGIR1,876 citationsSIGIR Test of Time Award

Introduces the Maximal Marginal Relevance (MMR) criterion to balance query relevance with information novelty, providing an effective framework for minimizing redundancy in document retrieval and multi-document text summarization.

Listen

As digital information continues to expand rapidly, users frequently encounter large sets of search results that contain repetitive or near-identical content. Standard search and retrieval systems rank items solely by direct relevance to a search query, which often forces users to spend valuable time reviewing redundant text rather than discovering new insights.

The article evaluates a ranking method designed to balance topical relevance with information novelty, aiming to minimize redundancy across retrieved items and automatically generated summaries.

The approach introduces a decision criterion called Maximal Marginal Relevance, which balances query relevance with dissimilarity to items already selected. The analysis demonstrates this method across several settings: a small user study evaluating document retrieval preferences, a multi-system industry evaluation of automated single-document summaries, and controlled passage extraction experiments across single- and multi-document sets.

The findings show strong operational benefits. In a user test, 80 percent of participants preferred the diversity-based ranking method for exploration tasks because it presented a broader range of topics more quickly. In formal summarization benchmarks against 15 competing systems, the approach achieved top performance with a 0.73 relevance F-score and a 70 percent accuracy rate on answering key questions from summaries. Controlled tests on passage extraction showed that incorporating diversity reduced redundancy without sacrificing sentence-level precision, while the strongest improvements occurred in multi-document summarization, where duplicate and overlapping information across news stories was substantially reduced.

These results demonstrate that balancing novelty and relevance helps users navigate information spaces faster, reducing cognitive load and time wasted reading duplicate passages. The tunable parameter allows organizations to adapt search and summarization tools either for broad, panoramic discovery or for narrow, focused analysis.

Organizations developing or deploying search and summarization platforms should consider incorporating novelty-aware ranking into their retrieval workflows. An effective recommended search pattern is to present diverse results initially to help users map a topic, followed by higher-precision relevance ranking once users refine their queries.

While the results are promising, the user evaluation relies on a very small sample size, and benchmark summaries varied in length across competing systems. Consequently, stakeholders should view these comparative gains as strong preliminary indicators and conduct larger-scale user trials and domain-specific pilot testing before broad deployment.

Carnegie Mellon Universityelectronic documentgenetic selectionmarginalsdiversity-based rerankingmaximal marginal relevancesummarization systems
Carbonell et al (1998).pdf
  • Paper: A trainable document summarizer, J. Kupiec et al. (1995). Read this early trainable extraction-based summarizer first to see the sentence-selection problem that MMR later addresses by explicitly reducing redundancy.
Cover for The use of MMR, diversity-based reranking for reordering documents and producing summaries

Abstract

This paper presents a method for combining query-relevance with information-novelty in the context of text retrieval and summarization. The Maximal Marginal Relevance (MMR) criterion strives to reduce redundancy while maintaining query relevance in re-ranking retrieved documents and in selecting appropriate passages for text summarization. Preliminary results indicate some benefits for MMR diversity ranking in document retrieval and in single document summarization. The latter are borne out by the recent results of the SUMMAC conference in the evaluation of summarization systems. However, the clearest advantage is demonstrated in constructing non-redundant multi-document summaries, where MMR results are clearly superior to non-MMR passage selection.

Topics

electronic documentgenetic selectionmarginalstext documentsentence extractionautomatic summarizationquestion answering

Keywords

diversity-based rerankingmaximal marginal relevancesummarization systemsreordering documents

Table of Contents

  • 1 Introduction
  • 2 Maximal Marginal Relevance
  • 3 Document Reordering
  • 4 Summarization
  • 5 Concluding Remarks
  • References

Knowls

  1. Knowl 1 — Maximal Marginal Relevance Criterion for Information Retrieval and Reranking

    equation

    Maximal Marginal Relevance (MMR) is a criterion designed to balance query relevance and information novelty when iteratively selecting documents or passages from a candidate set. Given a document collection or stream CC, a query or user profile QQ, an initial ranked set of retrieved candidates R=IR⁡(C,Q,θ)R = \operatorname{IR}(C, Q, \theta) filtered by a relevance threshold θ\theta, and the subset of candidates already selected S⊆RS \subseteq R, the next item Di∈R∖SD_i \in R \setminus S is chosen by:

    MMR⁡=defArg max⁡Di∈R∖S[λSim⁡1(Di,Q)−(1−λ)max⁡Dj∈SSim⁡2(Di,Dj)]\operatorname{MMR} \stackrel{\text{def}}{=} \operatorname*{Arg\,max}_{D_i \in R \setminus S} \left[ \lambda \operatorname{Sim}_1(D_i, Q) - (1 - \lambda) \max_{D_j \in S} \operatorname{Sim}_2(D_i, D_j) \right]

    where:

    • Sim⁡1(Di,Q)\operatorname{Sim}_1(D_i, Q) is a similarity metric measuring relevance between an unselected document or passage DiD_i and the query QQ (e.g., cosine similarity).
    • Sim⁡2(Di,Dj)\operatorname{Sim}_2(D_i, D_j) is a similarity metric measuring redundancy/similarity between candidate DiD_i and an already selected candidate DjD_j (which can be identical to Sim⁡1\operatorname{Sim}_1 or a different metric).
    • λ∈[0,1]\lambda \in [0, 1] is a tunable interpolation parameter. Setting λ=1\lambda = 1 computes a standard relevance ranking, while λ=0\lambda = 0 computes a maximal diversity ranking among the documents in RR.

  2. Knowl 2 — Two-Stage Search and Exploration Strategy via Marginal Relevance Tuning

    model/method

    In the Maximal Marginal Relevance (MMR) framework, the interpolation parameter λ∈[0,1]\lambda \in [0, 1] controls the trade-off between broad sampling and focused drilling:

    • Smaller values of λ\lambda (e.g., λ=0.3\lambda = 0.3) maximize diversity among retrieved items, producing a panoramic sampling of the information space surrounding the query.
    • Larger values of λ\lambda closer to 11 (e.g., λ=0.7\lambda = 0.7) emphasize relevance, allowing the user to focus on multiple overlapping or mutually reinforcing documents.

    An effective interactive search workflow uses a two-stage strategy:

    1. Exploration phase: Begin with a low value of λ\lambda (e.g., λ=0.3\lambda = 0.3) to understand the diverse aspects of the information space around an initial query.
    2. Focus phase: Reformulate the query (optionally using relevance feedback) and select a higher value of λ\lambda (e.g., λ=0.7\lambda = 0.7) to drill down on specific, important subtopics.

  3. Knowl 3 — Query-Relevant Extractive Summarization via Maximal Marginal Relevance

    model/method

    Extractive text summarization using Maximal Marginal Relevance (MMR) selects passages that are relevant to a query while minimizing redundancy:

    • Single-Document Summarization: A document is segmented into passages (e.g., sentences). MMR reranks the passages using cosine similarity to maximize relevance to a user-generated or system-generated query while penalizing similarity to already chosen passages. The top-ranked passages are then output in their original document order.
    • Multi-Document Summarization: When creating summaries across multiple documents on the same topic, passages are pooled across documents. News collections frequently contain repeated background sentences; MMR filters out near-duplicate and redundant passages across sources, producing a consolidated summary.

    MMR-based passage selection is especially advantageous for longer documents (which contain structural redundancy across sections such as abstracts, introductions, and conclusions) and multi-document clusters.

  4. Knowl 4 — Sentence Precision Under Varying Diversity Weighting and Compression Rates

    data/table

    To evaluate the trade-off between diversity gain and relevance loss in single-document summarization, 50 articles from a 200-article TIPSTER topic were evaluated at compression ratios of 10%10\% and 25%25\% across various λ\lambda settings, compared against a lead-sentence baseline.

    Precision was measured under two relevance benchmarks:

    1. TREC and CMU Relevant: The document was judged relevant by TREC and at least one CMU assessor (23 documents).
    2. CMU Relevant: At least two of three CMU assessors judged the document as relevant (18 documents).
    Document Percentage λ\lambda TREC and CMU Relevant CMU Relevant
    10% 1.0 0.78 0.83
    10% 0.7 0.76 0.83
    10% 0.3 0.74 0.79
    10% Lead Sentences 0.74 0.83
    25% 1.0 0.74 0.76
    25% 0.7 0.73 0.74
    25% 0.3 0.74 0.76
    25% Lead Sentences 0.60 0.65

    The differences in sentence precision between λ=1.0\lambda = 1.0, λ=0.7\lambda = 0.7, and λ=0.3\lambda = 0.3 are not statistically significant. In certain cases, lower λ\lambda settings reranked and captured relevant information that pure relevance ranking (λ=1.0\lambda = 1.0) missed due to redundancy, showing that diversity can be enhanced without sacrificing precision. At 25%25\% compression, all MMR configurations substantially outperformed the lead-sentence baseline (0.730.73--0.760.76 vs. 0.600.60--0.650.65).

  5. Knowl 5 — Performance of MMR Extractive Summarizer in the SUMMAC Evaluation

    empirical result

    In the May 1998 SUMMAC government-run evaluation comparing 15 summarization systems:

    • The MMR-based summarizer achieved the highest utility score for query-relevant summaries, reaching an F-score of 0.730.73 based on precision and recall from assessors making topic-relevance judgments directly from summaries.
    • The system also scored highest on informative summaries, reaching 70%70\% accuracy when assessors judged whether summaries contained the required information to answer key questions.

  6. Knowl 6 — User Evaluation of Diversity-Based Document Reranking

    empirical result

    In a blind pilot study with 5 undergraduate users comparing standard relevance ranking against Maximal Marginal Relevance (MMR) reranking:

    • A majority of users preferred MMR because it provided a broader and more interesting set of topics.
    • When prompted to choose a ranking method to perform a search task, 80%80\% (4 of 5 users) selected MMR.
    • Users reported a distinct operational utility: MMR was preferred for initial navigation and fast discovery of candidate document regions, while pure relevance ranking was preferred when inspecting related documents within a single narrow band.

Coverage note — No substantial contributed material was omitted; all core definitions, search strategies, summarization procedures, and empirical evaluations are represented.

References

  1. 1.Buckley C. Implementation of the smart information retrieval system. Technical Report TR 85-686, Cornell University.
  2. 2.J.G.Carbonell, Y. Geng, and J. Goldstein. Automated query-relevant summarization and diversity-based reranking. In 15th International Joint Conference on Artificial Intelligence, Workshop: AI in Digital Libraries, pages 9–14, Nagoya, Japan, August 1997.
  3. 3.J.M. Kupiec, J. Pedersen, and F. Chen. A trainable document summarizer. In Proceedings of the 18th Annual Int. ACM/SIGIR Conference on Research and Development in IR, pages 68–73, Seattle, WA, July 1995.
  4. 4.P.H. Luhn. Automatic creation of literature abstracts. IBM Journal, pages 159–165, 1958.
  5. 5.G. Salton. Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer. Addison-Wesley, 1989.
  6. 6.In TIPSTER Text Phase III 18-Month Workshop, Fairfax, VA, May 1998.

Citation

MLA
Carbonell, J., and J. Goldstein. “The Use of MMR, Diversity-based Reranking for Reordering Documents and Producing Summaries”. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 1998, pp. 335–36, https://doi.org/10.1145/290941.291025.
APA
Carbonell, J., & Goldstein, J. (1998). The use of MMR, diversity-based reranking for reordering documents and producing summaries. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 335–336. https://doi.org/10.1145/290941.291025
Chicago
Carbonell, J., and J. Goldstein. 1998. “The Use of MMR, Diversity-based Reranking for Reordering Documents and Producing Summaries”. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 335–36. https://doi.org/10.1145/290941.291025.
Harvard
Carbonell, J. and Goldstein, J. (1998) “The use of MMR, diversity-based reranking for reordering documents and producing summaries”, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, pp. 335–336. Available at: https://doi.org/10.1145/290941.291025.
Vancouver
1. Carbonell J, Goldstein J (1998) The use of MMR, diversity-based reranking for reordering documents and producing summaries. In: Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, pp 335–336

BibTeX

@inproceedings{Carbonell_1998, series={SIGIR98}, title={The use of MMR, diversity-based reranking for reordering documents and producing summaries}, url={http://dx.doi.org/10.1145/290941.291025}, DOI={10.1145/290941.291025}, booktitle={Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval}, publisher={ACM}, author={Carbonell, Jaime and Goldstein, Jade}, year={1998}, month=Aug, pages={335–336}, collection={SIGIR98} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF