Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

MQuAKE benchmark

The MQuAKE benchmark, short for Multi-hop Question Answering for Knowledge Editing, is an evaluation framework designed to test how effectively edited large language models handle multi-step reasoning over modified facts. Rather than merely assessing whether a model can recall an isolated, directly edited piece of information, MQuAKE evaluates whether the model correctly propagates that updated knowledge to answer complex questions whose solutions logically depend on the change. The benchmark incorporates multi-hop question chains based on counterfactual modifications and real-world temporal updates to determine whether knowledge editing methods maintain consistent, logically entailed reasoning across interdependent beliefs.

1 item

MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions

MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions

Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, Danqi Chen

OrganizationsPrinceton UniversityStanford University

Why you should read this

Introduces the MQuAKE benchmark to evaluate whether knowledge-edited language models can propagate updates through multi-hop reasoning, revealing that existing weight-editing methods fail catastrophically and proposing a memory-based prompting alternative, MeLLo, that outperforms them by a large margin.

The information stored in large language models (LLMs) falls out of date quickly, and retraining from scratch is often not an option. This has recently given rise to a range of techniques for injecting new facts through updating model weights. Current evaluation paradigms are extremely limited, mainly validating the recall of edited facts, but changing one fact should cause rippling changes to the model's related beliefs. If we edit the UK Prime Minister to now be Rishi Sunak, then we should get a different answer to Who is married to the British Prime Minister? In this work, we present a benchmark, MQuAKE (Multi-hop Question Answering for Knowledge Editing), comprising multi-hop questions that assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. While we find that current knowledge-editing approaches can recall edited facts accurately, they fail catastrophically on the constructed multi-hop questions. We thus propose a simple memory-based approach, MeLLo, which stores all edited facts externally while prompting the language model iteratively to generate answers that are consistent with the edited facts. While MQuAKE remains challenging, we show that MeLLo scales well with LLMs (up to 175B) and outperforms previous model editors by a large margin.¹

Added

2026-09-26