Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors

Automated text summarization systems are increasingly deployed across enterprise workflows, but their tendency to generate factual errors presents significant compliance, operational, and reputational risks. While numerous automated factuality metrics have been developed to detect these inaccuracies, evaluation benchmarks have historically combined evaluations across different generations of summarization models. This blending makes it difficult to assess whether modern evaluation tools can reliably detect errors produced by modern summarization systems.

The main objective of the article is to systematically evaluate how modern factuality evaluation metrics perform across summaries generated by different generations of summarization systems and across distinct types of factual errors.

To conduct this evaluation, the authors created AGGREFACT, a standardized benchmark aggregating nine human-annotated datasets across two primary news sources (CNN/DM and XSum). The dataset models were stratified into three evolutionary categories: older pre-Transformer systems, early Transformer models, and state-of-the-art fine-tuned models such as BART, PEGASUS, and T5. Nine factuality metrics—including specialized trained models and large language model prompting approaches using ChatGPT—were evaluated under balanced accuracy conditions alongside a unified taxonomy of intrinsic and extrinsic error types.

The analysis reveals several critical findings. First, reported improvements in factuality detection metrics are largely an illusion of outdated benchmarks; performance gains have occurred primarily on older models rather than modern state-of-the-art summarizers. Second, metric performance degrades substantially on modern models, exhibiting an approximate 10% drop in balanced accuracy from older architectures to fine-tuned state-of-the-art systems on CNN/DM data. Third, no single factuality metric consistently outperforms all others across different datasets or model categories; for example, QuestEval achieved top performance on fine-tuned models for CNN/DM summaries (70.2% balanced accuracy), whereas DAE performed best on fine-tuned XSum summaries (70.2% balanced accuracy). Finally, metrics demonstrate large disparities in identifying specific error types, with recall dropping by 10% to 30% when shifting between different source text distributions.

These findings indicate that organizations relying on aggregate benchmark scores are likely overestimating their ability to automatically catch factual inaccuracies in production deployments. Standard off-the-shelf metrics frequently succeed only at flagging crude, outdated generation errors (such as repetition) while missing the subtle, hallucinated facts produced by modern language models. Consequently, deploying these metrics without domain and model alignment introduces unmonitored risk.

Decision-makers should immediately shift factuality evaluation protocols to focus exclusively on summaries generated by current state-of-the-art models and modern large language models rather than legacy benchmarks. Engineering teams must avoid universal evaluation solutions and instead select specific metrics tailored to their target source domain and predominant error types. Furthermore, organizations should establish continuously updated 'living' benchmarks to track new model iterations as generation capabilities evolve.

These conclusions are bounded by the article's focus on English-language newswire datasets and the reliance on historical human annotations with inherent labeling noise. Confidence is high regarding the performance drop-off of current metrics on modern summarization models, but practitioners should exercise caution and conduct domain-specific pilot validations when applying these tools to specialized environments such as dialogue or medical summarization.