FEVER: a Large-scale Dataset for Fact Extraction and VERification

James ThorneAndreas VlachosChristos ChristodoulopoulosArpit Mittal

article2018NAACL2,413 citations

Introduces FEVER, a benchmark dataset of over 185,000 claims that challenges automated fact-checking systems to jointly verify assertions and retrieve supporting textual evidence from Wikipedia.

Listen

The rapid growth of online text and digital information sharing has created an urgent need for automated fact-checking systems across journalism, science, and consumer applications. Existing automated verification benchmarks have historically been constrained to a few hundred claims or provided pre-selected evidence passages, failing to reflect the real-world challenge where systems must actively retrieve and assemble justification from large, open text corpora.

The article introduces and evaluates FEVER (Fact Extraction and VERification), a new large-scale publicly available benchmark designed to evaluate automated systems on extracting supporting or refuting evidence from Wikipedia and classifying claim truthfulness. The authors developed a standardized, multi-step pipeline combining document retrieval, sentence selection, and textual entailment to measure baseline machine performance against human-annotated standards.

The dataset comprises 185,445 claims generated and manually verified against introductory sections of approximately 50,000 Wikipedia pages by a trained 50-member annotation team. Claims were classified into three categories—supported, refuted, or not enough info—with precise sentence-level evidence recorded for the supported and refuted categories. The evaluation pipeline coupled statistical term-matching retrieval tools with a neural decomposable attention entailment model to predict veracity and return justifying sentences.

The analysis yielded several key findings regarding task feasibility and system capabilities. First, the baseline pipeline achieved 31.87% accuracy on the test set when requiring the correct supporting evidence alongside the classification, but achieved 50.91% accuracy when evidence requirements were ignored. Second, component-level oracle evaluations and manual error analyses revealed that information retrieval—specifically selecting the exact justification sentences—is the primary bottleneck, accounting for 58.27% of system failures. Third, multi-hop reasoning is essential: 16.82% of claims require combining multiple sentences as evidence, and 12.15% require evidence spanning multiple distinct documents. Finally, data ablation demonstrated that model accuracy steadily scales with increased training volume, confirming that the dataset is sufficiently large to train complex modern machine learning architectures.

These results indicate that automated claim verification is feasible but remains constrained by retrieval precision. For organizational decision-makers and technology leaders, deploying verification systems without requiring verifiable evidence links poses a substantial risk of producing ungrounded decisions. Unlike traditional natural language inference tasks that assume the reference text is already provided, practical automated verification requires robust end-to-end evidence retrieval to ensure reliability, explainability, and compliance.

To drive progress, researchers and practitioners should prioritize developing advanced retrieval architectures capable of multi-hop sentence selection across multiple documents, rather than focusing solely on classification classifiers. Future initiatives could explore combining automated question generation, neural theorem proving, and reading comprehension models. Before operational deployment in high-stakes domains, automated verification systems should incorporate crowd-sourced human validation to handle ambiguous phrasing, missing evidence paths, and conflicting source information.

The findings are bounded by the dataset reliance on Wikipedia introductory sections and the observation that human annotators did not exhaustively capture every alternate valid piece of evidence, achieving 72.36% recall against unconstrained expert annotators. Nevertheless, strong inter-annotator agreement (a Fleiss kappa score of 0.6841) and high baseline annotation quality (91.2% author-validated precision) provide high confidence that the benchmark offers an effective, rigorous testbed for developing the next generation of fact-checking technologies.

Amazon University of Sheffield verification method Evidence Claims Claim

arXiv: 1803.05355 sheffieldnlp/naacl2018-fever

Paper: Natural Questions: A Benchmark for Question Answering Research, Tom Kwiatkowski et al. (2019). Natural Questions establishes the methodology of extracting questions and evidence directly from Wikipedia logs, serving as a foundational benchmark that FEVER extends to fact verification.
Paper: Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering, Todor Mihaylov et al. (2018). OpenBookQA demonstrates how to construct datasets requiring multi-step fact combination, providing essential context for the evidence-retrieval challenges explored in FEVER.
Paper: Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, Peter Clark et al. (2018). ARC highlights the limitations of surface-level text matching in reasoning benchmarks, directly motivating the rigorous evidence extraction and verification tasks introduced in FEVER.
Paper: BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions, Christopher Clark et al. (2019). BoolQ investigates the complex inference demands of natural user queries, establishing evaluation patterns that inform the claim verification challenges in FEVER.

Paper: HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, Zhilin Yang et al. (2018). HotpotQA builds directly upon the multi-hop evidence extraction paradigm established by FEVER to test complex reasoning across multiple Wikipedia documents.
Paper: BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models, Nandan Thakur et al. (2021). BEIR adopts fact-checking datasets like FEVER into its heterogeneous retrieval benchmark to evaluate how well dense and lexical models generalize in zero-shot settings.
Paper: Text Embeddings by Weakly-Supervised Contrastive Pre-training, Liang Wang et al. (2022). E5 extends retrieval methodologies by applying weakly supervised contrastive pre-training to improve performance on text verification and search tasks pioneered by FEVER.
Paper: Precise Zero-Shot Dense Retrieval without Relevance Labels, Luyu Gao et al. (2023). HyDE leverages zero-shot dense retrieval techniques to improve upon the baseline evidence-finding pipelines initially evaluated in datasets like FEVER.

Cover for FEVER: a Large-scale Dataset for Fact Extraction and VERification

Abstract

In this paper we introduce a new publicly available dataset for verification against textual sources, FEVER: Fact Extraction and VERification. It consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from. The claims are classified as Supported, Refuted or NotEnoughInfo by annotators achieving 0.6841 in Fleiss $\kappa$ . For the first two classes, the annotators also recorded the sentence(s) forming the necessary evidence for their judgment. To characterize the challenge of the dataset presented, we develop a pipeline approach and compare it to suitably designed oracles. The best accuracy we achieve on labeling a claim accompanied by the correct evidence is 31.87%, while if we ignore the evidence we achieve 50.91%. Thus we believe that FEVER is a challenging testbed that will help stimulate progress on claim verification against textual sources.

Topics

verification method

Keywords

Evidence Claims Claim Dataset Cation FEVER VERi Cation Fact Fact Extraction Sentences

1 Introduction
2 Related Works
3 Fact extraction and verification dataset
3.1 Task 1 - Claim Generation
3.2 Task 2 - Claim Labeling
3.3 Annotators
3.4 Data Validation
3.4.1 5-way Agreement
3.4.2 Agreement against Super-Annotators
3.4.3 Validation by the Authors
3.4.4 Findings
4 Baseline System Description
5 Experiments
5.1 Dataset Statistics
5.2 Evaluation
5.3 Document Retrieval
5.4 Sentence Selection
5.5 Recognizing Textual Entailment
5.6 Full Pipeline
5.7 Evaluating Full Pipeline on Test Set
5.8 Manual Error Analysis
5.9 Ablation of Training Data
6 Discussion
7 Conclusions
References
A Annotation Guidelines
A.1 Task 1 Definitions
A.2 Task 1 (subtask 1) Guidelines
A.3 Task 1 (subtask 1) Examples
A.4 Task 1 (substask 2) Guidelines
A.5 Task 1 (subtask 2) Examples
A.6 Task 2 Guidelines
A.7 Task 2 Examples
A.7.1 What does it mean to Support or Refute a claim
A.7.2 Adding Custom Pages
A.7.3 Skipping
A.8 Task 2 additional guidelines
B Manual Error Analysis

Knowls

Knowl 1 — FEVER Dataset Specification and Partitioning

definition

The Fact Extraction and VERification (FEVER) dataset consists of 185,445 claims evaluated against the introductory sections of approximately 50,000 Wikipedia pages from the June 2017 dump. Each claim is labeled with one of three classes:

SUPPORTED: The claim is confirmed by factual evidence present in Wikipedia.
REFUTED: The claim is contradicted by factual evidence present in Wikipedia.
NOTENOUGHINFO: The claim cannot be verified or refuted using the available Wikipedia content.

For claims classified as SUPPORTED or REFUTED, the dataset provides sentence-level evidence annotations identifying the exact combination of sentences required to substantiate the verdict.

Key composition characteristics include:

In $31.75\%$ of claims, more than one sentence was identified as appropriate evidence.
Multi-sentence composition (multi-hop reasoning across sentences) is required for $16.82\%$ of claims.
Evidence spans across multiple distinct Wikipedia articles in $12.15\%$ of claims.
Average claim length is $9.4$ tokens.
On a 5-way inter-annotator agreement evaluation on a $4\%$ sample ( $n = 7,506$ ), annotators achieved Fleiss $\kappa = 0.6841$ .

Split	SUPPORTED	REFUTED	NOTENOUGHINFO	Total
Training	80,035	29,775	35,639	145,449
Development	3,333	3,333	3,333	9,999
Test	3,333	3,333	3,333	9,999
Reserved	6,666	6,666	6,666	19,998
Total	93,367	43,107	48,971	185,445

The Wikipedia articles used to generate claims are partitioned such that every article appears in exactly one split (training, development, test, or reserved).

Knowl 2 — Two-Stage Fact Generation and Mutation Framework
model/method
The FEVER dataset is constructed using a two-stage human annotation protocol designed to prevent trivial overlap between claims and evidence while avoiding unconstrained world knowledge:
1. Claim Generation (Task 1): Annotators receive randomly sampled sentences from Wikipedia introductory sections and generate claims stating a single fact about the target entity. To introduce controlled complexity beyond paraphrasing without relying on external world knowledge, annotators are provided a dictionary comprising hyperlinked terms and the introductory sentences of their corresponding Wikipedia pages. Annotators then generate mutations of each claim using six transformation operators:
  - Paraphrasing
  - Non-trivial negation (discouraging literal insertion of "not")
  - Entity or relation substitution with a similar entity/relation
  - Entity or relation substitution with a dissimilar entity/relation
  - Generalization (making the claim more general)
  - Specialization (making the claim more specific)
2. Claim Labeling and Evidence Selection (Task 2): Independent annotators review the generated and mutated claims without knowing the source sentence. Annotators assign a verdict ( $\text{SUPPORTED}$ , $\text{REFUTED}$ , or $\text{NOTENOUGHINFO}$ ) and select all sentences across Wikipedia introductory sections that constitute the required evidence. Annotators are instructed not to rely on exterior world knowledge and to assign $\text{NOTENOUGHINFO}$ if the available text is insufficient to decisively verify or refute the claim.
Knowl 3 — Fact Verification Evaluation Metrics: Label-Only and Evidence-Aware Accuracy
definition
Evaluation of fact extraction and verification systems on the FEVER dataset uses two primary accuracy metrics:
- Evidence-Aware Verification Accuracy ( $\text{ScoreEv}$ ): A prediction is marked correct if and only if the predicted 3-way label ( $\text{SUPPORTED}$ , $\text{REFUTED}$ , or $\text{NOTENOUGHINFO}$ ) matches the gold label and, for claims labeled $\text{SUPPORTED}$ or $\text{REFUTED}$ , the system retrieves a complete, valid set of sentence-level evidence. If a claim requires multi-hop reasoning spanning multiple sentences, all sentences in at least one gold evidence set must be selected. For claims labeled $\text{NOTENOUGHINFO}$ , no evidence is required.
- Label-Only Verification Accuracy ( $\text{NoScoreEv}$ ): Standard 3-way classification accuracy assessing only the predicted verdict ( $\text{SUPPORTED}$ , $\text{REFUTED}$ , or $\text{NOTENOUGHINFO}$ ), completely disregarding whether the retrieved evidence was correct. On balanced evaluation splits, a random guess baseline achieves approximately $33.33\%$ .
Additionally, sentence retrieval quality is evaluated using sentence-level Precision, Recall, and $F_1$ score against human-annotated evidence, as well as the proportion of claims for which a fully supporting or refuting evidence set was retrieved.
Knowl 4 — Modular Three-Stage Fact Verification Pipeline
model/method
The baseline architecture for end-to-end fact verification processes an input claim through three sequential components:
1. Document Retrieval: Identifies the top $k$ relevant Wikipedia articles from the corpus. It computes cosine similarity over binned unigram and bigram Term Frequency–Inverse Document Frequency (TF-IDF) vectors between the claim and the document introductory sections (adapted from the DrQA system).
2. Sentence Selection: Ranks all sentences within the top $k$ retrieved documents by TF-IDF similarity to the claim and selects the top $l$ candidate sentences.
3. Recognizing Textual Entailment (RTE): Takes the concatenated string of the top $l$ selected sentences as the premise and the claim as the hypothesis to perform 3-way classification into $\text{SUPPORTED}$ , $\text{REFUTED}$ , or $\text{NOTENOUGHINFO}$ .
Optimal hyperparameters determined via grid search on the development set are $k = 5$ retrieved documents and $l = 5$ selected sentences.
Knowl 5 — Synthetic Evidence Generation for Uninformative Claims in RTE Training
model/method
Because claims with the label $\text{NOTENOUGHINFO}$ lack human-annotated supporting or refuting evidence sentences in the FEVER dataset, synthetic premise-hypothesis pairs must be constructed to train 3-way Recognizing Textual Entailment (RTE) models. Two sampling strategies are used:
- Random Sampling ( $\text{RANDOMS}$ ): Selects a sentence uniformly at random from the entire Wikipedia corpus to serve as the pseudo-evidence paired with the $\text{NOTENOUGHINFO}$ claim. This generates premises that are largely semantically irrelevant to the claim.
- Nearest-Page Sampling ( $\text{NEARESTP}$ ): Selects a sentence from the highest-ranked Wikipedia article returned by the document retrieval component for that claim. This provides pseudo-evidence that is topically related to the claim but insufficient to confirm or refute it, training the RTE classifier to recognize related yet uninformative context.

Knowl 6 — End-to-End Pipeline Performance on FEVER

empirical result

The baseline fact verification pipeline using DrQA document retrieval ( $k=5$ ), DrQA sentence selection ( $l=5$ ), and a Decomposable Attention (DA) RTE model achieves the following verification accuracies on the FEVER development and test sets:

Model	Training Strategy	NoScoreEv Accuracy (%)	ScoreEv Accuracy (%)
Development Set
MLP	NEARESTP (NP)	41.86	19.04
MLP	RANDOMS (RS)	40.63	19.42
DA	NEARESTP (NP)	52.09	32.57
DA	RANDOMS (RS)	50.37	23.53
Test Set
DA	NEARESTP (NP)	50.91	31.87

On the test set, the best-performing pipeline ( $\text{DA} / \text{NP}$ ) achieves an evidence-aware verification accuracy ( $\text{ScoreEv}$ ) of $31.87\%$ and a label-only verification accuracy ( $\text{NoScoreEv}$ ) of $50.91\%$ . For claims requiring evidence on the test set, the document and sentence retrieval modules achieve an evidence recall of $45.89\%$ , an evidence precision of $10.79\%$ , and an evidence $F_1$ score of $17.47\%$ .

The performance gap between $\text{NoScoreEv}$ ( $50.91\%$ ) and $\text{ScoreEv}$ ( $31.87\%$ ) demonstrates that in many instances where the classifier predicts the correct label, it does so using incorrect or incomplete evidence.

Knowl 7 — Oracle Evidence Upper Bounds for Retrieval and Entailment

data/table

Oracle evaluations isolate the upper-bound performance of individual pipeline stages on the FEVER development set:

Documents ( $k$ )	Fully Supported Claims (%)	Oracle Accuracy (%)
1	25.31	50.21
5	55.30	70.20
10	65.86	77.24
25	75.92	83.95
50	82.49	90.13
100	86.59	91.06

When isolating sentence selection with $k = 5$ retrieved documents and $l = 5$ selected sentences:

DrQA bigram TF-IDF sentence selection retains complete evidence for $44.22\%$ of claims (oracle accuracy $62.81\%$ ).
NLTK unigram TF-IDF sentence selection retains complete evidence for $34.03\%$ of claims (oracle accuracy $56.02\%$ ).

When isolating the RTE component by supplying gold-standard evidence sentences:

These results demonstrate that evidence retrieval (specifically sentence selection) represents the primary bottleneck in automated fact verification compared to textual entailment classification.

Knowl 8 — Ablation of Sentence Selection in Verification Pipelines

empirical result

Removing the sentence selection component and passing entire retrieved Wikipedia documents ( $k=5$ ) directly to the Recognizing Textual Entailment (RTE) module degrades performance in both oracle and pipeline settings on the development set:

Setting / Model	NEARESTP (%)	RANDOMS (%)
Oracle Accuracy (Gold Documents)
MLP	57.16	73.36
DA	63.68	69.05
Pipeline NoScoreEv Accuracy ( $k=5$ Docs)
MLP	38.85	40.45
DA	41.57	40.62

In the oracle setting using gold documents, Decomposable Attention (DA) accuracy drops by $17.14$ percentage points (from $80.82\%$ with gold sentences to $63.68\%$ with gold documents) under the $\text{NEARESTP}$ regime. In the end-to-end pipeline setting, removing sentence selection decreases $\text{NoScoreEv}$ accuracy by approximately $10.52$ percentage points (from $52.09\%$ to $41.57\%$ ). Sentence-level filtering is therefore critical for attention-based entailment models to focus on relevant factual evidence and ignore extraneous document context.

Knowl 9 — Test-Set Error Breakdown for Fact Verification Pipelines
empirical result
A manual error analysis of $961$ test-set predictions produced by the Decomposable Attention pipeline ( $\text{DA}/\text{NP}$ ) that failed the evidence-aware evaluation metric ( $\text{ScoreEv}$ ) revealed the following distribution of error causes:
- Information Retrieval Failure ( $58.27\%$ , $n = 560$ ): The document and sentence retrieval components failed to retrieve any valid evidence sentences, representing the dominant cause of verification failure.
- Evidence Formatting / Incompleteness ( $28.51\%$ , $n = 274$ ): The system predicted the correct 3-way verdict, but failed to return the complete set of required gold evidence sentences.
- RTE Classification Failure ( $13.84\%$ , $n = 133$ ): Valid supporting or refuting evidence was successfully retrieved, but the entailment model misclassified the claim.
- Unannotated Alternative Evidence ( $21.85\%$ , $n = 210$ ): In over one-fifth of sampled errors, the retrieval system found valid supporting/refuting evidence from Wikipedia that human annotators had not marked.
- Annotation / Claim Noise ( $4.05\%$ , $n = 41$ ): Claims violated dataset guidelines (such as tautologies, self-contradictions, or parsing errors). Incorrect gold labels assigned by annotators accounted for only $1.35\%$ ( $n = 13$ ) of incorrectly classified claims.
- Wikipedia Inconsistency / Contradiction ( $0.52\%$ , $n = 5$ ): The system retrieved evidence that contradicted the gold evidence due to conflicting facts across different Wikipedia pages.
Knowl 10 — Sample Complexity and Scaling Behavior of Verification Entailment Models
empirical result
Empirical evaluation of learning curves on the FEVER dataset reveals substantial differences in sample complexity between linear-feature models and deep attention models:
- For training set sizes below $6,000$ instances, the test-set oracle accuracy of the Decomposable Attention (DA) model is unstable and underperforms the Multi-Layer Perceptron (MLP) baseline.
- With more than $6,000$ training instances, the accuracy of DA scales logarithmically with the number of training examples, consistently outpacing the MLP model and reaching over $80\%$ oracle accuracy at $10^5$ instances (compared to approximately $67\%$ for MLP).
This demonstrates that large-scale datasets (on the order of $10^5$ claims) are necessary to train attention-based textual entailment architectures for fact extraction and verification.

Coverage note — Omitted specific mechanical UI design details and standard tokenization preprocessing scripts, as they represent implementation tooling rather than primary research contributions.

References

1.Gabor Angeli and Christopher D. Manning. 2014. NaturalLI: Natural logic inference for common sense reasoning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. pages 534–545.
2.Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
3.Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 1870–1879. https://doi.org/10.18653/v1/P17-1171.
4.Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering 15(4):i–xvii. https://doi.org/10.1017/S1351324909990209.
5.Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie Strassel. 2016. Overview of Linguistic Resources for the TAC KBP 2016 Evaluations : Methodologies and Results. Proceedings of TAC KBP 2016 Workshop, National Institute of Standards and Technology, Maryland, USA (Ldc).
6.William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego, California, pages 1163–1168.
7.Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76(5):378.
8.Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2017. AllenNLP: A Deep Semantic Natural Language Processing Platform .
9.Michael Heilman and Noah A. Smith. 2010. Good Question! statistical ranking for question generation. In Proceedings of the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. pages 609–617.
10.Mio Kobayashi, Ai Ishii, Chikara Hoshino, Hiroshi Miyashita, and Takuya Matsuzaki. 2017. Automated historical fact-checking by passage retrieval, word statistics, and virtual question-answering. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). volume 1, pages 967–975.
11.Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics - Volume 1. Association for Computational Linguistics, Stroudsburg, PA, USA, ETMTNLP ’02, pages 63–70. https://doi.org/10.3115/1118108.1118117.
12.Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL (System Demonstrations). pages 55–60.
13.Mausam, Michael Schmitz, Robert Bart, Stephen Soderland, and Oren Etzioni. 2012. Open language learning for information extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. pages 523–534.
14.Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2249–2255. https://aclweb.org/anthology/D16-1244.
15.Dean Pomerleau and Delip Rao. 2017. Fake news challenge. http://fakenewschallenge.org/.
16.P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP).
17.Benjamin Riedel, Isabelle Augenstein, George Spithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the Fake News Challenge stance detection task. CoRR abs/1707.03264. http://arxiv.org/abs/1707.03264.
18.Tim Rocktäschel and Sebastian Riedel. 2017. End-to-end differentiable proving. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, California, United States. volume abs/1705.11040. http://arxiv.org/abs/1705.11040.
19.Álvaro Rodrigo, Anselmo Peñas, and Felisa Verdejo. 2009. Overview of the answer validation exercise 2008. In Carol Peters, Thomas Deselaers, Nicola Ferro, Julio Gonzalo, Gareth J. F. Jones, Mikko Kurimo, Thomas Mandl, Anselmo Peñas, and Vivien Petras, editors, Evaluating Systems for Multilingual and Multimodal Information Access: 9th Workshop of the Cross-Language Evaluation Forum. pages 296–313.
20.Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. CoRR abs/1611.01603. http://arxiv.org/abs/1611.01603.
21.Craig Silverman. 2015. Lies, Damn Lies and Viral Content. http://towcenter.org/research/lies-damn-lies-and-viral-content/.
22.Niket Tandon, Gerard de Melo, and Gerhard Weikum. 2011. Deriving a Web-scale common sense fact database. In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI 2011). AAAI Press, Palo Alto, CA, USA, pages 152–157.
23.Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science. ACL. http://www.aclweb.org/anthology/W14-2508.
24.William Yang Wang. 2017. “‘Liar, Liar Pants on Fire’”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). http://aclweb.org/anthology/P17-2067.

Citation

MLA

Thorne, J., et al. “FEVER: A Large-scale Dataset for Fact Extraction and VERification”. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018, pp. 809–19, https://doi.org/10.18653/v1/N18-1074.

APA

Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018). FEVER: a Large-scale Dataset for Fact Extraction and VERification. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 809–819. https://doi.org/10.18653/v1/N18-1074

Chicago

Thorne, J., A. Vlachos, C. Christodoulopoulos, and A. Mittal. 2018. “FEVER: A Large-scale Dataset for Fact Extraction and VERification”. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 809–19. https://doi.org/10.18653/v1/N18-1074.

Harvard

Thorne, J. et al. (2018) “FEVER: a Large-scale Dataset for Fact Extraction and VERification”, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, pp. 809–819. Available at: https://doi.org/10.18653/v1/N18-1074.

Vancouver

1. Thorne J, Vlachos A, Christodoulopoulos C, Mittal A (2018) FEVER: a Large-scale Dataset for Fact Extraction and VERification. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, pp 809–819

BibTeX

@inproceedings{Thorne_2018, title={FEVER: a Large-scale Dataset for Fact Extraction and VERification}, url={http://dx.doi.org/10.18653/v1/N18-1074}, DOI={10.18653/v1/n18-1074}, booktitle={Proceedings of the 2018 Conference of the North American Chapter of
          the Association for Computational Linguistics: Human Language
          Technologies, Volume 1 (Long Papers)}, publisher={Association for Computational Linguistics}, author={Thorne, James and Vlachos, Andreas and Christodoulopoulos, Christos and Mittal, Arpit}, year={2018}, pages={809–819} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

RTE Model	NEARESTP Accuracy (%)	RANDOMS Accuracy (%)	SNLI Pre-trained (%)
MLP	65.13	73.81	–
DA	80.82	88.00	38.54

FEVER: a Large-scale Dataset for Fact Extraction and VERification

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — FEVER Dataset Specification and Partitioning

Knowl 2 — Two-Stage Fact Generation and Mutation Framework

Knowl 3 — Fact Verification Evaluation Metrics: Label-Only and Evidence-Aware Accuracy

Knowl 4 — Modular Three-Stage Fact Verification Pipeline

Knowl 5 — Synthetic Evidence Generation for Uninformative Claims in RTE Training