Language Models as Knowledge Bases?

Fabio PetroniTim RocktäschelPatrick LewisAnton BakhtinYuxiang WuAlexander H. MillerSebastian Riedel

article2019EMNLP3,304 citations

Demonstrates that pretrained language models store substantial relational facts, enabling them to function as queryable knowledge bases for open-domain question answering without fine-tuning or schema engineering.

Listen

Recent advances in language models pretrained on large text collections have produced strong results on many natural-language tasks, yet it remains unclear how much factual and commonsense knowledge these models already contain. The paper examines whether such models can act as knowledge bases by answering fill-in-the-blank queries without any task-specific training or structured schemas.

The authors introduce the LAMA probe, which converts facts from Wikidata, ConceptNet, Google relation-extraction data, and a subset of SQuAD into simple cloze templates and measures how highly each model ranks the correct object among roughly 21,000 candidate tokens. They evaluate six publicly available models, ranging from convolutional and recurrent networks to BERT-base and BERT-large, and compare them against a frequency baseline, a supervised relation extractor with both naïve and oracle entity linking, and the DrQA open-domain question-answering system.

BERT-large emerges as the strongest model. On the T-REx subset it reaches 32 percent mean precision at one, nearly matching an oracle-aided relation extractor, and on the cloze version of SQuAD it attains 57 percent precision at ten, close to DrQA’s 64 percent. Performance is highest for one-to-one relations and lower for many-to-many relations; the model is also more stable across different phrasings of the same fact than earlier architectures. These results hold even though the language models receive no fine-tuning and no explicit retrieval step.

The findings indicate that large-scale language-model pretraining already encodes substantial relational knowledge that can be accessed simply by prompting. This capability could reduce reliance on brittle extraction pipelines and fixed schemas, while also supporting unsupervised open-domain question answering. At the same time, accuracy varies markedly by relation type, and models may partly succeed by exploiting co-occurrence patterns rather than deeper understanding.

Future work should test newer models with LAMA, explore more varied natural-language prompts, and extend evaluation to multi-token answers. Until those extensions are completed, practitioners should treat the reported figures as lower bounds and verify high-stakes facts with conventional knowledge bases.

Meta University College London engineering knowledge base language model Language Model Knowledge Base BERT

arXiv: 1909.01066 facebookresearch/LAMA

Paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin et al. (2019). Reading BERT provides the foundational understanding of bidirectional transformer pre-training that the source paper evaluates for relational knowledge retrieval.

Paper: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Patrick Lewis et al. (2020). This work directly extends the source by coupling a generative language model with a retriever to explicitly solve knowledge-intensive tasks.

Cover for Language Models as Knowledge Bases?

Abstract

Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fill-in-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at this https URL.

Topics

engineering knowledge base language model qa system

Keywords

Language Model Knowledge Base BERT LAMA Probe Relational Knowledge Cloze Query WikiText-103 ConceptNet Open-Domain QA Precision@10

1 Introduction
2 Background
2.1 Unidirectional Language Models
2.2 Bidirectional “Language Models” $ ^{4} $
3 Related Work
4 The LAMA Probe
4.1 Knowledge Sources
4.1.1 Google-RE
4.1.2 T-REx
4.1.3 ConceptNet
4.1.4 SQuAD
4.2 Models
4.3 Baselines
4.4 Metrics
4.5 Considerations
5 Results
6 Discussion and Conclusion
Acknowledgments
References

Knowls

Knowl 1 — The LAMA (LAnguage Model Analysis) Probe for Factual Knowledge Extraction
model/method
The LAMA (LAnguage Model Analysis) probe evaluates the extent to which pretrained language models store factual and commonsense relational knowledge in their parameters without task-specific fine-tuning.

A factual assertion is framed as a subject-relation-object triple $(s, r, o)$ or a question-answer pair. For each relation $r$ , a cloze sentence template is defined with the object slot masked (for example, the triple $(\text{Dante}, \text{born-in}, \text{Florence})$ is converted to $\text{\"Dante was born in [MASK].\"}$ ). The language model is queried to rank candidate tokens for the masked position based on its conditional probability distribution.

To ensure fair ranking comparisons across models with distinct native token vocabularies, candidate generation is evaluated over a shared vocabulary $V_{\text{shared}} \approx 21,000$ case-sensitive tokens, defined as the intersection of the vocabularies of all evaluated models. Targets are restricted to single-token objects.

Model knowledge is evaluated using Mean Precision at $k$ ( $P@k$ ). For a fact with target token $o$ , $P@k = 1$ if $o$ appears in the top $k$ ranked tokens of $V_{\text{shared}}$ , and $0$ otherwise. When a subject-relation pair $(s, r)$ has multiple valid objects in the corpus (as in one-to-many or many-to-many relations), all other valid objects in the dataset for that pair except the test object $o$ are removed from the candidate set prior to ranking.
Knowl 2 — Knowledge Sources in the LAMA Benchmark
experimental setup
The LAMA benchmark evaluates relational knowledge across four distinct knowledge sources comprising a total of 51,329 factual queries:
1. Google-RE: A manually curated corpus extracted from Wikipedia containing 5,527 facts across 3 relations with single-token objects: "place of birth" (2,937 facts), "date of birth" (1,825 facts), and "place of death" (765 facts). Each fact is manually aligned to a supporting Wikipedia text passage.
2. T-REx: A large-scale subset of Wikidata containing 34,039 facts across 41 relations (subsampled up to 1,000 facts per relation), automatically aligned to Wikipedia sentences. The relations are categorized by cardinality: 1-to-1 relations (2 relations, 937 facts), N-to-1 relations (23 relations, 20,006 facts), and N-to-M relations (16 relations, 13,096 facts).
3. ConceptNet: A commonsense knowledge base comprising 11,458 facts across 16 relations from the English Open Mind Common Sense (OMCS) dataset with single-token objects. Cloze queries are formed by masking the object token in the natural language OMCS sentence connecting the subject and object.
4. SQuAD: 305 context-insensitive, open-domain question-answer pairs with single-token answers selected from the SQuAD development set, manually rewritten into cloze sentences (e.g., "Who developed the theory of relativity?" is transformed to "The theory of relativity was developed by [MASK].").

Knowl 3 — Factual Knowledge Retrieval Precision Across Pretrained Language Models and Baselines

data/table

The table below reports the mean Precision at 1 ( $P@1$ , in percent) for factual knowledge extraction across the LAMA datasets. Evaluated language models include convolutional networks (fairseq-fconv, $Fs$ ), autoregressive Transformers (Transformer-XL large, $Txl$ ), bidirectional LSTMs (ELMo original, $Eb$ ; ELMo 5.5B, $E5B$ ), and masked bidirectional Transformers (BERT-base, $Bb$ ; BERT-large, $Bl$ ). Baselines include a relation-frequency prior ( $Freq$ ), an open-domain QA system ( $DrQA$ ), and a supervised relation extraction LSTM model trained on Wikidata-annotated Wikipedia sentences using either naive string matching ( $RE_n$ ) or an oracle entity linker ( $RE_o$ ) that guarantees correct subject/object linking if the correct relation type is extracted from the aligned text.

Corpus	Relation	#Facts	Freq	DrQA	$RE_n$	$RE_o$	$Fs$	$Txl$	$Eb$	$E5B$	$Bb$	$Bl$
Google-RE	birth-place	2937	4.6	-	3.5	13.8	4.4	2.7	5.5	7.5	14.9	16.1
Google-RE	birth-date	1825	1.9	-	0.0	1.9	0.3	1.1	0.1	0.1	1.5	1.4
Google-RE	death-place	765	6.8	-	0.1	7.2	3.0	0.9	0.3	1.3	13.1	14.0
Google-RE	Total	5527	4.4	-	1.2	7.6	2.6	1.6	2.0	3.0	9.8	10.5
T-REx	1-1	937	1.78	-	0.6	10.0	17.0	36.5	10.1	13.1	68.0	74.5
T-REx	N-1	20006	23.85	-	5.4	33.8	6.1	18.0	3.6	6.5	32.4	34.2
T-REx	N-M	13096	21.95	-	7.7	36.7	12.0	16.5	5.7	7.4	24.7	24.3
T-REx	Total	34039	22.03	-	6.1	33.8	8.9	18.3	4.7	7.1	31.1	32.3
ConceptNet	Total	11458	4.8	-	-	-	3.6	5.7	6.1	6.2	15.6	19.2
SQuAD	Total	305	-	37.5	-	-	3.6	3.9	1.6	4.3	14.1	17.4

BERT-large consistently outperforms all other language models across all benchmarks. On Google-RE, BERT-large achieves 10.5% $P@1$ , exceeding the supervised relation extraction baseline with oracle entity linking ( $RE_o$ , 7.6%). On T-REx, BERT-large achieves 32.3% overall $P@1$ , rivaling the 33.8% achieved by $RE_o$ without receiving any relation extraction supervision or gold-aligned text during inference.

Knowl 4 — Relation Cardinality Effects on Language Model Knowledge Retrieval
empirical result
The factual retrieval performance of pretrained language models depends heavily on relation cardinality (1-to-1, N-to-1, or N-to-M):
- 1-to-1 relations (e.g., capital of): BERT-large achieves 74.5% $P@1$ and BERT-base achieves 68.0%, substantially outperforming both the strongest autoregressive model (Transformer-XL at 36.5%), the supervised relation extraction model with oracle entity linking ( $RE_o$ at 10.0%), and the frequency prior baseline (1.78%).
- N-to-1 relations: BERT-large achieves 34.2% $P@1$ (BERT-base 32.4%), performing on par with $RE_o$ (33.8%) and exceeding the frequency baseline (23.85%).
- N-to-M relations: BERT-large achieves 24.3% $P@1$ (BERT-base 24.7%), failing to significantly outperform the frequency prior baseline (21.95%) and trailing supervised $RE_o$ (36.7%).
These results indicate that pretrained bidirectional masked language models store unique functional relationships between entities with high accuracy, but struggle to retrieve facts belonging to many-to-many relation sets.
Knowl 5 — Top-k Precision Scaling in Pretrained Language Models on T-REx
empirical result
When evaluating ranking performance on the 34,039 facts in T-REx across varying rank cutoffs $k \in [1, 100]$ , the probability mass assigned by BERT models to the ground-truth token scales rapidly beyond $k=1$ :
- For both BERT-large and BERT-base, the ground-truth object is ranked within the top 10 candidates ( $P@10$ ) in approximately 60% of all test facts.
- Within the top 100 candidates ( $P@100$ ), the ground-truth object is retrieved in approximately 80% of all test facts.
- In contrast, Transformer-XL reaches approximately 40% at $k=10$ and 60% at $k=100$ , while fairseq-fconv, ELMo original, and ELMo 5.5B reach between 10% and 20% at $k=10$ and 40% to 50% at $k=100$ .
This indicates that output representations of pretrained BERT models reliably surface factual candidate tokens near the top of the distribution, even when the model does not place the target at rank 1.
Knowl 6 — Correlates of Factual Retrieval Accuracy in BERT
empirical result
Pearson correlation analysis between BERT-large $P@1$ performance on T-REx facts and lexical/statistical properties reveals the underlying drivers of masked prediction success:
- Prediction Confidence: The log-probability score of the first predicted token (LPFP) exhibits the strongest positive correlation with accuracy ( $r = 0.42$ ), indicating that BERT's output distribution is well-calibrated with factual correctness.
- Semantic Similarity: The cosine similarity between subject and object vector representations (SOCS, computed using spaCy vectors) positively correlates with $P@1$ ( $r = 0.31$ ).
- Object Frequency: The frequency of object mentions in the pretraining corpus positively correlates with accuracy ( $r = 0.20$ ), whereas subject mention frequency shows virtually no correlation ( $r = -0.05$ ).
- Subject Length: Subject token length exhibits a weak positive correlation ( $r = 0.12$ for standard tokenization, $r = 0.035$ for WordPiece tokenization).
Knowl 7 — Query Phrasing Robustness Across Language Model Architectures
empirical result
Analyzing the distribution of target object ranks when querying individual facts using 10 different aligned Wikipedia sentence formulations per fact (evaluated on 100 facts per relation in T-REx) demonstrates distinct architectural sensitivity to natural language framing:
- BERT-large, BERT-base, and ELMo 5.5B exhibit the lowest rank variability and maintain average object ranks closest to 1 across different sentence phrasings.
- ELMo original, despite not including Wikipedia text in its pretraining corpus, achieves an average rank and variance close to that of BERT.
- Unidirectional models (Transformer-XL) and convolutional models (fairseq-fconv) exhibit much higher interquartile ranges and higher average ranks across paraphrased query sentences, demonstrating greater brittleness to the specific syntactic surface realization of the factual query.
Knowl 8 — Unsupervised Cloze Question Answering via Language Models vs. Supervised Pipeline QA
empirical result
Evaluating zero-shot language models on the 305 cloze-rewritten questions from SQuAD without fine-tuning, training question-answer pairs, or document retrieval reveals competitive open-domain QA capabilities:
- At $P@1$ , BERT-large achieves 17.4% and BERT-base achieves 14.1%, trailing the supervised DrQA pipeline (37.5%), which uses TF/IDF information retrieval over Wikipedia followed by neural reading comprehension.
- At $P@10$ , the performance gap narrows significantly: unsupervised BERT-large achieves 57.1% precision compared to 63.5% for supervised DrQA.
This indicates that large-scale pretraining allows deep bidirectional language models to act as unsupervised open-domain question-answering systems directly from internal parametric memory.
Knowl 9 — Methodological Scope and Limitations of the LAMA Benchmark
limitation
The LAMA probing framework operates under several specific methodological constraints and boundaries:
1. Single-Token Restriction: Target objects are restricted strictly to single tokens in the vocabulary intersection ( $V_{\text{shared}}$ ). Multi-token decoding introduces hyperparameter tuning (beam size, length normalization, n-gram repetition penalties) that confounds measurement of internal relational representations.
2. Object Slot Masking Only: Only object slots in $(s, r, ?)$ queries are tested. Relations are not queried due to multi-token surface realizations and lack of gold canonical patterns; subject slot queries are approximated via inverse relations.
3. Lower Bound Estimation: Manually chosen cloze templates provide a lower bound on model knowledge; different prompt phrasings yield varying accuracy for the same fact.
4. Vocabulary Intersection Bias: Restricting evaluation to the $\sim 21\text{K}$ shared vocabulary excludes facts whose object tokens fall outside the intersection.
5. Memorization vs. Reasoning: High accuracy on Wikipedia-derived benchmarks (Google-RE, T-REx) may reflect surface co-occurrence memorization from pretraining text rather than relational reasoning.

Coverage note — None was omitted; all contributed benchmark components, quantitative evaluation tables, correlation analyses, stability experiments, and methodological limitations are fully represented.

References

1.Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. 2019. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations (ICLR).
2.Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long Papers, pages 238–247.
3.Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155.
4.Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pages 2787–2795.
5.S. R. K. Branavan, David Silver, and Regina Barzilay. 2011. Learning to win by reading manuals in a monte-carlo framework. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA, pages 268–277.
6.Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. CoRR, abs/1704.00051.
7.Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. 2018. Babyai: First steps towards grounded language learning with a human in the loop. CoRR, abs/1810.08272.
8.Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. CoRR, abs/1901.02860.
9.Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 933–941.
10.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018a. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
11.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018b. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs]. ArXiv: 1810.04805.
12.Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
13.Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. CoRR, abs/1901.05287.
14.Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695.
15.Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780.
16.Tom Kwiatkowski, Jennimaria Palomaki, Olivia Rhinehart, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, et al. 2019. Natural questions: a benchmark for question answering research.
17.Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. 2019. A Survey of Reinforcement Learning Informed by Natural Language. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, August 10-16 2019, Macao, China.
18.Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1192–1202.
19.R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.
20.Gábor Melis, Chris Dyer, and Phil Blunsom. 2017. On the state of the art of evaluation in neural language models. CoRR, abs/1707.05589.
21.Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. CoRR, abs/1609.07843.
22.Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), Miami, FL, USA, December 2-5, 2012, pages 234–239.
23.Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2016. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11–33.
24.Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227–2237.
25.Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1499–1509.
26.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
27.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
28.Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
29.Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. Coqa: A conversational question answering challenge. CoRR, abs/1808.07042.
30.Daniil Sorokin and Iryna Gurevych. 2017. Context-aware representations for knowledge base relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1784–1789.
31.Robert Speer and Catherine Havasi. 2012. Representing general relational knowledge in conceptnet 5. In LREC, pages 3679–3686.
32.Mihai Surdeanu and Heng Ji. 2014. Overview of the English Slot Filling Track at the TAC2014 Knowledge Base Population Evaluation. page 15.
33.Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4149–4158.
34.Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
35.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 6000–6010.
36.Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels, Belgium, November 1, 2018, pages 353–355.
37.Sean Welleck, Kianté Brantley, Hal Daumé III, and Kyunghyun Cho. 2019. Non-monotonic sequential text generation. arXiv preprint arXiv:1902.02192.
38.Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. CoRR, abs/1409.2329.
39.Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2018. From recognition to cognition: Visual commonsense reasoning. CoRR, abs/1811.10830.
40.Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 19–27.

Citation

MLA

Petroni, F., et al. “Language Models as Knowledge Bases?”. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 2463–73, https://doi.org/10.18653/v1/D19-1250.

APA

Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019). Language Models as Knowledge Bases?. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2463–2473. https://doi.org/10.18653/v1/D19-1250

Chicago

Petroni, F., T. Rocktäschel, S. Riedel, et al. 2019. “Language Models as Knowledge Bases?”. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2463–73. https://doi.org/10.18653/v1/D19-1250.

Harvard

Petroni, F. et al. (2019) “Language Models as Knowledge Bases?”, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, pp. 2463–2473. Available at: https://doi.org/10.18653/v1/D19-1250.

Vancouver

1. Petroni F, Rocktäschel T, Riedel S, Lewis P, Bakhtin A, Wu Y, Miller A (2019) Language Models as Knowledge Bases?. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, pp 2463–2473

BibTeX

@inproceedings{petroni-etal-2019-language,
    title = "Language Models as Knowledge Bases?",
    author = {Petroni, Fabio  and
      Rockt{\"a}schel, Tim  and
      Riedel, Sebastian  and
      Lewis, Patrick  and
      Bakhtin, Anton  and
      Wu, Yuxiang  and
      Miller, Alexander},
    editor = "Inui, Kentaro  and
      Jiang, Jing  and
      Ng, Vincent  and
      Wan, Xiaojun",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/D19-1250/",
    doi = "10.18653/v1/D19-1250",
    pages = "2463--2473"
}

Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Language Models as Knowledge Bases?

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — The LAMA (LAnguage Model Analysis) Probe for Factual Knowledge Extraction

Knowl 2 — Knowledge Sources in the LAMA Benchmark

Knowl 3 — Factual Knowledge Retrieval Precision Across Pretrained Language Models and Baselines

Knowl 4 — Relation Cardinality Effects on Language Model Knowledge Retrieval

Knowl 5 — Top-k Precision Scaling in Pretrained Language Models on T-REx

Knowl 6 — Correlates of Factual Retrieval Accuracy in BERT

Knowl 7 — Query Phrasing Robustness Across Language Model Architectures

Knowl 8 — Unsupervised Cloze Question Answering via Language Models vs. Supervised Pipeline QA

Knowl 9 — Methodological Scope and Limitations of the LAMA Benchmark

References

Citation

BibTeX

Source Code

Access the Paper