BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Christopher ClarkKenton LeeMing-Wei ChangTom KwiatkowskiMichael CollinsKristina Toutanova

article2019NAACL2,668 citations

Introduces BoolQ, a benchmark of naturally occurring yes/no questions, establishing that natural binary queries demand complex inference beyond standard extractive reading comprehension and reveal substantial reasoning gaps in advanced language models.

Listen

The article addresses the challenge of building reading comprehension systems that can handle naturally occurring yes/no questions, which often require complex inferences beyond simple fact lookup or paraphrasing. Such questions arise frequently in real user queries yet remain difficult for current models, creating a gap between benchmark performance and practical utility.

The article set out to create a dedicated dataset of these questions and to measure how effectively transfer learning from related tasks could solve them.

The authors collected 16,000 yes/no questions from Google search logs paired with Wikipedia passages that contain the answers, then tested a range of pre-training sources followed by fine-tuning on their data. They compared entailment corpora, extractive and multiple-choice QA sets, paraphrase data, and strong unsupervised language models including BERT.

The key findings are that natural yes/no questions demand diverse inference types, with fewer than 40 percent solvable by paraphrase detection alone; pre-training on the MultiNLI entailment corpus produces the strongest single supervised transfer result; combining MultiNLI pre-training with BERT yields the highest accuracy of 80.4 percent; and this figure remains well below the 90 percent human performance level.

These results indicate that even large-scale unsupervised pre-training leaves substantial headroom on inference-heavy yes/no questions drawn from real usage, and that high-quality labeled entailment data supplies complementary signal that improves final performance.

The authors recommend that future modeling efforts incorporate entailment pre-training when targeting yes/no QA and suggest extending the task to full-document settings to increase realism.

The main limitations are the modest dataset size relative to other QA benchmarks and the restriction to single passages rather than entire documents; the reported gains are consistent across multiple runs but still rest on Wikipedia text and English-language questions only.

Columbia University Google University of Washington inference language model question answering BoolQ Yes/No Question Answering Natural Question

arXiv: 1905.10044 google-research-datasets/boolean-questions

Paper: A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference, Adina Williams et al. (2016). Introduces the MultiNLI dataset that serves as the crucial intermediate supervised pre-training corpus for BoolQ's primary transfer learning findings.
Paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin et al. (2019). Presents the foundational pre-trained Transformer architecture (BERT) that BoolQ fine-tunes and evaluates to demonstrate state-of-the-art performance on yes/no comprehension.
Paper: SQuAD: 100,000+ Questions for Machine Comprehension of Text, Pranav Rajpurkar et al. (2016). Establishes the standard extractive question-answering benchmark paradigm that BoolQ directly builds upon and contrasts with natural inference-heavy yes/no queries.
Paper: Improving Language Understanding by Generative Pre-Training, Alec Radford et al. (2018). Introduces generative pre-training and task-agnostic fine-tuning methodologies that underpin BoolQ's investigation into transfer learning across diverse natural language understanding tasks.
Paper: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data, Alexis Conneau et al. (2017). Demonstrates the power of using natural language inference datasets for transfer learning to downstream tasks, providing the conceptual groundwork for BoolQ's cross-task transfer experiments.
Paper: HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, Zhilin Yang et al. (2018). Pioneers multi-hop, inference-driven question answering benchmarks from Wikipedia, establishing the challenge of complex passage reasoning that BoolQ adapts for binary question formats.
Paper: Know What You Don't Know: Unanswerable Questions for SQuAD, Pranav Rajpurkar et al. (2018). Highlights the critical limitations of shallow pattern-matching in reading comprehension systems, motivating BoolQ's exploration of challenging inferential queries.

Paper: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, Colin Raffel et al. (2020). Integrates BoolQ into the SuperGLUE benchmark to evaluate unified text-to-text language models on complex comprehension tasks.
Paper: RoBERTa: A Robustly Optimized BERT Pretraining Approach, Yinhan Liu et al. (2019). Demonstrates how enhanced pre-training optimization recipes substantially push performance bounds on reading comprehension benchmarks including BoolQ.
Paper: Language Models are Few-Shot Learners, T. B. Brown et al. (2020). Evaluates BoolQ within an extensive suite of tasks to measure zero-shot and few-shot reasoning capabilities across scale.
Paper: BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, Mike Lewis et al. (2020). Applies a denoising sequence-to-sequence model to natural language comprehension and inference benchmarks established by earlier question-answering studies.
Paper: Dense Passage Retrieval for Open-Domain Question Answering, Vladimir Karpukhin et al. (2020). Extends passage-level question answering to full-corpus open-domain retrieval using dense vector representations.
Paper: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Patrick Lewis et al. (2020). Builds end-to-end architectures combining dense passage retrieval with generative sequence models for knowledge-intensive question answering.
Paper: Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering, Gautier Izacard et al. (2021). Pushes open-domain question answering further by using generative reader-decoder models across multiple retrieved passages.

Cover for BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Abstract

In this paper we study yes/no questions that are naturally occurring --- meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.

Topics

inference language model question answering transfer learning

Keywords

BoolQ Yes/No Question Answering Natural Question MultiNLI BERT Entailment Transfer Learning Inference Paraphrasing Dataset

1 Introduction
2 Related Work
3 The BoolQ Dataset
3.1 Data Collection
3.2 Analysis
3.3 Annotation Quality
3.4 Question Types
3.5 Types of Inference
3.6 Discussion
4 Training Yes/No QA Models
5 Results
5.1 Shallow Models
5.2 Neural Models
5.3 Question/Passage Only Results
5.4 Transfer Learning Results
5.5 Multi-Step Transfer Results
5.6 Sample Efficiency
5.7 Discussion
6 Conclusion
References
A Appendices
A.1 Randomly Selected Examples
A.2 Recurrent Model

Knowls

Knowl 1 — The BoolQ Dataset
definition
BoolQ (Boolean Questions) is a reading comprehension dataset composed of 15,974 naturally occurring yes/no question-passage pairs. Questions are unprompted, anonymized queries submitted to the Google search engine that begin with an indicator word (such as did, do, does, is, are, was, were, have, has, can, could, will, would) and for which a Wikipedia page was returned among the top five search results.

Each example in the dataset consists of:
1. A question $q$ , with an average length of 8.9 tokens.
2. A Wikipedia passage $p$ , with an average length of 108 tokens, selected by human annotators as containing sufficient information to answer $q$ .
3. The title $t$ of the source Wikipedia article.
4. A binary label $y \in \{\text{"yes"}, \text{"no"}\}$ .
The dataset is divided into a 9,427-example training set (which incorporates 3k yes/no questions from the Natural Questions dataset), a 3,270-example development set, and a 3,245-example test set. The majority label is "yes", accounting for 62.31% of the training set, 62.17% of the development set, and 62.31% of the test set. Human agreement with expert gold-standard annotations on BoolQ is 90%.

Knowl 2 — Reasoning and Question Categorization of BoolQ

data/table

Analysis of BoolQ reveals that over 60% of questions require complex, non-paraphrase inferential reasoning to resolve from the provided passage. The distribution of inference types and question topics/types across BoolQ is categorized as follows:

Reasoning Category	Definition	Proportion
Paraphrasing	Passage explicitly asserts or refutes the question	38.7%
Other Inference	Passage states facts allowing direct truth/falsity inference	25.9%
By Example	Passage gives an instance or counter-example answering the query	11.8%
Factual Reasoning	Requires external world-knowledge to connect passage to query	8.5%
Implicit	Contextual framing of entities implies truth/falsity	8.5%
Missing Mention	Absence of mention in exhaustive context implies "no"	6.6%

Topic	Share	Yes%	Question Type	Share	Yes%
Entertainment Media	22.0%	65.9%	Other Entity Fact	30.0%	63.3%
Nature/Science	22.0%	56.8%	Other General Fact	29.5%	62.7%
Sports	11.0%	54.5%	Definitional	14.5%	55.2%
Law/Government	10.0%	70.0%	Existence	14.5%	69.0%
History	5.0%	70.0%	Event Occurrence	11.5%	73.9%
Fictional Events	4.0%	87.5%
Other	26.0%	65.4%

These distributions show that natural yes/no questions predominantly query for non-factoid, implicit, or property-based relationships rather than simple entity-factoid lookups.

Knowl 3 — Recurrent Co-Attention Baseline Architecture for Yes/No QA
model/method
The recurrent baseline model for yes/no question answering processes a question (hypothesis) $\mathbf{u}$ of length $m$ and a passage (premise) $\mathbf{v}$ of length $n$ through five sequential stages:
1. Embedding: Words are embedded by concatenating Character-level CNN representations with pre-trained FastText crawl embeddings. A shared bidirectional LSTM computes context-aware hypothesis representations $\langle u_1, \dots, u_m \rangle$ and premise representations $\langle v_1, \dots, v_n \rangle$ .
2. Co-Attention: A similarity matrix $A \in \mathbb{R}^{m \times n}$ is computed across all token pairs $(i, j)$ : $A_{ij} = w_1 \cdot u_i + w_2 \cdot v_j + w_3 \cdot (u_i \circ v_j)$ where $\circ$ denotes elementwise multiplication and $w_1, w_2, w_3$ are learnable weight vectors.
3. Attended Representation: Softmax normalization across rows and columns of $A$ generates attended vectors: $\tilde{u}_i = \sum_{j=1}^n \operatorname{softmax}(A_{i,:})_j v_j, \quad \tilde{v}_j = \sum_{i=1}^m \operatorname{softmax}(A_{:,j}^T)_i u_i$
4. Pooling: A second BiLSTM processes $[v_j; \tilde{v}_j; \tilde{v}_j \circ v_j]$ to produce vectors $h_j$ . Passage representation $v^*$ is pooled via attention scores $a_j = w \cdot h_j$ , $p = \operatorname{softmax}(a)$ , $v^* = \sum_j p_j h_j$ . Hypothesis vector $p^*$ is computed symmetrically from $u_i$ .
5. Classification: The concatenated vector $[v^*; p^*]$ is passed through a 100-dimensional fully connected layer and a softmax output layer. The network uses 200-dimensional LSTMs, a dropout rate of 0.2, and is optimized using Adam with a learning rate decayed by 0.999 every 100 steps.
Knowl 4 — Transfer Learning Adaptations for Yes/No QA
model/method
To transfer knowledge from diverse NLP tasks to yes/no question answering, source task formulations are mapped to binary prediction:
- Natural Language Inference (MultiNLI, SNLI): Models are pre-trained on three-way classification (entailment, contradiction, neutral). During BoolQ fine-tuning, the predicted probability assigned to the entailment class is used directly as the probability of answering "yes": $P(\text{yes}) = P(\text{entailment})$ .
- Multiple Choice QA (RACE): Questions and answer candidates are converted into declarative statements. The model scores each statement independently, applies softmax across all candidate statements for a question, and minimizes cross-entropy loss. When fine-tuning on BoolQ, the sigmoid of the model's raw score for $(q, p)$ gives $P(\text{yes}) = \sigma(\operatorname{score}(q, p))$ .
- Extractive QA (QNLI, SQuAD 2.0, NQ Long Answer):
  - QNLI: Formulated as binary sentence-level entailment (sentence contains answer vs. not).
  - SQuAD 2.0: Concatenates questions with correct answers or negative distractor answers, training the model to classify validity given the passage context.
  - NQ Long Answer: Models score $(q, p)$ pairs against up to 15 non-answer candidate paragraphs from the same document via softmax cross-entropy; fine-tuned to BoolQ using $\sigma(\operatorname{score}(q, p))$ .
- Paraphrase Identification (QQP): Binary classification of whether two question strings are semantically equivalent.
- Heuristically Filtered QA (Y/N MS Marco): 38k web-snippet QA pairs where the free-form answer begins with "yes" or "no".

Knowl 5 — Comparative Performance of Single-Step Transfer Learning on BoolQ

data/table

Evaluating various pre-training sources fine-tuned on the BoolQ training set demonstrates that natural language inference (MultiNLI) provides the strongest supervised transfer signal, outperforming extractive QA, paraphrase detection, and multiple choice QA, while approaching large-scale unsupervised language model pre-training:

Transfer Task	Model	Transfer Data	# Examples	BoolQ Dev Acc. (%)
None	Majority Baseline	-	-	62.17
None	Recurrent	-	-	69.60
Extractive QA	Recurrent	QNLI	108k	71.36
Extractive QA	Recurrent	SQuAD 2.0	130k	69.83
Extractive QA	Recurrent	NQ Long Answer	93k	72.78
Paraphrasing	Recurrent	QQP	364k	71.30
Heuristic Y/N	Recurrent	Y/N MS Marco	39k	71.40
MC QA	Recurrent	RACE	549k	68.40
Entailment	Recurrent	MultiNLI	392k	75.57
Entailment	Recurrent	SNLI	351k	73.16
Unsupervised LM	Recurrent + ELMo	Billion Word	1000M words	71.41
Unsupervised LM	OpenAI GPT	Books Corpus	800M words	72.87
Unsupervised LM	BERTL	Books + Wikipedia	3,300M words	76.90

All results represent averages over five independent runs. MultiNLI yields 75.57% accuracy with a lightweight recurrent model, outperforming all other supervised data sources despite sentence-pair vs. question-passage format differences.

Knowl 6 — Two-Step Transfer Learning with Pre-Trained BERT and MultiNLI

empirical result

Combining unsupervised language model pre-training with intermediate supervised natural language inference (MultiNLI) yields substantial gains on BoolQ. In this two-step regime, $\text{BERT}_{\text{L}}$ (24 layers, 1024 hidden dimensions) is first fine-tuned on MultiNLI and subsequently fine-tuned on the BoolQ training set for 3 epochs with a batch size of 24 and a learning rate of $1\times 10^{-5}$ .

Model	Dev Accuracy (%)	Test Accuracy (%)
Majority Class Baseline	62.17	62.31
Recurrent Baseline	70.28	67.52
Recurrent + MultiNLI	76.15	74.24
Pre-trained $\text{BERT}_{\text{L}}$	78.09	76.70
$\text{BERT}_{\text{L}}$ + MultiNLI	82.20	80.43
Human Performance	-	90.00

Supervised pre-training on MultiNLI provides a $+3.73\%$ test accuracy improvement over $\text{BERT}_{\text{L}}$ alone (reaching $80.43\%$ ) and a $+6.72\%$ test accuracy improvement over the recurrent model with MultiNLI, demonstrating that NLI pre-training captures inferential signal orthogonal to BERT's masked language modeling and next-sentence prediction objectives.

Knowl 7 — Critical Role of the Contradiction Class in MultiNLI Transfer
empirical result
Ablation of individual target classes in MultiNLI demonstrates that learning contradiction is essential for effective transfer to naturally occurring yes/no questions:
1. Recurrent Model Class Ablations (BoolQ Dev Accuracy):
  - Full MultiNLI (392k examples): 75.57%
  - MultiNLI without Neutral (262k examples): 74.83% ( $\Delta = -0.74\%$ )
  - MultiNLI without Entailment (262k examples): 72.95% ( $\Delta = -2.62\%$ )
  - MultiNLI without Contradiction (262k examples): 72.85% ( $\Delta = -2.72\%$ )
2. Two-Step BERT Transfer Ablation:
  - Fine-tuning $\text{BERT}_{\text{L}}$ on MultiNLI without the contradiction class before BoolQ fine-tuning achieves only 78.43% dev accuracy, compared to 82.20% with full MultiNLI.
Removing the contradiction class regresses $\text{BERT}_{\text{L}} + \text{MultiNLI}$ performance almost entirely back to the level of pre-trained $\text{BERT}_{\text{L}}$ alone (78.09%), confirming that explicit supervision on contradictions is the primary driver of MultiNLI transfer to yes/no QA.
Knowl 8 — Superficial Artifact and Bias Evaluation via Partial-Input Baselines
empirical result
To verify whether BoolQ contains superficial annotation artifacts or dataset biases, $\text{BERT}_{\text{L}}$ was evaluated under partial-input conditions where only the question or only the passage is provided to the model:
- Question-only $\text{BERT}_{\text{L}}$ : 64.48% dev set accuracy.
- Passage-only $\text{BERT}_{\text{L}}$ : 66.74% dev set accuracy.
- Majority class baseline: 62.17% dev set accuracy.
The minimal gain of the question-only model over the majority baseline ( $+2.31\%$ ) demonstrates that BoolQ questions do not contain exploitable lexical cues or syntactic patterns indicative of the answer. The passage-only model shows a minor correlation ( $+4.57\%$ over baseline), likely reflecting structural characteristics of introductory Wikipedia paragraphs that correlate slightly with affirmative factual statements.
Knowl 9 — Sample Efficiency Gains from MultiNLI Pre-training
empirical result
Intermediate pre-training on MultiNLI significantly increases the sample efficiency of models fine-tuned on BoolQ:
- When trained on only 1,000 BoolQ examples, MultiNLI pre-training improves dev accuracy by 5 to 6 percentage points for the recurrent model and by nearly 10 percentage points for $\text{BERT}_{\text{L}}$ .
- In low-data regimes (1,000 to 3,000 BoolQ training examples), a lightweight recurrent model pre-trained on MultiNLI outperforms $\text{BERT}_{\text{L}}$ fine-tuned directly without MultiNLI pre-training.
- The performance gap between MultiNLI-initialized models and their non-MultiNLI counterparts persists across all training set sizes up to the full 9.4k training set.

Coverage note — None was omitted; all key contributions including dataset construction, qualitative inference analysis, model architectures, transfer learning strategies, class ablations, partial-input analyses, and sample efficiency results are fully covered.

References

1.Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In Proceedings of the IEEE international conference on computer vision.
2.Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The Sixth PASCAL Recognizing Textual Entailment Challenge. In TAC.
3.Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2011. The Seventh PASCAL Recognizing Textual Entailment Challenge. In TAC.
4.Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. EMNLP.
5.Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for Natural Language Inference. In ACL.
6.Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question Answering in Context. EMNLP.
7.Alexis Conneau and Douwe Kiela. 2018. Senteval: An Evaluation Toolkit for Universal Sentence Representations. In LREC.
8.Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming Question Answering Datasets Into Natural Language Inference Datasets. arXiv:1809.02922. Version 2.
9.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. Version 1.
10.Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI Systems with Sentences that Require Simple Lexical Inferences. ACL.
11.Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation Artifacts in Natural Language Inference Data. NAACL.
12.Minghao Hu, Yuxing Peng, Zhen Huang, Nan Yang, Ming Zhou, et al. 2018. Read+ Verify: Machine Reading Comprehension with Unanswerable Questions. CoRR.
13.Robin Jia and Percy Liang. 2017. Adversarial Examples for Evaluating Reading Comprehension Systems. EMNLP.
14.Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ACL.
15.Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A Textual Entailment Dataset from Science Question Answering. In Proceedings of AAAI.
16.Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. ICLR.
17.Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: a Benchmark for Question Answering Research. TACL.
18.Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-Scale Reading Comprehension Dataset from Examinations. EMNLP.
19.R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. Computing Research Repository, arXiv:1902.01007. Version 1.
20.Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In EMNLP.
21.Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in Pre-Training Distributed Word Representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
22.Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated Machine Reading Comprehension Dataset. arXiv:1611.09268. Version 3.
23.Ankur P Parikh, Oscar Tackstrom, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference. In EMNLP.
24.Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In NAACL.
25.Jason Phang, Thibault Fevry, and Samuel R Bowman. 2018. Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks. Computing Research Repository, arXiv:1811.01088. Version 2.
26.Adam Poliak, Aparajita Haldar, Rachel Rudinger, J Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation. In EMNLP.
27.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-training.
28.Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don't Know: Unanswerable Questions for SQuAD. ACL.
29.Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ Questions for Machine Comprehension of Text. EMNLP.
30.Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. CoQA: A Conversational Question Answering Challenge. In TACL.
31.Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktaschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of Natural Language Rules in Conversational Machine Reading. In EMNLP.
32.Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional Attention Flow for Machine Comprehension. In ICLR.
33.Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.
34.Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. In ACL.
35.Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merrienboer, Armand Joulin, and Tomas Mikolov. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. In ICLR.
36.Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. NAACL.
37.Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton van den Hengel. 2017. Visual Question Answering: A Survey of Methods and Datasets. In Computer Vision and Image Understanding. Elsevier.
38.Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
39.Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In EMNLP.
40.Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension. Computing Research Repository, arXiv:1810.12885. Version 1.
41.Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In Proceedings of the IEEE international conference on computer vision, pages 19–27.

Citation

MLA

Clark, C., et al. “BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions”. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 2924–36, https://doi.org/10.18653/v1/N19-1300.

APA

Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019). BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2924–2936. https://doi.org/10.18653/v1/N19-1300

Chicago

Clark, C., K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. 2019. “BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions”. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2924–36. https://doi.org/10.18653/v1/N19-1300.

Harvard

Clark, C. et al. (2019) “BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions”, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, pp. 2924–2936. Available at: https://doi.org/10.18653/v1/N19-1300.

Vancouver

1. Clark C, Lee K, Chang M-W, Kwiatkowski T, Collins M, Toutanova K (2019) BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, pp 2924–2936

BibTeX

@inproceedings{clark-etal-2019-boolq,
    title = "{B}ool{Q}: Exploring the Surprising Difficulty of Natural Yes/No Questions",
    author = "Clark, Christopher  and
      Lee, Kenton  and
      Chang, Ming-Wei  and
      Kwiatkowski, Tom  and
      Collins, Michael  and
      Toutanova, Kristina",
    editor = "Burstein, Jill  and
      Doran, Christy  and
      Solorio, Thamar",
    booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)",
    month = jun,
    year = "2019",
    address = "Minneapolis, Minnesota",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/N19-1300/",
    doi = "10.18653/v1/N19-1300",
    pages = "2924--2936"
}

Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — The BoolQ Dataset

Knowl 2 — Reasoning and Question Categorization of BoolQ

Knowl 3 — Recurrent Co-Attention Baseline Architecture for Yes/No QA

Knowl 4 — Transfer Learning Adaptations for Yes/No QA