Back

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Todor MihaylovPeter ClarkTushar KhotAshish Sabharwal

article2018EMNLP2,727 citations

Introduces OpenBookQA, a benchmark that challenges question-answering systems to perform multi-hop reasoning by combining explicit science facts with broad commonsense knowledge, exposing a substantial gap between machine and human performance.

Listen

The article introduces OpenBookQA, a dataset of nearly 6000 multiple-choice questions built on 1326 elementary science facts. It targets a gap in existing question answering resources, which typically test only linguistic matching or self-contained documents and therefore fail to assess how well systems combine core facts with everyday knowledge to handle new situations.

The work set out to create and benchmark a resource that forces models to retrieve one core fact from the provided set and combine it with simple external common knowledge for multi-hop reasoning. The authors generated questions through a multi-stage crowdsourcing pipeline that included automatic hardness filters against retrieval and word-association solvers, followed by human answerability checks and expert review of the development and test sets.

Human solvers reached approximately 92 percent accuracy. In contrast, several established science question answering systems scored near the 25 percent random baseline. Simple neural models that ignore the question text or exploit choice biases reached about 50 percent. When models were given the exact core fact used to write each question plus an author-provided supporting fact, accuracy rose to roughly 77 percent but remained well below human performance.

These results show that current methods struggle with the retrieval and combination steps required for this task, even though the needed common knowledge is usually basic taxonomic or property information. The gap matters because many real-world applications depend on precisely this form of flexible, knowledge-augmented reasoning rather than surface-level pattern matching.

The authors recommend focusing future work on improved multi-hop retrieval from broad knowledge sources and on reasoning mechanisms that reliably chain a core fact with one or two additional pieces of information. They release the dataset with the core fact for each training and development question as partial supervision to support such research.

The main limitations are that the auxiliary facts collected during crowdsourcing are often noisy or incomplete, and the exact common-knowledge facts required can be somewhat subjective. The reported human performance estimate rests on a statistical bound derived from the original annotators rather than entirely new subjects, so readers should treat the precise 92 percent figure as a conservative lower bound.

Allen Institute for AIHeidelberg UniversityResearch Training Group AIPHESelectricityknowledge basemultihopOpenBookQAMulti‑hop reasoningCore fact
arXiv: 1809.02789allenai/OpenBookQA
Cover for Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Abstract

We present a new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject. The open book that comes with our questions is a set of 1329 elementary level science facts. Roughly 6000 questions probe an understanding of these facts and their application to novel situations. This requires combining an open book fact (e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of armor is made of metal) obtained from other sources. While existing QA datasets over documents or knowledge bases, being generally self-contained, focus on linguistic understanding, OpenBookQA probes a deeper understanding of both the topic---in the context of common knowledge---and the language it is expressed in. Human performance on OpenBookQA is close to 92%, but many state-of-the-art pre-trained QA methods perform surprisingly poorly, worse than several simple neural baselines we develop. Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts. We leave it as a challenge to solve the retrieval problem in this multi-hop setting and to close the large gap to human performance.

Topics

electricityknowledge basemultihopquestion answeringsocial sciences

Keywords

OpenBookQAMulti‑hop reasoningCore factCommon knowledgeKnowledge retrieval bottleneckOracle experimentBaseline modelHuman performanceDataset statisticKnowledge‑aware reader

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 OpenBookQA Dataset
  • 3.1 Crowdsourcing Process
  • 3.2 Human Performance
  • 3.3 Question Set Analysis
  • 4 Baseline Models
  • 4.1 No Training, External Knowledge Only
  • 4.2 No Training; ℱ\mathcal{F} and Extr. Knowledge
  • 4.3 Trained Models, No Knowledge
  • 4.4 Trained Model with External Knowledge
  • 5 Baseline Performance
  • 6 Conclusion
  • References
  • A Knowledge Retrieval Module
  • B Implementation and Training
  • C Additional Experiments
  • C.1 Question Answering: ARC
  • C.2 Textual Entailment: SciTail
  • D Success and Failure Examples
  • D.1 Neural Baseline Successes
  • D.2 Neural Baseline Failures, Oracle Success
  • D.3 Neural Baseline and Oracle Failures

Knowls

  1. Knowl 1 — OpenBookQA Dataset Specification and Structure

    definition

    OpenBookQA is a question answering dataset designed to evaluate multi-hop reasoning that combines core elementary science facts with broad commonsense or external world knowledge.

    The dataset consists of:

    1. Q\mathcal{Q}: A set of 5,957 four-way multiple-choice science questions divided into 4,957 training, 500 development, and 500 test questions.
    2. F\mathcal{F}: An "open book" of 1,326 general elementary-level science facts filtered from the 2,287 core facts of the WorldTree explanation corpus. Each question q∈Qq \in \mathcal{Q} is derived from and associated with exactly one core fact f∈Ff \in \mathcal{F}.
    3. K\mathcal{K}: An auxiliary set of approximately 6,000 crowdsourced secondary common-knowledge facts (one per question), provided as optional, non-gold supervision.

    Structural properties and token statistics across the 5,957 questions:

    • Number of answer choices per question: exactly 4.
    • Question sentence count: mean 1.081.08, maximum 6.
    • Question token length: mean 11.4611.46, maximum 76.
    • Choice token length: mean 2.892.89, maximum 23.
    • Core science fact token length: mean 9.389.38, maximum 28.
    • Vocabulary size: 11,855 unique tokens for questions and choices (q+cq+c); 12,839 unique tokens including science facts (q+c+fq+c+f).
    • Length bias distribution: The correct answer is the longest candidate choice in 18.6%18.6\% of questions (1,108 cases) and the shortest choice in 3.6%3.6\% of questions (216 cases).

  2. Knowl 2 — OpenBookQA Question Generation and Adversarial Filtering Protocol

    algorithm

    OpenBookQA questions are generated and filtered through a multi-stage crowdsourcing and automated adversarial filtering pipeline designed to eliminate questions answerable by simple surface heuristics, direct information retrieval (IR), or word association.

    Input: Core science fact set F\mathcal{F}, Information Retrieval (IR) solver, ACME word-association solver
    Output: Question set Q\mathcal{Q}, Auxiliary common-knowledge fact set K\mathcal{K}
    Initialize Q←∅\mathcal{Q} \leftarrow \emptyset, K←∅\mathcal{K} \leftarrow \emptyset
    for each fact f∈Ff \in \mathcal{F} do
        Worker ww is shown ff
        Worker ww writes a common knowledge fact kk and combines ff with kk to derive assertion ss
        Worker ww converts ss into a 4-choice question qmc=(q,{c1,c2,c3,c4})q_{mc} = (q, \{c_1, c_2, c_3, c_4\}) with 1 correct and 3 incorrect choices
        
        // Check 1: Structural and linguistic constraints
        if qmcq_{mc} lacks exactly 4 choices or contains negation words ("no", "none", "not", "isn't", "doesn't", "aren't", "don't", "won't", "except", "can't", "shouldn't", "wouldn't", "couldn't", "mustn't") or choices have non-uniform lengths (not all ≤3\le 3 words or not all ≥4\ge 4 words) then
            Discard qmcq_{mc} or request revision
        end if
        
        // Check 2: Adversarial solver filtering
        if IR solver answers qmcq_{mc} correctly or ACME solver answers qmcq_{mc} correctly or top 3 IR-retrieved sentences are sufficient to answer qmcq_{mc} then
            Worker ww edits and retries qmcq_{mc}
        end if
        
        // Check 3: Human answerability validation
        Present qmcq_{mc} to 5 independent crowd workers
        m←m \leftarrow number of workers who answer qmcq_{mc} correctly
        if m<4m < 4 then
            Discard qmcq_{mc}
        else
            Randomly shuffle the choices {c1,c2,c3,c4}\{c_1, c_2, c_3, c_4\} of qmcq_{mc}
            Q←Q∪{(qmc,f)}\mathcal{Q} \leftarrow \mathcal{Q} \cup \{ (q_{mc}, f) \}
            K←K∪{k}\mathcal{K} \leftarrow \mathcal{K} \cup \{ k \}
        end if
    end for
    Expert Filter: In-house experts manually review and verify the Dev and Test splits.

  3. Knowl 3 — Human Performance Lower Bound Estimation via Hoeffding's Inequality

    theoretical result

    To assess human accuracy on a question set Q\mathcal{Q} without introducing positive selection bias from the answerability crowd filter (which required ≥4/5\ge 4/5 workers to answer correctly), human performance is modeled using independent Bernoulli trials.

    Let pq∈[0,1]p_q \in [0, 1] denote the unknown probability that a randomly chosen human subject from worker pool H\mathcal{H} answers question q∈Qq \in \mathcal{Q} correctly. The average human accuracy on Q\mathcal{Q} is: H(Q)=1∣Q∣∑q∈QpqH(\mathcal{Q}) = \frac{1}{|\mathcal{Q}|} \sum_{q \in \mathcal{Q}} p_q For each question qq, obtaining ∣I∣=5|I| = 5 independent judgments Xq,i∼Bernoulli(pq)X_{q,i} \sim \text{Bernoulli}(p_q) yields the empirical estimator: H~(Q)=1∣Q∣⋅∣I∣∑q∈Q∑i∈IXq,i\tilde{H}(\mathcal{Q}) = \frac{1}{|\mathcal{Q}| \cdot |I|} \sum_{q \in \mathcal{Q}} \sum_{i \in I} X_{q,i} Under the assumption of independence across all n=∣Q∣⋅∣I∣n = |\mathcal{Q}| \cdot |I| samples, Hoeffding's inequality provides two-sided concentration bounds: P(H~(Q)≤H(Q)+t)≥1−exp⁡(−2nt2)P\left(\tilde{H}(\mathcal{Q}) \le H(\mathcal{Q}) + t\right) \ge 1 - \exp(-2 n t^2) P(H~(Q)≥H(Q)−t)≥1−exp⁡(−2nt2)P\left(\tilde{H}(\mathcal{Q}) \ge H(\mathcal{Q}) - t\right) \ge 1 - \exp(-2 n t^2) For evaluation sets with ∣Q∣=500|\mathcal{Q}| = 500 and ∣I∣=5|I| = 5 (n=2500n = 2500):

    • Setting t=0.03t = 0.03 guarantees H(Q)≥H~(Q)−0.03H(\mathcal{Q}) \ge \tilde{H}(\mathcal{Q}) - 0.03 with probability ≥1−exp⁡(−2⋅2500⋅0.032)=1−exp⁡(−4.5)>0.988\ge 1 - \exp(-2 \cdot 2500 \cdot 0.03^2) = 1 - \exp(-4.5) > 0.988 (98.8% confidence).
    • Setting t=0.025t = 0.025 provides a bound with 95.6%95.6\% confidence.

    The conservative lower bound estimate H~(Q)−3%\tilde{H}(\mathcal{Q}) - 3\% gives human performance scores of 89.3%89.3\% on the development set and 91.7%91.7\% on the test set.

  4. Knowl 4 — Taxonomy and Distribution of External Common Knowledge in OpenBookQA

    data/table

    Analysis of 100 training questions in OpenBookQA shows that answering a question requires an average of 1.161.16 additional common knowledge facts beyond the core science fact f∈Ff \in \mathcal{F}. In 21%21\% of crowdsourced questions, the question tested common knowledge directly without strictly needing the core science fact. For questions requiring external knowledge, the required facts are categorized into six structural classes:

    Fact Type % Questions % Facts Example Relation
    PROPERTY 29.11% 25.81% madeof(belt buckle, metal), has(mammals, four legs)
    ISA 20.25% 17.20% isa(tree, living thing), isa(granite, rock)
    BASIC 17.72% 19.35% General scientific facts, e.g., squirrels eat nuts for food
    DEFINITION 17.72% 15.05% Definitions, e.g., telescope uses mirrors to view objects
    CAUSAL 11.39% 9.68% causes(adding lemon juice to milk, milk to break down)
    OTHERS 13.92% 12.90% Commonsense facts, world knowledge, and lexical rewrites

    The % Questions column represents the percentage of analyzed questions that require at least one fact of that type (values sum to >100%>100\% as questions can require multiple types). The % Facts column is the proportion of total identified supporting facts belonging to that category. Taxonomic relations (ISA) and entity attributes (PROPERTY) form the largest share of required external knowledge.

  5. Knowl 5 — Knowledge-Free Neural Artifact Baselines: Plausible Answer Detector and Odd-One-Out Solver

    model/method

    To evaluate dataset bias and crowd-authoring artifacts, two knowledge-free neural baselines solve multiple-choice questions without performing question-to-evidence reasoning.

    Let s∈{q,c1,c2,c3,c4}s \in \{q, c_1, c_2, c_3, c_4\} represent a token sequence of length nsn_s with word embeddings e1s,…,enss∈Rd\mathbf{e}^s_1, \dots, \mathbf{e}^s_{n_s} \in \mathbb{R}^d (d=300d=300 GloVe). A BiLSTM context encoder outputs representations hs,1…nsctx∈Rns×2h\mathbf{h}^{ctx}_{s, 1\dots n_s} \in \mathbb{R}^{n_s \times 2h}, which are aggregated via element-wise max-pooling into: rsctx=max⁡j=1…ns(hs,jctx)∈R2h\mathbf{r}^{ctx}_s = \max_{j=1\dots n_s} (\mathbf{h}^{ctx}_{s, j}) \in \mathbb{R}^{2h}

    1. Plausible Answer Detector (Choice-Only): This baseline completely ignores the question qq and predicts an answer purely based on whether choice cic_i looks like a plausible scientific claim: αci=wcTrcictx∈R\alpha_{c_i} = \mathbf{w}_c^T \mathbf{r}^{ctx}_{c_i} \in \mathbb{R} where wc∈R2h\mathbf{w}_c \in \mathbb{R}^{2h} is a learned parameter vector. The choice is selected as i^=arg⁡max⁡i∈{1,2,3,4}softmax(αc1…4)i\hat{i} = \arg\max_{i \in \{1,2,3,4\}} \text{softmax}(\alpha_{c_{1\dots4}})_i.

    2. Odd-One-Out Solver (Choice-to-Choices Attention): This baseline exploits the tendency of human workers to write three similar distractor options that differ in style or domain from the correct choice. For candidate choices cic_i and cjc_j, similarity is computed via a linear attention layer: Att(u,v)=wattT[u;v;u⊙v;∣u−v∣]∈R\text{Att}(\mathbf{u}, \mathbf{v}) = \mathbf{w}_{att}^T [\mathbf{u}; \mathbf{v}; \mathbf{u} \odot \mathbf{v}; |\mathbf{u} - \mathbf{v}|] \in \mathbb{R} where watt∈R8h\mathbf{w}_{att} \in \mathbb{R}^{8h}. The total attention of choice cic_i to the other three choices is: αci→crest=∑j=1,j≠i4Att(rcictx,rcjctx)\alpha_{c_i \to c_{rest}} = \sum_{j=1, j \ne i}^4 \text{Att}(\mathbf{r}^{ctx}_{c_i}, \mathbf{r}^{ctx}_{c_j}) The solver selects the option with minimal cumulative attention to the others: i^=arg⁡min⁡i∈{1,2,3,4}softmax(αc1…4→crest)i\hat{i} = \arg\min_{i \in \{1,2,3,4\}} \text{softmax}(\alpha_{c_{1\dots4} \to c_{rest}})_i

  6. Knowl 6 — Knowledge-Free Question-Choice Matching Baselines

    model/method

    Two knowledge-free baselines match question text directly to candidate choices without access to core or external facts:

    1. Embedd+Sim (Logistic Regression): Computes centroid embedding vectors rsemb=1ns∑j=1nsejs∈Rd\mathbf{r}^{emb}_s = \frac{1}{n_s} \sum_{j=1}^{n_s} \mathbf{e}^s_j \in \mathbb{R}^d for s∈{q,c1,c2,c3,c4}s \in \{q, c_1, c_2, c_3, c_4\}, and pairwise cosine similarities rq,cicos=cos⁡(rqemb,rciemb)r^{cos}_{q, c_i} = \cos(\mathbf{r}^{emb}_q, \mathbf{r}^{emb}_{c_i}). It constructs a concatenated feature vector: f=[rqemb;rc1emb;rc2emb;rc3emb;rc4emb;rq,c1cos;rq,c2cos;rq,c3cos;rq,c4cos]∈R5d+4\mathbf{f} = [\mathbf{r}^{emb}_q; \mathbf{r}^{emb}_{c_1}; \mathbf{r}^{emb}_{c_2}; \mathbf{r}^{emb}_{c_3}; \mathbf{r}^{emb}_{c_4}; r^{cos}_{q, c_1}; r^{cos}_{q, c_2}; r^{cos}_{q, c_3}; r^{cos}_{q, c_4}] \in \mathbb{R}^{5d + 4} and optimizes an L2L_2-regularized multi-class logistic regression model.

    2. BiLSTM Question Match: Encodes question qq and choices cic_i using a BiLSTM with max-pooling to obtain rqctx,rcictx∈R2h\mathbf{r}^{ctx}_q, \mathbf{r}^{ctx}_{c_i} \in \mathbb{R}^{2h}. The score for choice cic_i is computed via a linear attention layer: αq,ci=wT[rqctx;rcictx;rqctx⊙rcictx;∣rqctx−rcictx∣]∈R\alpha_{q, c_i} = \mathbf{w}^T [\mathbf{r}^{ctx}_q; \mathbf{r}^{ctx}_{c_i}; \mathbf{r}^{ctx}_q \odot \mathbf{r}^{ctx}_{c_i}; |\mathbf{r}^{ctx}_q - \mathbf{r}^{ctx}_{c_i}|] \in \mathbb{R} where w∈R8h\mathbf{w} \in \mathbb{R}^{8h}. The predicted choice is i^=arg⁡max⁡i∈{1,2,3,4}αq,ci\hat{i} = \arg\max_{i \in \{1,2,3,4\}} \alpha_{q, c_i}.

  7. Knowl 7 — Knowledge-Enhanced Reader Architecture

    model/method

    The Knowledge-Enhanced Reader integrates external knowledge facts KQ,C={k1,…,kNk}\mathcal{K}_{Q,C} = \{k_1, \dots, k_{N_k}\} retrieved for question qq and candidate choices c1…4c_{1\dots4}.

    1. Dynamic Memory Representation: Each fact kj∈KQ,Ck_j \in \mathcal{K}_{Q,C} is encoded using a BiLSTM followed by max-pooling to form representation rkjctx∈R2h\mathbf{r}^{ctx}_{k_j} \in \mathbb{R}^{2h}. These form a dynamic knowledge memory matrix: Mk=[rk1ctx;… ;rkNkctx]∈RNk×2h\mathbf{M}_k = [\mathbf{r}^{ctx}_{k_1}; \dots; \mathbf{r}^{ctx}_{k_{N_k}}] \in \mathbb{R}^{N_k \times 2h}

    2. Knowledge-Aware Representations: For each sequence s∈{q,c1,c2,c3,c4}s \in \{q, c_1, c_2, c_3, c_4\} with BiLSTM max-out representation rsctx∈R2h\mathbf{r}^{ctx}_s \in \mathbb{R}^{2h}, a knowledge-attended representation rskn\mathbf{r}^{kn}_s is computed via attention over Mk\mathbf{M}_k: rskn=∑j=1Nksoftmax(Mkrsctx)jrkjctx∈R2h\mathbf{r}^{kn}_s = \sum_{j=1}^{N_k} \text{softmax}(\mathbf{M}_k \mathbf{r}^{ctx}_s)_j \mathbf{r}^{ctx}_{k_j} \in \mathbb{R}^{2h} A combined representation is defined as: rsctx+kn=rsctx+rskn2∈R2h\mathbf{r}^{ctx+kn}_s = \frac{\mathbf{r}^{ctx}_s + \mathbf{r}^{kn}_s}{2} \in \mathbb{R}^{2h}

    3. Knowledge-Enhanced Attention Scoring: Using the linear attention function Att(u,v)=wattT[u;v;u⊙v;∣u−v∣]\text{Att}(\mathbf{u}, \mathbf{v}) = \mathbf{w}_{att}^T [\mathbf{u}; \mathbf{v}; \mathbf{u} \odot \mathbf{v}; |\mathbf{u} - \mathbf{v}|], the alignment score between question qq and choice cic_i combines all 9 pairwise interactions across context, knowledge, and combined representations:

    \text{Att}(\mathbf{r}^{ctx}_q, \mathbf{r}^{ctx}_{c_i}) \\ \text{Att}(\mathbf{r}^{kn}_q, \mathbf{r}^{kn}_{c_i}) \\ \text{Att}(\mathbf{r}^{ctx+kn}_q, \mathbf{r}^{ctx}_{c_i}) \\ \text{Att}(\mathbf{r}^{ctx}_q, \mathbf{r}^{ctx+kn}_{c_i}) \\ \text{Att}(\mathbf{r}^{ctx}_q, \mathbf{r}^{kn}_{c_i}) \\ \text{Att}(\mathbf{r}^{kn}_q, \mathbf{r}^{ctx}_{c_i}) \\ \text{Att}(\mathbf{r}^{ctx+kn}_q, \mathbf{r}^{kn}_{c_i}) \\ \text{Att}(\mathbf{r}^{kn}_q, \mathbf{r}^{ctx+kn}_{c_i}) \\ \text{Att}(\mathbf{r}^{ctx+kn}_q, \mathbf{r}^{ctx+kn}_{c_i}) \end{bmatrix} \in \mathbb{R}$$ where $\mathbf{W} \in \mathbb{R}^9$ is a learnable parameter vector initialized to all ones. The predicted choice is $\hat{i} = \arg\max_{i \in \{1,2,3,4\}} \alpha^{kn}_{q, c_i}$.

  8. Knowl 8 — OpenBookQA Benchmark Results Across Solver Paradigms

    data/table

    Performance of question answering solvers, trained neural baselines, and oracle models evaluated on the OpenBookQA Development and Test sets (500 questions each). For each question, solvers receive 1 point for a correct answer and 1/k1/k for a kk-way tie that includes the correct answer (random guessing baseline is 25.0%25.0\%):

    Solver Dev (%) Test (%)
    Human solver 89.3∗89.3^* 91.7∗91.7^*
    Guess All (“random”) 25.0 25.0
    No Training, External KB Only
    TupleInference 15.9 17.9
    PMI (Waterloo corpus) 19.7 21.2
    TableILP 20.0 23.4
    DGEM 27.4 24.4
    No Training, External KB + Core Facts F\mathcal{F}
    IR with F\mathcal{F} 25.5 24.8
    TupleInference with F\mathcal{F} 23.6 26.6
    DGEM with F\mathcal{F} 28.2 24.6
    Trained Models, No F\mathcal{F} or External KB
    Embedd+Sim 44.6 41.8
    ESIM 53.9±0.453.9 \pm 0.4 48.9±1.148.9 \pm 1.1
    Plausible Answer Detector 54.4±0.754.4 \pm 0.7 49.6±0.749.6 \pm 0.7
    Odd-one-out Solver 56.9±0.556.9 \pm 0.5 50.2±1.650.2 \pm 1.6
    Question Match 54.6±1.254.6 \pm 1.2 50.2±0.950.2 \pm 0.9
    Oracle Models, Knowledge-Enhanced Reader
    ff (Gold Science Fact) 63.0±2.363.0 \pm 2.3 55.8±2.355.8 \pm 2.3
    ff + WordNet 57.6±1.457.6 \pm 1.4 56.3±1.356.3 \pm 1.3
    ff + ConceptNet 57.0±1.657.0 \pm 1.6 53.7±1.553.7 \pm 1.5
    f+kf + k (Gold Fact + Author Fact) 80.2±1.180.2 \pm 1.1 76.9±0.776.9 \pm 0.7

    *Note: Human performance reflects conservative Hoeffding lower bounds (H~(Q)−3%\tilde{H}(\mathcal{Q}) - 3\%). Trained neural models report the mean ±\pm standard deviation across 5 random seeds.

    Key takeaways:

    1. Pre-trained science QA solvers perform near or below random guessing (17.9%−−24.4%17.9\%--24.4\% on Test), largely because questions were filtered against IR and word association during generation.
    2. Knowledge-free neural baselines achieve up to 50.2%50.2\% on Test, with the choice-only Plausible Answer Detector reaching 49.6%49.6\%, exposing crowd-authoring artifacts.
    3. Supplying the oracle gold science fact ff increases accuracy to 55.8%55.8\% on Test. Providing both the gold science fact ff and authored common knowledge fact kk reaches 76.9%76.9\%, demonstrating that while retrieval is critical, combining facts for multi-hop reasoning leaves a 14.8%14.8\% gap to human performance (91.7%91.7\%).

Coverage note — No substantial contributed material was omitted. All primary dataset definitions, generation and filtering pipelines, mathematical human performance estimation bounds, baseline models, taxonomy analyses, and experimental benchmark results have been formalized into knowls.

References

  1. 1.M. Banko, M. J. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni. 2007. Open information extraction from the web. In IJCAI.
  2. 2.D. Chen, J. Bolton, and C. D. Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In ACL, pages 2358–2367.
  3. 3.D. Chen, A. Fisch, J. Weston, and A. Bordes. 2017a. Reading wikipedia to answer open-domain questions. In ACL.
  4. 4.Q. Chen, X. Zhu, Z.-H. Ling, S. Wei, H. Jiang, and D. Inkpen. 2017b. Enhanced lstm for natural language inference. In ACL, pages 1657–1668.
  5. 5.P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. 2018. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. CoRR, abs/1803.05457.
  6. 6.P. Clark, O. Etzioni, T. Khot, A. Sabharwal, O. Tafjord, P. D. Turney, and D. Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In AAAI, pages 2580–2586.
  7. 7.A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP, pages 670–680.
  8. 8.M. Gardner, J. Grus, M. Neumann, O. Tafjord, P. Dasigi, N. F. Liu, M. Peters, M. Schmitz, and L. S. Zettlemoyer. 2017. AllenNLP: A deep semantic natural language processing platform. CoRR, abs/1803.07640.
  9. 9.S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. A. Smith. 2018. Annotation artifacts in natural language inference data. In NAACL.
  10. 10.K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. 2015. Teaching machines to read and comprehend. In NIPS, pages 1693–1701.
  11. 11.F. Hill, A. Bordes, S. Chopra, and J. Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. In ICLR.
  12. 12.W. Hoeffding. 1963. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13–30.
  13. 13.P. Jansen, N. Balasubramanian, M. Surdeanu, and P. Clark. 2016. What’s in an explanation? characterizing knowledge and inference requirements for elementary science exams. In COLING.
  14. 14.P. A. Jansen, E. Wainwright, S. Marmorstein, and C. T. Morrison. 2018. WorldTree: A corpus of explanation graphs for elementary science questions supporting multi-hop inference. In LREC.
  15. 15.T. Jenkins. 1995. Open book assessment in computing degree programmes 1. Technical Report 95.28, University of Leeds.
  16. 16.M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, pages 1601–1611.
  17. 17.A. Kembhavi, M. J. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In CVPR, pages 5376–5384.
  18. 18.D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay, and D. Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACL.
  19. 19.D. Khashabi, T. Khot, A. Sabharwal, P. Clark, O. Etzioni, and D. Roth. 2016. Question answering via integer programming over semi-structured knowledge. In IJCAI.
  20. 20.T. Khot, A. Sabharwal, and P. Clark. 2017. Answering complex questions using open information extraction. In ACL.
  21. 21.T. Khot, A. Sabharwal, and P. Clark. 2018. SciTail: A textual entailment dataset from science question answering. In AAAI.
  22. 22.D. P. Kingma and J. L. Ba. 2015. Adam: a Method for Stochastic Optimization. International Conference on Learning Representations 2015, pages 1–15.
  23. 23.T. Kocisky, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. 2017. The NarrativeQA reading comprehension challenge. CoRR, abs/1712.07040.
  24. 24.J. Landsberger. 1996. Study guides and strategies. Http://www.studygs.net/tsttak7.htm.
  25. 25.T. Mihaylov and A. Frank. 2016. Discourse relation sense classification using cross-argument semantic similarity based on word embeddings. In CoNLL-16 shared task, pages 100–107.
  26. 26.T. Mihaylov and A. Frank. 2017. Story Cloze Ending Selection Baselines and Data Examination. In LSDSem Shared Task.
  27. 27.T. Mihaylov and A. Frank. 2018. Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge. In ACL, pages 821–832.
  28. 28.T. Mihaylov and P. Nakov. 2016. SemanticZ at SemEval-2016 Task 3: Ranking relevant answers in community question answering using semantic similarity based on fine-tuned word embeddings. In SemEval ’16.
  29. 29.G. A. Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.
  30. 30.G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. J. Miller. 1990. Introduction to WordNet: An on-line lexical database. International Journal of Lexicography, 3(4):235–244.
  31. 31.B. D. Mishra, L. Huang, N. Tandon, W. tau Yih, and P. Clark. 2018. Tracking state changes in procedural text: A challenge dataset and models for process paragraph comprehension. In NAACL.
  32. 32.N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. 2016. A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories. In NAACL.
  33. 33.P. Nakov, L. Marquez, A. Moschitti, W. Magdy, H. Mubarak, a. A. Freihat, J. Glass, and B. Randeree. 2016. Semeval-2016 task 3: Community question answering. In SemEval ’16, pages 525–545.
  34. 34.T. Onishi, H. Wang, M. Bansal, K. Gimpel, and D. McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In EMNLP, pages 2230–2235, Austin, Texas.
  35. 35.A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. 2017. Automatic differentiation in pytorch. In NIPS-W.
  36. 36.F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830.
  37. 37.J. Pennington, R. Socher, and C. Manning. 2014. GloVe: Global vectors for word representation. In EMNLP, pages 1532–1543.
  38. 38.M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In NAACL.
  39. 39.P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, pages 2383–2392.
  40. 40.M. Richardson, C. J. Burges, and E. Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, pages 193–203.
  41. 41.P. Singh, T. Lin, E. Mueller, G. Lim, T. Perkins, and W. Zhu. 2002. Open mind common sense: Knowledge acquisition from the general public. In Lecture Notes in Computer Science, volume 2519, pages 1223–1237.
  42. 42.R. Speer, J. Chin, and C. Havasi. 2017. ConceptNet 5.5: An open multilingual graph of general knowledge. In AAAI.
  43. 43.K. Stasaski and M. A. Hearst. 2017. Multiple choice question generation utilizing an ontology. In BEA@EMNLP, 12th Workshop on Innovative Use of NLP for Building Educational Applications.
  44. 44.S. Sugawara, H. Yokono, and A. Aizawa. 2017. Prerequisite skills for reading comprehension: Multi-perspective analysis of mctest datasets and systems. In AAAI, pages 3089–3096.
  45. 45.A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200.
  46. 46.P. D. Turney. 2017. Leveraging term banks for answering complex questions: A case for sparse vectors. CoRR, abs/1704.03543.
  47. 47.D. Weissenborn, G. Wiese, and L. Seiffe. 2017. Making neural qa as simple as possible but not simpler. In CoNLL, pages 271–280.
  48. 48.J. Welbl, P. Stenetorp, and S. Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. TACL.
  49. 49.Y. Zhang, H. Dai, K. Toraman, and L. Song. 2018. KGˆ2: Learning to Reason Science Exam Questions with Contextual Knowledge Graph Embeddings. In arXiv.

Citation

MLA
Mihaylov, T., et al. “Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering”. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 2381–91, https://doi.org/10.18653/v1/D18-1260.
APA
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018). Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2381–2391. https://doi.org/10.18653/v1/D18-1260
Chicago
Mihaylov, T., P. Clark, T. Khot, and A. Sabharwal. 2018. “Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering”. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2381–91. https://doi.org/10.18653/v1/D18-1260.
Harvard
Mihaylov, T. et al. (2018) “Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering”, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 2381–2391. Available at: https://doi.org/10.18653/v1/D18-1260.
Vancouver
1. Mihaylov T, Clark P, Khot T, Sabharwal A (2018) Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp 2381–2391

BibTeX

@inproceedings{mihaylov-etal-2018-suit,
    title = "Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering",
    author = "Mihaylov, Todor  and
      Clark, Peter  and
      Khot, Tushar  and
      Sabharwal, Ashish",
    editor = "Riloff, Ellen  and
      Chiang, David  and
      Hockenmaier, Julia  and
      Tsujii, Jun{'}ichi",
    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",
    month = oct # "-" # nov,
    year = "2018",
    address = "Brussels, Belgium",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/D18-1260/",
    doi = "10.18653/v1/D18-1260",
    pages = "2381--2391"
}
Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/