Back

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Alon TalmorJonathan HerzigNicholas LourieJonathan Berant

article2019NAACL2,656 citationsBest Resource Paper

Introduces CommonsenseQA, a 12,247-question multiple-choice benchmark derived from ConceptNet semantic relations that tests language models on background world knowledge and exposes a wide performance gap with human reasoning.

Listen

The article addresses the challenge that current natural language understanding systems struggle with commonsense reasoning, even though humans routinely use background knowledge about spatial relations, causes, effects, and social conventions to answer questions. Existing question-answering benchmarks either supply explicit context or remain too small to test broad commonsense abilities at scale, leaving a gap in evaluating what models truly capture.

The article set out to build and release a large dataset of multiple-choice questions that require commonsense knowledge without any accompanying text. Researchers extracted subgraphs from the ConceptNet knowledge base to identify a source concept linked to three related target concepts by the same relation. Crowd workers then wrote natural-language questions that mentioned the source and pointed to only one of the targets as the correct answer, while also adding two extra distractors per question.

This process produced 12,247 questions. Strong baseline models were tested, including language-model approaches and reading-comprehension systems that received web search snippets. The best result came from fine-tuning BERT-large, which reached 55.9 percent accuracy on a random data split. Human performance on the same questions reached 88.9 percent. Using web snippets did not meaningfully improve results, and performance dropped further when questions shared source concepts with the training set but required different answers.

These findings show that even large pre-trained language models still miss many commonsense distinctions that humans find straightforward. The gap matters because reliable commonsense reasoning is essential for safer dialogue systems, better information retrieval, and more robust decision support tools. The dataset therefore provides a clear benchmark for measuring progress.

The authors recommend that future work focus on scaling training data, improving model architectures that explicitly incorporate structured knowledge, and conducting targeted error analysis on categories such as negation, granularity, and conjunctions where current models perform worst. They note that the questions depend on the coverage and biases of ConceptNet and that crowdsourced collection may introduce some artifacts, so results should be interpreted with those boundaries in mind. Overall confidence in the core performance gap is high because multiple strong baselines were evaluated against a clear human reference.

Allen Institute for AITel Aviv Universitycommonsense knowledgequestion answeringsemantic informationCommonsenseQAConceptNetBERT‑Large
arXiv: 1811.00937jonathanherzig/commonsenseqa
Cover for CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Abstract

When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from ConceptNet (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discriminate in turn between each of the target concepts. This encourages workers to create questions with complex semantics that often require prior knowledge. We create 12,247 questions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy, well below human performance, which is 89%.

Topics

commonsense knowledgequestion answeringsemantic informationmultiple targetscommon-sense knowledge baseselectronic document

Keywords

CommonsenseQAConceptNetBERT‑LargeMultiple‑Choice QuestionCrowd‑WorkersAtLocation RelationCommonsense KnowledgeDatasetBaseline ModelHuman Accuracy

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Dataset Generation
  • 4 Dataset Analysis
  • 5 Baseline Models
  • 6 Experiments
  • 7 Conclusion
  • References

Knowls

  1. Knowl 1 — CommonsenseQA Dataset Construction via ConceptNet Subgraph Priming

    model/method

    CommonsenseQA is generated through a multi-stage crowdsourcing and extraction pipeline designed to create question-answering examples that require non-trivial commonsense reasoning:

    1. ConceptNet Extraction and Filtering: From the ConceptNet knowledge graph G⊆C×R×CG \subseteq C \times R \times C (where CC is concepts and RR is relations), triplets (q,r,a)(q, r, a) are extracted where qq is a question concept and aa is an answer concept. The process filters out general relations (e.g., RelatedTo) and standard NLP relations (e.g., IsA), keeping 22 specific relations. It further discards concepts exceeding four words, non-English concepts, and pairs where the string edit distance between qq and aa is low, leaving 236,208 triplets.
    2. Subgraph Grouping (Question Sets): For each question concept qq and relation rr, three distinct triplets are grouped: {(q,r,a1),(q,r,a2),(q,r,a3)}\{(q, r, a_1), (q, r, a_2), (q, r, a_3)\}, forming roughly 43,000 question sets of semantically related target concepts.
    3. Crowdsourced Question Authoring: Crowdworkers on Amazon Mechanical Turk receive qq and {a1,a2,a3}\{a_1, a_2, a_3\} and author three natural language questions containing qq. Each question has exactly one aia_i as its correct answer, with the remaining two aja_j serving as distractors. Workers are instructed to avoid explicit surface words strongly tied to the target concept.
    4. Distractor Augmentation: Two additional distractors are added to each question: one selected from ConceptNet sharing relation rr with qq, and one manually authored by the worker to be plausible yet clearly incorrect. Each final question thus contains 5 candidate choices (1 correct answer, 4 distractors).
    5. Quality Verification: Disjoint annotators verify each question. Questions are discarded if not correctly answered by at least one verifier or if marked unanswerable (a 15% filtering rate).
    6. Context Retrieval: To support reading comprehension models, Google Search is queried for each question concatenated with each candidate answer, retrieving 100 snippets per candidate (500 web snippets total per question).

    The final dataset contains 12,247 validated questions from 16,242 originally authored questions.

  2. Knowl 2 — CommonsenseQA Dataset Statistics and ConceptNet Relation Distribution

    data/table

    CommonsenseQA comprises 12,247 five-way multiple-choice questions grounded in ConceptNet. Key dataset measurements and the distribution of underlying ConceptNet relations are summarized below:

    Measurement Value
    Distinct ConceptNet question nodes 2,254
    Distinct ConceptNet answer nodes 12,094
    Distinct ConceptNet nodes total 12,107
    Distinct ConceptNet relation labels 22
    Average question length (tokens) 13.41
    Long questions (>20>20 tokens) 10.3%
    Average answer length (tokens) 1.5
    Answers with >1>1 token 44%
    Distinct words in questions 14,754
    Distinct words in answers 4,911
    Total collected examples 12,247

    The top ConceptNet relations from which questions are generated are:

    Relation Percentage
    AtLocation 47.3%
    Causes 17.3%
    CapableOf 9.4%
    Antonym 8.5%
    HasSubevent 3.6%
    HasPrerequisite 3.3%
    CausesDesire 2.1%
    Desires 1.7%
    PartOf 1.6%
    HasProperty 1.2%

    The top source question concepts are Person (3.1%), People (2.0%), Human (0.7%), Water (0.5%), and Cat (0.5%). Syntactically, 44% of question first words are WH-words, 5% use first names to set narrative context, and 7% start with "if" to formulate hypothetical scenarios.

  3. Knowl 3 — Empirical Performance of Baseline Models on CommonsenseQA

    data/table

    Baselines evaluated on the 5-way multiple-choice test set of CommonsenseQA include heuristic and pre-trained embedding methods, language models, fine-tuned neural inference architectures, and reading comprehension models with retrieved web snippets. Human performance was evaluated on 100 randomly sampled questions using majority voting across 5 annotators not involved in dataset authoring. Results report classification accuracy (%):

    Random split Question concept split
    Model Accuracy SANITY Accuracy SANITY
    Untrained / Heuristic
    VecSim + Numberbatch 29.1 54.0 30.3 54.9
    LM1B-Rep 26.1 39.6 26.0 39.1
    LM1B-Concat 25.3 37.4 25.3 35.2
    VecSim + GloVe 22.3 26.8 20.8 27.1
    Trained without context
    BERT-Large 55.9 92.3 63.6 93.2
    GPT 45.5 87.2 55.5 88.9
    ESIM + ELMo 34.1 76.9 37.9 77.8
    ESIM + GloVe 32.8 79.1 40.4 78.2
    QABilinear + GloVe 31.5 74.8 34.2 71.8
    ESIM + Numberbatch 30.1 74.6 31.2 75.1
    QABilinear + Numberbatch 28.8 73.3 32.0 71.6
    QACompare + GloVe 25.7 69.2 34.1 71.3
    QACompare + Numberbatch 20.4 60.6 25.2 66.8
    Trained with web context
    BiDAF++ 32.0 71.0 38.4 72.0
    Human 88.9 – – –

    BERT-Large attains the highest model accuracy (55.9% on random split, 63.6% on question concept split), well above random chance (20.0%) but substantially below human accuracy (88.9%).

  4. Knowl 4 — Random Split Versus Question Concept Split in CommonsenseQA

    empirical result

    CommonsenseQA is partitioned into an 80/10/10 train/development/test split under two distinct regimes:

    1. Random Split: Questions are split uniformly at random across partitions.
    2. Question Concept Split: Partitions are constructed such that the set of source question concepts qq in train, development, and test are mutually disjoint.

    Across all trained models, performance is on average 5 percentage points lower on the random split than on the question concept split (e.g., BERT-Large achieves 55.9% on the random split versus 63.6% on the concept split; GPT achieves 45.5% versus 55.5%; ESIM + GloVe achieves 32.8% versus 40.4%).

    In the random split, identical question concepts appear in both training and test sets but paired with different target answers. Models that tend to memorize question-concept-to-answer associations suffer when the same question concept recurs with a different correct answer concept, making the random split harder and designated as the benchmark's primary split.

  5. Knowl 5 — Impact of Semantic Distractor Filtering via Sanity Evaluation

    empirical result

    To measure the difficulty introduced by semantic distractor selection (where distractors share ConceptNet relations with the source concept or are manually authored by workers), a Sanity evaluation mode replaces these structured distractors with randomly sampled ConceptNet concepts.

    Under the Sanity configuration, performance increases across all trained architectures:

    • BERT-Large increases from 55.9% to 92.3% on the random split, and from 63.6% to 93.2% on the question concept split.
    • GPT increases from 45.5% to 87.2% on the random split, and from 55.5% to 88.9% on the question concept split.
    • ESIM + GloVe increases from 32.8% to 79.1% on the random split, and from 40.4% to 78.2% on the question concept split.
    • BiDAF++ increases from 32.0% to 71.0% on the random split.

    This demonstrates that standard models can easily eliminate random distractors using broad semantic or topical associations, and that using related target concepts as distractors is critical to force fine-grained commonsense discrimination.

  6. Knowl 6 — Taxonomy and Frequency of Commonsense Reasoning Skills

    data/table

    Manual annotation of 100 randomly sampled development questions from CommonsenseQA categorizes the commonsense reasoning skills required to answer them. Questions frequently require multiple skills, with an average of 1.75 skills per question:

    Category Definition Frequency (%)
    Spatial Concept A appears near Concept B 41%
    Cause Effect Concept A causes Concept B 23%
    Has parts Concept A contains Concept B as one of its parts 23%
    Purpose Concept A is the purpose of Concept B 18%
    Is member of Concept A belongs to the larger class of Concept B 17%
    Social Social convention where Concept A correlates with Concept B 15%
    Activity Concept A is an activity performed in the context of Concept B 8%
    Definition Concept A is a definition of Concept B 6%
    Preconditions Concept A must hold true for Concept B to take place 3%

    Because an individual question can involve multiple reasoning categories simultaneously, skill frequencies do not sum to 100%.

  7. Knowl 7 — Baseline Architectures for Multiple-Choice Commonsense Question Answering

    model/method

    CommonsenseQA formalizes multiple baselines across three modeling paradigms for scoring candidate answer aia_i given question qq:

    1. Untrained Heuristics and Language Models:

      • VecSim: Computes cosine similarity between mean word embeddings of qq and aia_i.
      • LM1B-Concat: Concatenates aia_i to qq and scores by language model likelihood under an LM pre-trained on the One Billion Word Benchmark.
      • LM1B-Rep: Clusters questions by their first two words and rephrases the 5 most frequent prefixes into declarative sentences containing aia_i before computing LM likelihood.

    2. Trained Feature and Inference Models:

      • QABilinear: Computes answer score s(q,ai)=qWai⊤s(q, a_i) = q W a_i^\top, where q,ai∈Rdq, a_i \in \mathbb{R}^d are averaged pre-trained word embeddings and W∈Rd×dW \in \mathbb{R}^{d \times d} is a learned weight matrix, trained with cross-entropy loss over the candidate set.
      • QACompare: Constructs interaction vector h=relu([q;ai;q⊙ai;q−ai]W1+b1)h = \text{relu}([q; a_i; q \odot a_i; q - a_i] W_1 + b_1) where [;][;] is concatenation and ⊙\odot is element-wise multiplication, followed by score projection s(q,ai)=hW2+b2s(q, a_i) = h W_2 + b_2.
      • ESIM: Adapts the Enhanced Sequential Inference Model with BiLSTM and cross-attention by sizing the final output layer to the number of candidate answers.

    3. Pre-trained Transformers and Contextual Reading Comprehension:

      • GPT and BERT-Large: Linearize input as delimiter-separated tokens (e.g., [CLS] q [SEP] a_i [SEP]). The hidden vector of the boundary token ([end] for GPT, [CLS] for BERT) is mapped through a linear layer and softmax across the 5 choices.
      • BiDAF++: Extends BiDAF with self-attention and ELMo embeddings, using the top 500 Google search snippets retrieved for the question-answer pairs as background context.

  8. Knowl 8 — Fine-Grained Failure Modes and Category-Specific Accuracy of BERT-Large

    data/table

    Analysis of BERT-Large predictions on 100 development examples categorized by question characteristics reveals specific strengths and error modes:

    Category Frequency Accuracy (%) Example Question
    Surface clues 35% 77.7% How might a automobile get off a freeway?
    Negation / Antonym 7% 42.8% Where would you store a pillow case that is not in use?
    Factoid knowledge 13% 38.4% What geographic area is a lizard likely to be?
    Bad granularity 31% 35.4% Where is a well used toy car likely to be found?
    Conjunction 23% 23.8% What can you use to store a book while traveling?

    BERT-Large performs best (77.7%) when surface lexical cues point directly to the correct answer. Accuracy drops substantially on questions requiring understanding of negation or antonyms (42.8%), factoid knowledge (38.4%), granularity distinctions where distractors are broader concepts (35.4%), and conjunctions of constraints where distractors satisfy only one requirement (23.8%).

  9. Knowl 9 — Data Scaling and Learning Curves for BERT-Large on CommonsenseQA

    empirical result

    Evaluating BERT-Large on the CommonsenseQA development set across varying training set sizes (10210^2 to ∼104\sim 10^4 examples, adjusting training epochs to keep the total number of mini-batch updates constant and taking the best of 3 runs) shows logarithmic performance scaling.

    Accuracy increases from roughly 20% at 10210^2 examples, to ~45% at 10310^3 examples, and reaches ~56% at 10410^4 examples. Extrapolating this learning curve to 100,000 training examples predicts an accuracy of approximately 75%, indicating that standard fine-tuning on larger in-domain datasets is insufficient to close the gap to human performance (88.9%).

Coverage note — All core contributions—including dataset construction protocols, structural statistics, ConceptNet relation distributions, reasoning skill taxonomy, baseline model implementations, test accuracy comparisons, split comparisons, sanity evaluations, error analyses, and scaling curves—have been extracted. Standard hyperparameter values directly adopted from prior literature were omitted.

References

  1. 1.Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In ACL.
  2. 2.C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005.
  3. 3.Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced lstm for natural language inference. arXiv preprint arXiv:1609.06038.
  4. 4.Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge.
  5. 5.Ernest Davis. 2016. How to write science questions that are easy for people and hard for computers. AI magazine, 37(1):13–22.
  6. 6.J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
  7. 7.Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, AKBC ’13, pages 25–30, New York, NY, USA. ACM.
  8. 8.Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324.
  9. 9.Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693–1701.
  10. 10.Hsin-Yuan Huang, Eunsol Choi, and Wen-tau Yih. 2018. Flowqa: Grasping flow in history for conversational machine comprehension. arXiv preprint arXiv:1810.06683.
  11. 11.M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL).
  12. 12.Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410.
  13. 13.R Kowalski and M Sergot. 1986. A logic-based calculus of events. New Gen. Comput., 4(1):67–95.
  14. 14.Douglas B. Lenat. 1995. Cyc: A large-scale investment in knowledge infrastructure. Commun. ACM, 38:32–38.
  15. 15.Hector J. Levesque. 2011. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning.
  16. 16.Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. 2016. Learning natural language inference using bidirectional lstm model and inner-attention. arXiv preprint arXiv:1605.09090.
  17. 17.Peter LoBue and Alexander Yates. 2011. Types of common-sense knowledge needed for recognizing textual entailment. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 329–334. Association for Computational Linguistics.
  18. 18.J. McCarthy. 1959. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes.
  19. 19.John McCarthy and Patrick J. Hayes. 1969. Some philosophical problems from the standpoint of artificial intelligence. In B. Meltzer and D. Michie, editors, Machine Intelligence 4, pages 463–502. Edinburgh University Press. Reprinted in McC90.
  20. 20.Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering.
  21. 21.N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In North American Association for Computational Linguistics (NAACL).
  22. 22.T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Workshop on Cognitive Computing at NIPS.
  23. 23.Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. CoRR, abs/1803.05223.
  24. 24.J. Pennington, R. Socher, and C. D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP).
  25. 25.Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
  26. 26.Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proc. of *SEM.
  27. 27.A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018. Improving language understanding by generative pre-training. Technical Report, OpenAI.
  28. 28.P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP).
  29. 29.M. Roemmele, C. Bejan, and A. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning.
  30. 30.Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the roc story cloze task. In CoNLL.
  31. 31.M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv.
  32. 32.Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, pages 4444–4451.
  33. 33.O. Tange. 2011. Gnu parallel - the command-line power tool. ;login: The USENIX Magazine, 36(1):42–47.
  34. 34.Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847.
  35. 35.T. Winograd. 1972. Understanding Natural Language. Academic Press.
  36. 36.Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep learning for answer sentence selection. arXiv preprint arXiv:1412.1632.
  37. 37.Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2018a. From recognition to cognition: Visual commonsense reasoning. arXiv preprint arXiv:1811.10830.
  38. 38.Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018b. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326.
  39. 39.Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. TACL, 5:379–395.

Citation

MLA
Talmor, A., et al. “CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge”. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4149–58, https://doi.org/10.18653/v1/N19-1421.
APA
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019). CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4149–4158. https://doi.org/10.18653/v1/N19-1421
Chicago
Talmor, A., J. Herzig, N. Lourie, and J. Berant. 2019. “CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge”. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4149–58. https://doi.org/10.18653/v1/N19-1421.
Harvard
Talmor, A. et al. (2019) “CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge”, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, pp. 4149–4158. Available at: https://doi.org/10.18653/v1/N19-1421.
Vancouver
1. Talmor A, Herzig J, Lourie N, Berant J (2019) CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, pp 4149–4158

BibTeX

@inproceedings{talmor-etal-2019-commonsenseqa,
    title = "{C}ommonsense{QA}: A Question Answering Challenge Targeting Commonsense Knowledge",
    author = "Talmor, Alon  and
      Herzig, Jonathan  and
      Lourie, Nicholas  and
      Berant, Jonathan",
    editor = "Burstein, Jill  and
      Doran, Christy  and
      Solorio, Thamar",
    booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)",
    month = jun,
    year = "2019",
    address = "Minneapolis, Minnesota",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/N19-1421/",
    doi = "10.18653/v1/N19-1421",
    pages = "4149--4158"
}
Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/