Back

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Pan LuSwaroop MishraTony XiaLiang QiuKai-Wei ChangSong-Chun ZhuOyvind TafjordPeter ClarkAshwin Kalyan

article2022NeurIPS2,641 citations

Introduces ScienceQA, a multimodal benchmark of 21,000 science questions with annotated explanations, demonstrating that chain-of-thought reasoning significantly improves model accuracy and allows systems to match baseline performance using only forty percent of training data.

Listen

The article addresses the challenge of building AI systems that can reliably reason about science questions using multimodal inputs while revealing their thought processes, a gap that limits interpretability and generalization in current models. Existing science question datasets either omit explanations for answers or are confined to text-only formats, small scales, and narrow topics, making it hard to diagnose or improve multi-hop reasoning abilities.

The article set out to introduce a large-scale multimodal benchmark called SCIENCEQA and to test whether language models can generate lectures and explanations as a chain of thought to mimic human reasoning when answering questions.

Researchers assembled roughly 21,000 multiple-choice questions drawn from elementary and high school science curricula across natural, social, and language science, annotating most with lectures for background knowledge and explanations for specific reasoning steps. They benchmarked a range of vision-language models and large language models, then developed chain-of-thought variants: fine-tuning UnifiedQA to output answers plus lectures and explanations, and prompting GPT-3 with in-context examples that include reasoning chains.

The main results show that chain-of-thought generation raises UnifiedQA accuracy by 3.99 percentage points and GPT-3 by 1.20 points over baselines without explanations. Supplying gold explanations in the prompt lifts GPT-3 performance by nearly 19 points, while models using chain of thought reach equivalent accuracy with only 40 percent of the training data. Roughly 65 percent of GPT-3-generated explanations satisfy human standards for relevance, correctness, and completeness.

These outcomes indicate that explicit reasoning chains make large language models both more accurate and more data-efficient on complex multimodal tasks, narrowing the reliability gap with human performance and supporting applications that require traceable decisions.

The work recommends continued development of models that integrate visual and textual evidence more tightly and that generate higher-quality explanations before producing answers. Additional data collection, targeted error analysis on multimodal and domain-specific failures, and human-in-the-loop pilots would strengthen readiness for deployment.

Limitations include reliance on curriculum-sourced data that may not cover all real-world scenarios, dependence on image captions that lose fine-grained visual detail, and persistent model failures on uncommon terminology or complex reasoning chains; results should be viewed as promising but not yet general across all science domains.

Allen Institute for AIArizona State UniversityUniversity of California, Los Angelesdeep learninginterpretabilitymultihopSCIENCEQAMultimodalChain of Thought
arXiv: 2209.09513lupantech/ScienceQA
Cover for Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Abstract

When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT). This process is normally a black box in the case of deep learning models like large-scale language models. Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system. However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity. To this end, we present Science Question Answering (ScienceQA), a new benchmark that consists of ~21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations. We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions. ScienceQA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in few-shot GPT-3 and 3.99% in fine-tuned UnifiedQA. We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%. Our analysis further shows that language models, similar to humans, benefit from explanations to learn from fewer data and achieve the same performance with just 40% of the data. The data and code are available at this https URL.

Topics

deep learninginterpretabilitymultihopquestion answeringreasoningsocial sciencesstatistical language models

Keywords

SCIENCEQAMultimodalChain of ThoughtExplanationFew‑ShotGPT‑3UnifiedQAMulti‑Hop ReasoningDatasetLecture

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Dataset
  • 3.1 Data Analysis
  • 3.2 Comparisons with Existing Datasets
  • 4 Baselines and Chain-of-Thought Models
  • 4.1 Baselines
  • 4.2 Language Models with the Chain of Thought
  • 5 Experiments
  • 5.1 Experimental Setup
  • 5.2 Results for Question Answering
  • 5.3 Results for Generated Explanations
  • 5.4 Analysis
  • 6 Discussion and Conclusion
  • 7 Acknowledgment
  • References
  • A Dataset Analysis
  • A.1 Data Collection
  • A.2 Question Statistics
  • A.3 Choice Statistics
  • A.4 Subject Statistics
  • A.5 Grade Statistics
  • B Experiments
  • B.1 Experimental Details
  • B.2 Human Performance Study
  • B.3 Human Evaluation of Generated Explanations
  • B.4 Case Study and Limitations
  • B.5 Broader Impacts

Knowls

  1. Knowl 1 — ScienceQA Benchmark Dataset

    definition

    Science Question Answering (ScienceQA) is a multimodal multiple-choice benchmark consisting of 21,208 science curriculum questions collected from K-12 educational resources (grades 1 to 12). The dataset spans three overarching subjects: natural science, social science, and language science, divided into 26 topics, 127 categories, and 379 specific skills.

    The benchmark contains diverse input context combinations:

    • Questions with text context only: 3,688 (17.39%)
    • Questions with image context only: 3,800 (17.92%)
    • Questions with both image and text contexts: 6,532 (30.80%)
    • Questions with no context: 7,188 (33.89%)

    Visual contexts total 7,803 unique images covering natural images (~2,960; 14.0%) and diagrams (~7,372; 34.8%), while textual contexts comprise 4,651 unique passages. Beyond questions, choices (ranging from 2 to 5 options), and answers, ScienceQA annotates questions with grounded lectures (17,798 examples, 83.9%) that provide general background domain knowledge, and detailed explanations (19,202 examples, 90.5%) that supply step-by-step rationales for the correct choice. The dataset is split into training (12,726 examples; 60%), validation (4,241 examples; 20%), and test (4,241 examples; 20%) sets.

  2. Knowl 2 — Post-Answer Chain-of-Thought Formulation for Multimodal QA

    model/method

    To model multi-step reasoning while maintaining robust answer extraction in language models, ScienceQA formulates Chain-of-Thought (CoT) generation such that the predicted answer option is produced prior to the lecture and explanation, rather than after them (QCM→ALEQCM \to ALE instead of QCM→LEAQCM \to LEA or QCM→EAQCM \to EA).

    Given question text QQ, context text CC, and multiple choice options MM, visual contexts (when present) are first transformed into text captions using an image captioning model (a Vision Transformer encoder paired with a GPT-2 decoder). The target sequence consists of the correct answer AA, followed by the background lecture LL and specific explanation EE.

    In few-shot in-context learning with large language models such as GPT-3 (text-davinci-002), an nn-shot instruction prompt II is formed by concatenating nn training demonstrations with the test instance: I=({Ii}i=1n,It)I = (\{I_i\}_{i=1}^n, I_t) where each demonstration IiI_i is formatted as: Question: QiOptions: MiContext: CiAnswer: The answer is Ai. BECAUSE: lecture : Li explanation : Ei\text{Question: } Q_i \quad \text{Options: } M_i \quad \text{Context: } C_i \quad \text{Answer: The answer is } A_i \text{. BECAUSE: lecture : } L_i \text{ explanation : } E_i and the test prompt ItI_t ends at Answer: to prompt the model to generate At,Lt,EtA_t, L_t, E_t.

    In the supervised fine-tuning setting with UnifiedQA, the model is trained end-to-end to autoregressively decode the concatenated sequence of the answer token, lecture text, and explanation text given the tokenized input sequence QCMQCM.

  3. Knowl 3 — Comparative Performance of VQA and Language Models on ScienceQA

    empirical result

    Evaluating visual question answering (VQA) baselines, fine-tuned UnifiedQA, and few-shot prompted GPT-3 on the ScienceQA test set demonstrates that language models equipped with Chain-of-Thought (CoT) generation outperform standard classification and QA baselines, though remaining well below human performance.

    Model Learning Format NAT SOC LAN TXT IMG NO G1-6 G7-12 Avg
    Random chance - M→\toA 40.28 46.13 29.25 47.45 40.08 33.66 39.35 40.67 39.83
    MCAN train set QCM→\toA 56.08 46.23 58.09 59.43 51.17 55.40 51.65 59.72 54.54
    Top-Down train set QCM→\toA 59.50 54.33 61.82 62.90 54.88 59.79 57.27 62.16 59.02
    BAN train set QCM→\toA 60.88 46.57 66.64 62.61 52.60 65.51 56.83 63.94 59.37
    DFAF train set QCM→\toA 64.03 48.82 63.55 65.88 54.49 64.11 57.12 67.17 60.72
    ViLT train set QCM→\toA 60.48 63.89 60.27 63.20 61.38 57.00 60.72 61.90 61.14
    Patch-TRM train set QCM→\toA 65.19 46.79 65.55 66.96 55.28 64.95 58.04 67.50 61.42
    VisualBERT train set QCM→\toA 59.33 69.18 61.18 62.71 62.17 58.54 62.96 59.92 61.87
    UnifiedQABASE_{\text{BASE}} zero-shot QCM→\toA 50.13 44.54 48.18 53.08 48.09 46.69 47.58 50.03 48.46
    UnifiedQABASE_{\text{BASE}} train set QCM→\toA 68.16 69.18 74.91 63.78 61.38 77.84 72.98 65.00 70.12
    UnifiedQABASE_{\text{BASE}} (CoT) train set QCM→\toAE 70.60 74.02 78.36 65.69 64.80 81.53 75.48 69.48 73.33
    UnifiedQABASE_{\text{BASE}} (CoT) train set QCM→\toALE 71.00 76.04 78.91 66.42 66.53 81.81 77.06 68.82 74.11
    GPT-3 zero-shot QCM→\toA 75.04 66.59 78.00 74.24 65.74 79.58 76.36 69.87 74.04
    GPT-3 2-shot QCM→\toA 74.64 69.74 76.00 74.44 67.28 77.42 76.80 68.89 73.97
    GPT-3 (CoT) 2-shot QCM→\toAE 76.60 65.92 77.55 75.51 66.09 79.58 78.49 67.63 74.61
    GPT-3 (CoT) 2-shot QCM→\toALE 75.44 70.87 78.09 74.68 67.43 79.93 78.23 69.68 75.17
    Human - QCM→\toA 90.23 84.97 87.48 89.60 87.50 88.10 91.59 82.42 88.40

    Key findings include:

    1. Generating lectures and explanations (QCM→ALEQCM \to ALE) improves the accuracy of fine-tuned UnifiedQABASE_{\text{BASE}} by +3.99% (from 70.12% to 74.11%) over direct answer generation (QCM→AQCM \to A).
    2. In few-shot prompting, GPT-3 with 2-shot CoT (QCM→ALEQCM \to ALE) achieves 75.17% overall accuracy (+1.20% over 2-shot standard prompting at 73.97%).
    3. VisualBERT is the strongest VQA baseline at 61.87% accuracy.
    4. Human performance stands at 88.40% overall, establishing a 13.23% gap over the best model (75.17%) and a 20.07% gap specifically on image-context questions (87.50% human vs 67.43% GPT-3 CoT).

  4. Knowl 4 — Automatic vs. Human Evaluation of Model-Generated Explanations

    empirical result

    Model-generated explanations evaluated with automated n-gram/embedding metrics differ substantially from human rater evaluations. Automated lexical metrics favor fine-tuned models that mimic training set phrasing, whereas human ratings indicate that few-shot GPT-3 produces explanations that are more relevant, correct, and complete.

    In human evaluations, annotators evaluated whether generated explanations were:

    • Relevant: directly related to the question topic.
    • Correct: providing accurate reasoning that leads to the true answer.
    • Complete: fully explaining the problem without missing logical links.
    • Gold Standard: satisfying Relevance, Correctness, and Completeness simultaneously.
    Model Format BLEU-1 BLEU-4 ROUGE-L Similarity Relevant Correct Complete Gold
    UnifiedQABASE_{\text{BASE}} (CoT) QCM→\toALE 0.397 0.370 0.714 0.811 80.4% 76.6% 76.1% 56.9%
    GPT-3 (CoT) QCM→\toAE 0.234 0.048 0.351 0.561 76.9% 73.0% 70.5% 52.5%
    GPT-3 (CoT) QCM→\toALE 0.192 0.052 0.323 0.595 88.5% 78.8% 84.5% 65.2%

    While UnifiedQABASE_{\text{BASE}} achieves the highest BLEU-4 (0.370), ROUGE-L (0.714), and Sentence-BERT semantic similarity (0.811), GPT-3 (QCM→ALEQCM \to ALE) achieves the highest proportion of gold-standard explanations according to human judges (65.2% vs 56.9%), showing that surface lexical overlap does not fully capture explanatory quality.

  5. Knowl 5 — Upper Bound on Question Answering via Oracle Explanations

    empirical result

    Supplying ground-truth lectures (L∗L^*) and explanations (E∗E^*) directly inside the input prompt of GPT-3 provides an empirical upper bound on how much explanatory reasoning can benefit question answering.

    Prompt type Sampling Accuracy (%)
    QCML∗→^* \to A Random 73.59
    QCML∗→^* \to AE Random 74.32
    QCME∗→^* \to A Random 94.03 (+18.86)
    QCMLE∗→^* \to A Random 94.13 (+18.96)
    QCM →\to ALE Random 75.17

    Feeding the gold explanation directly into the input (QCME∗→AQCME^* \to A) boosts GPT-3 accuracy by +18.86% to 94.03%, and feeding both the gold lecture and explanation (QCMLE∗→AQCMLE^* \to A) yields 94.13% (+18.96% over the standard 2-shot CoT baseline of 75.17%). In contrast, providing only the background lecture (QCML∗→AQCML^* \to A) yields 73.59%, indicating that specific step-by-step explanatory rationales, rather than general background facts alone, drive the upper bound performance.

  6. Knowl 6 — Sensitivity of Language Model Accuracy to CoT Output Sequence Ordering

    empirical result

    In few-shot Chain-of-Thought prompting on ScienceQA, placing the target answer before the lecture and explanation (QCM→ALEQCM \to ALE) yields substantially higher accuracy than generating the explanation or lecture before the answer choice.

    Evaluated on 1,000 test examples with 2-shot GPT-3:

    Prompt type Sequence Order Description Accuracy (%)
    QCM →\to ALE Answer →\to Lecture →\to Explanation 73.6
    QCM →\to LA Lecture →\to Answer 60.6
    QCM →\to EA Explanation →\to Answer 56.0
    QCM →\to LEA Lecture →\to Explanation →\to Answer 55.4
    QCM →\to ELA Explanation →\to Lecture →\to Answer 51.5

    Placing the explanation before the answer drops accuracy by 17.6% (QCM→EAQCM \to EA) to 22.1% (QCM→ELAQCM \to ELA). This occurs because generating long explanatory text first increases the risk of premature model stopping or exceeding token generation limits before the final answer token is produced.

  7. Knowl 7 — Sample Efficiency Gains from Chain-of-Thought Supervised Fine-Tuning

    empirical result

    Supervised fine-tuning of language models to generate reasoning chains alongside answers enhances sample efficiency during training.

    Comparing UnifiedQABASE_{\text{BASE}} fine-tuned on subsets of the ScienceQA training set (from 10% to 100% data):

    • UnifiedQABASE_{\text{BASE}} fine-tuned with Chain-of-Thought (QCM→ALEQCM \to ALE) on 40% of the training data achieves approximately 70% test accuracy.
    • Standard UnifiedQABASE_{\text{BASE}} (QCM→AQCM \to A) requires 100% of the training data to reach the same ~70% accuracy level.

    Learning to generate explicit intermediate explanations enables language models to internalize coherent reasoning paths and attain equivalent task proficiency with 60% less labeled training data.

  8. Knowl 8 — In-Context Exemplar Count and Dynamic Sampling Effects in Few-Shot CoT

    empirical result

    Ablation studies on GPT-3 CoT prompt composition reveal the impact of prompt formatting, shot counts, and exemplar selection strategies:

    1. Prompt Type Stability: Prompting GPT-3 with both lectures and explanations (QCM→ALEQCM \to ALE) achieves higher average accuracy across random seeds and lower variance than prompting with explanations alone (QCM→AEQCM \to AE), which produces high variance and less stable predictions.
    2. Shot-Count Scaling: Varying the number of in-context training examples from 1 to 5 demonstrates that performance for QCM→ALEQCM \to ALE peaks at 2 shots (~75.2%), beyond which adding further in-context examples leads to gradual performance degradation.
    3. Dynamic Sampling vs. Random Sampling: Selecting in-context demonstration examples dynamically to match the test sample's category hierarchy yields negligible performance differences compared to uniform random sampling:
    • Random sampling: 75.17%
    • Dynamic sampling from same topic: 75.15%
    • Dynamic sampling from same category: 74.58%
    • Dynamic sampling from same skill: 75.10%

  9. Knowl 9 — Modality and Domain Failure Modes in Multimodal Science Question Answering

    limitation

    Language models relying on image-to-text captions for visual understanding face systematic failure modes on multimodal science questions:

    1. Loss of Fine-Grained Diagrammatic Information: Generic captioning models (such as ViT-GPT-2) generate coarse natural scene descriptions that omit critical structural data, numbers, labels, and relational semantics in science diagrams and climate graphs. This causes language models to hallucinate facts or generate irrelevant explanations.
    2. Complex Domain Terminology and Multi-Stage Scientific Processes: Models struggle to correctly reason over domain-specific terminology (such as distinguishing personification from metaphors in language arts) and complex multi-stage chemical and physical transformations (such as the mineralization sequence occurring during dinosaur fossil formation).

Coverage note — None was omitted. All key dataset statistics, baseline benchmarks, Chain-of-Thought model formulations, human study results, upper bound investigations, sequence ordering ablations, sample efficiency analyses, and failure modes are fully captured.

References

  1. 1.Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  2. 2.Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision (CVPR), pages 2425–2433, 2015.
  3. 3.Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. Flex: Unifying evaluation for few-shot nlp. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
  4. 4.Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 33:1877–1901, 2020.
  5. 5.Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems (NeurIPS), 33:22243–22255, 2020.
  6. 6.Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
  7. 7.Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. Explaining answers with entailment trees. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
  8. 8.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In The International Conference on Learning Representations (ICLR), 2021.
  9. 9.Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6639–6648, 2019.
  10. 10.Peng Gao, Hongsheng Li, Shuang Li, Pan Lu, Yikang Li, Steven CH Hoi, and Xiaogang Wang. Question-guided hybrid convolution for visual question answering. In The European Conference on Computer Vision (ECCV), pages 469–485, 2018.
  11. 11.Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  12. 12.Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700–6709, 2019.
  13. 13.Peter A Jansen, Elizabeth Wainwright, Steven Marmorstein, and Clayton T Morrison. Worldtree: A corpus of explanation graphs for elementary science questions supporting multi-hop inference. arXiv preprint arXiv:1802.03052, 2018.
  14. 14.Harsh Jhamtani and Peter Clark. Learning to explain: Datasets and models for identifying valid reasoning chains in multihop question-answering. arXiv preprint arXiv:2010.03274, 2020.
  15. 15.Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901–2910, 2017.
  16. 16.Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5648–5656, 2018.
  17. 17.Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
  18. 18.Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4999–5007, 2017.
  19. 19.Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. In Findings of the Association for Computational Linguistics (EMNLP), pages 1896–1907, 2020.
  20. 20.Tushar Khot, Peter Clark, Michal Guerquin, Peter Alexander Jansen, and Ashish Sabharwal. Qasc: A dataset for question answering via sentence composition. ArXiv, abs/1910.11473, 2020.
  21. 21.Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 1571–1581, 2018.
  22. 22.Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 5583–5594, 2021.
  23. 23.Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV), pages 32–73, 2017.
  24. 24.Jayant Krishnamurthy, Oyvind Tafjord, and Aniruddha Kembhavi. Semantic parsing to probabilistic programs for situated question answering. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 160–170, 2016.
  25. 25.Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329, 2022.
  26. 26.Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019.
  27. 27.Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. What does bert with vision look at? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 5265–5275, 2020.
  28. 28.Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
  29. 29.Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021.
  30. 30.Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems (NeurIPS), pages 13–23, 2019.
  31. 31.Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Intergps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
  32. 32.Pan Lu, Lei Ji, Wei Zhang, Nan Duan, Ming Zhou, and Jianyong Wang. R-vqa: learning visual relation facts with semantic attention for visual question answering. In The ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD), pages 1880–1889, 2018.
  33. 33.Pan Lu, Hongsheng Li, Wei Zhang, Jianyong Wang, and Xiaogang Wang. Co-attending free-form regions and detections with multi-modal multiplicative feature embedding for visual question answering. In The AAAI Conference on Artificial Intelligence (AAAI), 2018.
  34. 34.Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610, 2022.
  35. 35.Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021.
  36. 36.Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786, 2021.
  37. 37.Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
  38. 38.Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. Lila: A unified benchmark for mathematical reasoning. In The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
  39. 39.Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to gptk’s language. ACL Findings, 2021.
  40. 40.Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. The 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
  41. 41.Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. Wt5?! training text-to-text models to explain their predictions. arXiv preprint arXiv:2004.14546, 2020.
  42. 42.Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021.
  43. 43.Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.
  44. 44.Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (ACL), pages 311–318, 2002.
  45. 45.Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral. In-BoXBART: Get instructions into biomedical multi-task learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 112–128, Seattle, United States, July 2022. Association for Computational Linguistics.
  46. 46.Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
  47. 47.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  48. 48.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21:1–67, 2020.
  49. 49.Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 11 2019.
  50. 50.Mrinmaya Sachan, Kumar Dubey, and Eric Xing. From textbooks to knowledge: A case study in harvesting axiomatic knowledge from textbooks to solve geometry problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 773–784, 2017.
  51. 51.Mrinmaya Sachan and Eric Xing. Learning to solve geometry problems from natural language demonstrations in textbooks. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics ( SEM 2017)*, pages 251–261, 2017.
  52. 52.Shailaja Keyur Sampat, Yezhou Yang, and Chitta Baral. Visuo-lingustic question answering (vlqa) challenge. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings (EMNLP), pages 4606–4616, 2020.
  53. 53.Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. The International Conference on Learning Representations (ICLR), 2021.
  54. 54.Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
  55. 55.Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6281–6290, 2019.
  56. 56.Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and answering binary visual questions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  57. 57.Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning (ICML), pages 12697–12706. PMLR, 2021.
  58. 58.Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

Citation

MLA
Lu, P., et al. “Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering”. Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 2507–21, https://proceedings.neurips.cc/paper_files/paper/2022/file/11332b6b6cf4485b84afadb1352d3a9a-Paper-Conference.pdf.
APA
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022). Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. Advances in Neural Information Processing Systems, 35, 2507–2521. https://proceedings.neurips.cc/paper_files/paper/2022/file/11332b6b6cf4485b84afadb1352d3a9a-Paper-Conference.pdf
Chicago
Lu, P., S. Mishra, T. Xia, et al. 2022. “Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering”. Advances in Neural Information Processing Systems 35: 2507–21. https://proceedings.neurips.cc/paper_files/paper/2022/file/11332b6b6cf4485b84afadb1352d3a9a-Paper-Conference.pdf.
Harvard
Lu, P. et al. (2022) “Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering”, Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 2507–2521. Available at: https://proceedings.neurips.cc/paper_files/paper/2022/file/11332b6b6cf4485b84afadb1352d3a9a-Paper-Conference.pdf.
Vancouver
1. Lu P, Mishra S, Xia T, Qiu L, Chang K-W, Zhu S-C, Tafjord O, Clark P, Kalyan A (2022) Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., pp 2507–2521

BibTeX

@inproceedings{lu2022learn,
  title = {Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering},
  author = {Lu, Pan and Mishra, Swaroop and Xia, Tanglin and Qiu, Liang and Chang, Kai-Wei and Zhu, Song-Chun and Tafjord, Oyvind and Clark, Peter and Kalyan, Ashwin},
  year = {2022},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {35},
  pages = {2507-2521},
  url = {https://proceedings.neurips.cc/paper_files/paper/2022/file/11332b6b6cf4485b84afadb1352d3a9a-Paper-Conference.pdf}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/