EntailmentBank dataset

Our goal is for a system to be able to answer questions and justify its answers with a faithful and truthful chain of reasoning. We present Entailer, a system that produces chains of reasoning by decomposing the question into subquestions and then answering each subquestion using a textual entailment model. The model is trained on a new dataset of 10K examples of multistep reasoning, and uses a novel method to generate the training data automatically from a large corpus. Entailer achieves state-of-the-art results on two multistep reasoning datasets, and its reasoning chains are more faithful and truthful than those of a large language model, as measured by human evaluation and a new automatic metric. We also show that Entailer can be used to improve the performance of a large language model by providing it with a faithful derivation of the answer.