Training Verifiers to Solve Math Word Problems

Karl CobbeVineet KosarajuMohammad BavarianMark ChenHeewoo JunLukasz KaiserMatthias PlappertJerry TworekJacob HiltonReiichiro Nakano

article2021arXiv10,563 citations

Introduces the GSM8K benchmark and demonstrates that training verifier models to score candidate solutions substantially improves multi-step mathematical reasoning in language models while scaling more effectively than standard fine-tuning.

Listen

Researchers at OpenAI created GSM8K, a dataset of 8,500 grade-school math word problems, to diagnose why even the largest language models still fail at multi-step mathematical reasoning despite strong results on many other tasks. The problems require only elementary arithmetic yet demand careful sequencing of steps, exposing a core limitation: autoregressive models cannot recover once they make an early error.

The work set out to measure how much performance could be improved by training a separate verifier model to score candidate solutions and select the best one at test time, rather than relying solely on finetuning a generator.

The team generated the dataset through human writers with extensive quality checks, then ran controlled experiments using GPT-3 models ranging from 3 billion to 175 billion parameters. They compared standard finetuning against verification on training sets of varying sizes, sampling up to 100 solutions per problem and training verifiers to predict whether each solution reached the correct final answer.

Verification raised test accuracy by roughly the same margin as a 30-fold increase in model size; a 6-billion-parameter verifier slightly outperformed a 175-billion-parameter finetuned model on the full dataset. Performance gains grew steadily with more training data, while pure finetuning showed diminishing returns. Token-level verifiers that scored every step outperformed solution-level verifiers, and adding residual dropout improved both approaches by reducing overfitting. Larger generators helped verification far more than larger verifiers.

These results indicate that verification offers a more efficient path to higher reliability than simply scaling model size, and that the benefit becomes pronounced once datasets exceed a few thousand examples. The approach therefore lowers the compute and data cost of reaching usable accuracy on this class of problems.

Further gains are likely from combining verification with additional test-time compute such as majority voting among top-ranked solutions, and from extending the method to harder datasets. The main limitations are that verifiers still accept some flawed reasoning that reaches the right answer and that gains require several thousand training problems before they appear. Results are consistent across multiple runs at the 6-billion scale and align with scaling trends observed at 175 billion parameters.

OpenAI candidate solution encoder-decoder language model GSM8K Verification Finetuning

arXiv: 2110.14168 openai/grade-school-math

Paper: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Jason Wei et al. (2022). Introduces chain-of-thought prompting, establishing the core autoregressive reasoning paradigm that the source paper's verifiers evaluate and improve.
Paper: Let's Verify Step by Step, Hunter Lightman et al. (2023). Develops process supervision and reward modeling for mathematical reasoning, directly preceding and inspiring the verifier-based scaling approach in the source.

Paper: Generative Verifiers: Reward Modeling as Next-Token Prediction, Lunjun Zhang et al. (2025). Builds directly on the source's verification concept by reformulating reward modeling as generative next-token prediction rather than discriminative scoring.

Cover for Training Verifiers to Solve Math Word Problems

Abstract

State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.

Topics

candidate solution encoder-decoder language model reasoning empirical analysis verification method

Keywords

GSM8K Verification Finetuning Dropout Token‑Level Verifier Generator Large Language Model Multi‑Step Mathematical Reasoning Natural Language Solution Scaling Law

1 Introduction
2 Dataset
3 Related Work
3.1 Related Datasets
3.2 Related Methods
4 Methods
4.1 Finetuning
4.2 Verification
4.3 Verification Ablations
5 Additional Experiments
5.1 Test Time Compute
5.2 Regularization
6 Conclusion
References
A Dataset Details
B Hyperparameters
C Calculator Annotations
D Example Model Solutions
E Verifier Details
F Verifier Visualization

Knowls

Knowl 1 — Solution Verification Pipeline for Mathematical Reasoning
algorithm
The verification pipeline trains a dedicated discriminator to judge the correctness of generated solutions and reranks candidate solutions at inference time.
Input: Training dataset $D = \{(Q_i, A_i^*)\}_{i=1}^M$ , pretrained language model $M_{\text{pre}}$ , candidate count per training problem $N_{\text{train}} = 100$ , candidate count per test problem $N_{\text{test}} = 100$ , sampling temperature $T = 0.7$
Output: Predicted answer for test problem $Q_{\text{test}}$
1. Initialize generator $G \leftarrow M_{\text{pre}}$
2. Finetune $G$ on $D$ using standard autoregressive next-token prediction for 2 epochs
3. Initialize verifier dataset $D_{\text{ver}} \leftarrow \emptyset$
4. for each $(Q_i, A_i^*) \in D$ :
    a. Autoregressively sample $N_{\text{train}}$ candidate solutions $\{S_{i,j}\}_{j=1}^{N_{\text{train}}} \sim G(\cdot \mid Q_i)$ at temperature $T$
    b. for $j = 1$ to $N_{\text{train}}$ :
        i. Extract final numerical answer $A_{i,j}$ from $S_{i,j}$
        ii. Assign binary correctness label $Y_{i,j} \leftarrow 1$ if $A_{i,j} = A_i^*$ else $0$
        iii. $D_{\text{ver}} \leftarrow D_{\text{ver}} \cup \{(Q_i, S_{i,j}, Y_{i,j})\}$
5. Initialize verifier $V \leftarrow G$
6. Train $V$ on $D_{\text{ver}}$ for 1 epoch using a joint objective of token-level verification loss and masked language modeling loss
7. Given a test question $Q_{\text{test}}$ :
    a. Sample $N_{\text{test}}$ candidate solutions $\{S_k\}_{k=1}^{N_{\text{test}}} \sim G(\cdot \mid Q_{\text{test}})$ at temperature $T$
    b. Score each candidate $S_k$ using $V(Q_{\text{test}}, S_k)$
    c. Select and return the candidate with the highest verifier score $\arg\max_k V(Q_{\text{test}}, S_k)$
Limiting generator training to 2 epochs is necessary to preserve sample diversity; continuing training on small datasets causes candidate solution diversity (measured as $\text{test@}100$ solve rate) to degrade rapidly.
Knowl 2 — Grade School Math 8K (GSM8K) Dataset
definition
GSM8K is a benchmark dataset of 8,500 curated, linguistically diverse grade school math word problems, split into 7,500 training problems and 1,000 test problems. Each problem requires 2 to 8 multi-step arithmetic steps using basic operations ( $+$ , $-$ , $\times$ , $\div$ ) and early algebra concepts without requiring explicit variable definitions. Solutions are authored in natural language detailing intermediate reasoning steps rather than isolated mathematical formulas. Quality control by independent worker agreement checks limits breaking errors or ambiguities to under 2% of the dataset.
Knowl 3 — Verification Scaling Efficiency Compared to Generative Finetuning
empirical result
On the GSM8K benchmark, generating 100 candidate completions from a 6B parameter GPT-3 generator and selecting the top solution with a 6B parameter verifier achieves a test solve rate of approximately 38% on the full 7.5K problem training set, slightly outperforming a 175B parameter GPT-3 model evaluated with greedy ( $T=0$ ) generative finetuning (~34%). This corresponds to a performance boost equivalent to a ~30x increase in model parameters.

Additionally, verification exhibits a substantially steeper scaling slope with respect to training set size than standard finetuning. While verification underperforms finetuning at low dataset sizes (e.g., $\le 1\text{K}$ problems) due to verifier overfitting, it scales significantly faster once sufficient training data is provided.
Knowl 4 — Token-Level Value Prediction vs Solution-Level Verification
model/method
Verifiers can be trained either to output a single scalar correctness score conditioned on the entire completion (solution-level) or to predict the probability of solution correctness at every individual token position in the solution sequence (token-level value function).

While solution-level verifiers quickly overfit to the training set and degrade in validation performance during training, token-level verifiers act as dense value functions across intermediate reasoning steps. Token-level training provides an auxiliary training signal throughout the solution path that strongly resists overfitting and yields superior test solve rates (~39% vs ~34% on 6B models without dropout).
Knowl 5 — Joint Objective and Architecture for Token-Level Verifiers
model/method
A verifier is parameterized as an autoregressive language model equipped with a scalar prediction head. The scalar head consists of a single learnable gain parameter $w \in \mathbb{R}$ and bias parameter $b \in \mathbb{R}$ applied to the unembedding logit $z_{t,\text{special}}$ of a reserved special vocabulary token at sequence position $t$ :

$\hat{y}_t = \sigma(w \cdot z_{t,\text{special}} + b)$

where $\sigma(x) = \frac{1}{1 + e^{-x}}$ denotes the sigmoid function, outputting the predicted probability of solution correctness. All other token logits remain dedicated to the standard language modeling objective.

The verifier is trained on candidate solutions labeled with $Y \in \{0, 1\}$ indicating whether the candidate reaches the correct final answer. The joint training objective per sequence is the unweighted sum:

$\mathcal{L} = \mathcal{L}_{\text{LM}} + \mathcal{L}_{\text{verifier}}$

where $\mathcal{L}_{\text{LM}}$ is the autoregressive cross-entropy loss evaluated only over tokens in the solution (masking out tokens in the problem prompt), and $\mathcal{L}_{\text{verifier}} = \frac{1}{|S|} \sum_{t \in S} (\hat{y}_t - Y)^2$ is the mean squared error evaluated across all solution token positions $t \in S$ . Joint training strictly outperforms training on verification loss alone.
Knowl 6 — Asymmetric Impact of Generator vs Verifier Model Capacity
empirical result
In a generate-and-rerank setup on GSM8K, the overall performance depends asymmetrically on the parameter scale of the generator versus that of the verifier:
- Pairing a large 175B generator with a small 6B verifier achieves a test solve rate of ~55%, approaching the performance of a 175B generator paired with a 175B verifier (~57%).
- Pairing a small 6B generator with a large 175B verifier achieves only ~40% solve rate, performing marginally better than a 6B generator with a 6B verifier (~38%).
This asymmetry indicates that generator capacity is the primary performance bottleneck, and smaller verifiers can effectively filter candidate solutions using relatively coarse heuristics.
Knowl 7 — Majority Voting Over Top Verifier-Ranked Candidates
model/method
Instead of selecting solely the top-1 candidate solution ranked by the verifier, test solve rates are improved by taking an unweighted majority vote among the final answers extracted from the top- $k$ verifier-ranked completions.

The optimal voting pool size $k$ depends on the total candidate pool size $N$ :
- For $N = 100$ candidate completions, restricting voting to the top $k \in [3, 5]$ verifier-ranked samples achieves optimal performance (~40% solve rate for a 6B model).
- For $N = 3,200$ candidate completions, expanding voting to the top $k \approx 30$ samples achieves ~44% solve rate, whereas top-1 selection degrades to ~37%.
Without majority voting, raw search over candidate solutions peaks at approximately $N = 400$ and degrades at larger $N$ due to verifier over-optimization (selecting adversarial false positives that fool the verifier). Majority voting over top- $k$ samples mitigates this failure mode.
Knowl 8 — In-Line Calculator Tool Annotations and Sampling Override
model/method
To prevent arithmetic calculation mistakes in language models, training solutions are annotated with calculator tags of the format <<expression=result>> (for example, <<20+10=30>>). During training, these tags are treated as normal text tokens without distinct loss weighting.

During inference, autoregressive token generation is monitored for the pattern <<expression=. When this delimiter is detected, sampling is temporarily interrupted, the mathematical expression is parsed and evaluated using an external deterministic calculator engine (Python eval), and the computed numerical result followed by >> is forced into the output sequence before standard autoregressive sampling resumes. If the expression is ill-formed or times out, sampling proceeds without override.
Knowl 9 — Performance Impact of Natural Language Scratchpads vs Direct Answer Generation
empirical result
Generating intermediate natural language reasoning steps before outputting the final answer is critical for mathematical reasoning performance. When a 6B parameter GPT-3 model is finetuned on GSM8K to emit the final numerical answer directly without generating intermediate reasoning steps, the test solve rate drops from 20.6% to 5.2%.
Knowl 10 — Regularization of Finetuned Language Models and Verifiers Using Residual Dropout
empirical result
Applying 20% residual dropout along the residual paths of transformer layers serves as an effective regularizer for multi-step reasoning models on GSM8K:
- In generative finetuning, 20% dropout improves the 6B test solve rate across training set sizes, increasing final performance from 20.6% to approximately 25% on the full 7.5K training set.
- For solution-level verifiers, 20% dropout prevents the rapid overfitting seen in unregularized baselines, lifting performance from ~34% to ~39% (matching token-level verifiers).
When applying dropout to models initially pretrained without dropout (such as GPT-3), an intermediate pretraining phase with dropout is required prior to finetuning to prevent severe distribution shift.

Coverage note — No substantial contributed material was omitted; the knowls cover the GSM8K dataset, verification algorithm and architecture, scaling properties, token vs solution verifiers, tool use, ensembling via majority voting, scratchpad analysis, and dropout regularization.

References

1.A. Amini, S. Gabriel, P. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019.
2.T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
3.K. Chen, Q. Huang, H. Palangi, P. Smolensky, K. D. Forbus, and J. Gao. Mapping natural-language problems to formal-language solutions using structured neural representations. In ICML, 2020.
4.X. Chen, C. Liang, A. W. Yu, D. Zhou, D. Song, and Q. V. Le. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference on Learning Representations, 2019.
5.T.-R. Chiang and Y.-N. Chen. Semantically-aligned equation generation for solving and reasoning math word problems. arXiv preprint arXiv:1811.00720, 2018.
6.D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
7.D. Huang, S. Shi, C.-Y. Lin, J. Yin, and W.-Y. Ma. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887–896, 2016.
8.D. Huang, J. Liu, C.-Y. Lin, and J. Yin. Neural math word problem solver with reinforcement learning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 213–223, 2018.
9.J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
10.B. Kim, K. S. Ki, D. Lee, and G. Gweon. Point to the expression: Solving algebraic word problems using the expression-pointer transformer model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3768–3779, 2020.
11.N. Kushman, Y. Artzi, L. Zettlemoyer, and R. Barzilay. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 271–281, 2014.
12.G. Lample and F. Charton. Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412, 2019.
13.S. Li, L. Wu, S. Feng, F. Xu, F. Xu, and S. Zhong. Graph-to-tree neural networks for learning structured input-output translation with applications to semantic parsing and math word problem. EMNLP, 2020.
14.Z. Liang, J. Zhang, J. Shao, and X. Zhang. Mwp-bert: A strong baseline for math word problems, 07 2021.
15.W. Ling, D. Yogatama, C. Dyer, and P. Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146, 2017.
16.J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In IJCAI, 2020.
17.S.-Y. Miao, C.-C. Liang, and K.-Y. Su. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.15772, 2021.
18.E. Nichols, L. Gao, and R. Gomez. Collaborative storytelling with large-scale neural language models. arXiv preprint arXiv:2011.10208, 2020.
19.S. Peng, K. Yuan, L. Gao, and Z. Tang. Mathbert: A pre-trained model for mathematical formula understanding. ArXiv, abs/2105.00377, 2021.
20.S. Roy and D. Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752, Lisbon, Portugal, Sept. 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1202. URL https://aclanthology.org/D15-1202.
21.J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034, 2021a.
22.J. T. Shen, M. Yamashita, E. Prihar, N. Heffernan, X. Wu, B. Graff, and D. Lee. Mathbert: A pre-trained language model for general nlp tasks in mathematics education, 08 2021b.
23.I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112, 2014.
24.A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018.
25.A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.
26.A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019.
27.Y. Wang, X. Liu, and S. Shi. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 845–854, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1088. URL https://aclanthology.org/D17-1088.
28.Z. Xie and S. Sun. A goal-driven tree-structured neural model for math word problems. In IJCAI, 2019.
29.W. Zhao, M. Shang, Y. Liu, L. Wang, and J. Liu. Ape210k: A large-scale and template-rich dataset of math word problems. arXiv preprint arXiv:2009.11506, 2020.

Citation

MLA

Cobbe, K., et al. “Training Verifiers to Solve Math Word Problems”. arXiv, 2021, http://arxiv.org/abs/2110.14168v2.

APA

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). Training Verifiers to Solve Math Word Problems. arXiv. http://arxiv.org/abs/2110.14168v2

Chicago

Cobbe, K., V. Kosaraju, M. Bavarian, et al. 2021. “Training Verifiers to Solve Math Word Problems”. arXiv. http://arxiv.org/abs/2110.14168v2.

Harvard

Cobbe, K. et al. (2021) “Training Verifiers to Solve Math Word Problems”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2110.14168v2.

Vancouver

1. Cobbe K, Kosaraju V, Bavarian M, et al (2021) Training Verifiers to Solve Math Word Problems. arXiv

BibTeX

@article{cobbe2021training,
  title = {Training Verifiers to Solve Math Word Problems},
  author = {Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2110.14168v2},
  eprint = {2110.14168}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission

Training Verifiers to Solve Math Word Problems

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Solution Verification Pipeline for Mathematical Reasoning

Knowl 2 — Grade School Math 8K (GSM8K) Dataset

Knowl 3 — Verification Scaling Efficiency Compared to Generative Finetuning

Knowl 4 — Token-Level Value Prediction vs Solution-Level Verification

Knowl 5 — Joint Objective and Architecture for Token-Level Verifiers

Knowl 6 — Asymmetric Impact of Generator vs Verifier Model Capacity

Knowl 7 — Majority Voting Over Top Verifier-Ranked Candidates

Knowl 8 — In-Line Calculator Tool Annotations and Sampling Override

Knowl 9 — Performance Impact of Natural Language Scratchpads vs Direct Answer Generation

Knowl 10 — Regularization of Finetuned Language Models and Verifiers Using Residual Dropout

References

Citation

BibTeX

Source Code

Access the Paper