Puzzle Solving using Reasoning of Large Language Models: A Survey

As organizations deploy large language models to automate complex decision-making, understanding their actual logical reasoning and strategic capabilities is critical. Standard benchmarks often fail to isolate whether these models genuinely reason through unfamiliar situations or simply retrieve memorized training data. Puzzles serve as a rigorous testing ground to evaluate core cognitive functions, including deduction, lateral thinking, spatial orientation, and planning under uncertainty. The article systematically reviews how language models perform on text-based puzzles and evaluates the prompting, translation, and fine-tuning strategies developed to improve their reasoning.

To conduct this evaluation, the article introduces a functional framework that divides text-based challenges into rule-based puzzles—which feature formal constraints and closed environments, either deterministic (such as Sudoku or Rubik's Cube) or stochastic (such as Minesweeper or card games)—and rule-less puzzles, which require lateral thinking and contextual interpretation (such as riddles, programming snippets, and detective-style commonsense problems). The review synthesizes empirical findings across recent benchmarks, datasets, and methodologies published over a four-year period, comparing prompting frameworks, neuro-symbolic solvers, and specialized model fine-tuning against traditional computational algorithms.

The analysis reveals several core findings. First, advanced structured prompting methods, such as Tree-of-Thoughts and Everything-of-Thoughts, substantially outperform basic sequential reasoning prompts on deterministic puzzles, raising success rates by 26% to 70% and reaching up to 93.2% accuracy on spatial benchmarks like the 8-puzzle. Second, neuro-symbolic translation delivers the highest accuracy on formal logic tasks; translating natural-language rules into Answer Set Programming allowed GPT-4 to achieve 92% accuracy, compared to just 7% under standard few-shot prompting. Third, a substantial performance gap remains between language models and human reasoning on rule-less puzzles, particularly those involving metaphors, counterfactual deduction, and code analysis. Fourth, models struggle heavily in stochastic environments; in games like Minesweeper and poker, they fail to complete full boards or sustain multi-step risk calculations despite grasping isolated rules. Finally, task formatting heavily biases performance, with multiple-choice structures artificially inflating accuracy compared to open-ended, free-text formulations.

These results demonstrate that organizations cannot rely on base language models for mission-critical tasks requiring multi-step deduction, complete verification, or risk management under incomplete information. While traditional deterministic algorithms guarantee complete solutions with transparent execution, standalone language models remain unpredictable and prone to logical breakdown. Advanced prompting approaches mitigate some failures but introduce operational trade-offs, significantly increasing computational latency and invocation costs. Fine-tuning improves domain-specific execution but exhibits narrow transferability across distinct reasoning tasks.

Decision-makers implementing reasoning pipelines should adopt hybrid neuro-symbolic architectures that utilize language models to interpret natural language specifications and formal external solvers to verify logic and execute decisions. Research and development teams should prioritize constructing robust benchmarks for stochastic puzzles and code-translation tasks, which remain underrepresented in current evaluations. Further pilot testing is necessary before deploying purely generative models in complex reasoning workflows.

These findings are based on empirical literature spanning four years, and confidence in the broad performance trends is high. However, readers should note that the field is evolving rapidly, and newer model architectures or alternative prompting designs may alter specific accuracy baselines across individual puzzle benchmarks.