Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

ZebraLogic dataset

The ZebraLogic dataset is a benchmark dataset designed to evaluate the logical reasoning and deductive capabilities of large language models using logic grid puzzles. Derived from constraint satisfaction problems similar to classic Zebra or Einstein puzzles, each task requires determining unique associations between entities and attributes based on a series of natural language clues. The dataset features programmatically generated puzzles with systematically controlled search space sizes and diverse logical constraints, providing a standardized environment to analyze how reasoning performance scales as problem complexity increases.

1 item

ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning

ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning

Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, Yejin Choi

Why you should read this

Introduces the ZebraLogic benchmark to quantify the scaling limits of large language models on logic grid puzzles, revealing that current models suffer a severe collapse in reasoning accuracy as problem complexity grows unless inference-time compute and backtracking mechanisms are scaled.

Added

2026-10-01