Benchmark Dataset Generation and Evaluation for Excel Formula Repair with LLMs

Spreadsheets are among the most widely used computing platforms globally, yet non-technical users frequently encounter complex formula errors. Most automated repair tools have historically targeted isolated syntax issues, leaving semantic and runtime errors—which make up the majority of real-world spreadsheet problems—unaddressed. Advancing automated repair systems has been severely constrained by the lack of high-quality benchmark datasets containing both faulty formulas and their broader spreadsheet contexts.

The article demonstrates an automated, cost-effective pipeline to generate a benchmark dataset for repairing Excel runtime errors and evaluates how effectively modern language models can resolve these errors when supplied with spreadsheet context.

The researchers collected and manually validated 59 real-world seed cases from online Excel user forums across five standard runtime error categories: #DIV/0!, #N/A, #NAME?, #REF!, and #VALUE!. Using these seeds, they implemented a bootstrap data generation pipeline that used few-shot prompting with GPT-4o to synthetically expand the dataset. Generated cases were subjected to dual-layer quality control comprising automated formula execution checks and an automated evaluation model that assessed semantic intent and difficulty. The resulting dataset, named FoRepBench, contains 618 validated benchmark examples. The authors then tested a context-aware formula repair baseline across proprietary and open-source language models, including GPT-4.1, GPT-4o, Phi-3, and Mistral.

The evaluation revealed several critical findings. First, synthetic dataset creation is highly economical; generating FoRepBench required an average of approximately 2.09 model calls per accepted sample, translating to an estimated generation cost of only $0.026 per sample. Second, the automated generation pipeline produced greater functional diversity than the original seed data, introducing common functions such as AVERAGE and CONCATENATE. Third, modern language models solved synthetic repairs with high accuracy, with GPT-4.1 achieving an 80% exact execution match and GPT-4o reaching 73%. However, repair performance dropped steeply when models faced real-world seed data, with GPT-4.1 dropping to 41% accuracy and GPT-4o to 35%. Open-source models lagged further behind, achieving execution match rates between 19% and 24% on real-world examples.

These results indicate that automated synthetic pipelines can efficiently scale benchmark datasets, but synthetic examples skew significantly toward simpler, localized formula corrections. Real-world user errors frequently involve multi-step logic rewrites and deeply nested functions that current generation techniques underrepresent. Consequently, organizations relying solely on synthetic benchmarks risk overestimating how well automated assistants will perform for actual end users in operational settings.

To bridge the gap between synthetic data and real-world complexity, the article recommends incorporating human-in-the-loop review or iterative feedback agents into synthetic generation pipelines. Technical teams developing spreadsheet assistants should also implement structure-aware context retrieval to feed relevant table areas into repair models, rather than relying on raw formula text alone.

Readers should interpret the synthetic performance metrics with caution. While execution correctness is guaranteed via programmatic checks, the automated evaluation filter showed only moderate agreement with human raters on the contextual realism of table data (Cohen's Kappa of 0.42). Additionally, the benchmark remains limited to single-sheet, English-language spreadsheets, meaning performance on multi-sheet workbooks and collaborative enterprise environments requires further evaluation.