Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

performance estimates

Performance estimates are quantitative approximations of a system or machine learning model expected capability, accuracy, or efficiency on a given task or dataset. In computational and artificial intelligence evaluation, these estimates are typically derived using statistical sampling, representative subsets of benchmark data, or proxy metrics to reliably project full-scale performance without incurring the prohibitive cost or time required for exhaustive testing. By balancing precision with resource constraints, performance estimates allow practitioners to compare models, track optimization progress, and predict generalization behavior across diverse workloads.

1 item

tinyBenchmarks: evaluating LLMs with fewer examples

tinyBenchmarks: evaluating LLMs with fewer examples

Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, Mikhail Yurochkin

OrganizationsIBMMassachusetts Institute of TechnologyMIT-IBM Watson AI LabUniversitat Pompeu FabraUniversity of Michigan

Why you should read this

Proposes Item Response Theory and clustering techniques to drastically cut large language model evaluation costs by estimating full benchmark performance on datasets like MMLU and HELM using only 100 representative examples per scenario within an average 2% error margin.

The versatility of large language models (LLMs) led to the creation of diverse benchmarks that thoroughly test a variety of language models’ abilities. These benchmarks consist of tens of thousands of examples making evaluation of LLMs very expensive. In this paper, we investigate strategies to reduce the number of evaluations needed to assess the performance of an LLM on several key benchmarks. For example, we show that to accurately estimate the performance of an LLM on MMLU, a popular multiple-choice QA benchmark consisting of 14K examples, it is sufficient to evaluate this LLM on 100 curated examples. We release evaluation tools and tiny versions of popular benchmarks: Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0. Our empirical analysis demonstrates that these tools and tiny benchmarks are sufficient to reliably and efficiently reproduce the original evaluation results¹.

Added

2026-09-26