Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

multi-agent judges

Multi-agent judges are automated evaluation frameworks composed of multiple interacting artificial intelligence agents that collaborate to assess, score, or compare model outputs. Instead of relying on a single large language model to evaluate text, these systems assign distinct roles, such as critics, advocates, fact-checkers, and aggregators, across several agents to deliberate on the quality, factual accuracy, and reasoning of candidate responses. By employing structured interactions like iterative critique, multi-turn debate, and consensus voting, multi-agent judges aim to reduce individual model biases, mitigate evaluation errors, and deliver more robust and objective assessments on complex tasks.

1 item

JudgeBench: A Benchmark for Evaluating LLM-Based Judges

JudgeBench: A Benchmark for Evaluating LLM-Based Judges

Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca A. Popa, Ion Stoica

OrganizationsUniversity of California BerkeleyWashington University in St. Louis

Why you should read this

Introduces JudgeBench, a benchmark that tests LLM-based evaluators on objective correctness across complex reasoning, math, and coding tasks, revealing that frontier models like GPT-4o perform barely above random guessing.

LLM-based judges have emerged as a scalable alternative to human evaluation and are increasingly used to assess, compare, and improve models. However, the reliability of LLM-based judges themselves is rarely scrutinized. As LLMs become more advanced, their responses grow more sophisticated, requiring stronger judges to evaluate them. Existing benchmarks primarily focus on a judge's alignment with human preferences, but often fail to account for more challenging tasks where crowdsourced human preference is a poor indicator of factual and logical correctness. To address this, we propose a novel evaluation framework to objectively evaluate LLM-based judges. Based on this framework, we propose JudgeBench, a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding. JudgeBench leverages a novel pipeline for converting existing difficult datasets into challenging response pairs with preference labels reflecting objective correctness. Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks, with many strong models (e.g., GPT-4o) performing just slightly better than random guessing. Overall, JudgeBench offers a reliable platform for assessing increasingly advanced LLM-based judges. Data and code are available at this https URL.

Added

2026-09-26