Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

LLMJudge challenge

The LLMJudge challenge is a research benchmark and competition focused on automated evaluation within information retrieval, where participants employ large language models to generate relevance judgments for query-document pairs. Held in conjunction with academic evaluation workshops, the challenge tasks systems with producing automated labels that maximize correlation and agreement with human relevance assessments. Its primary goal is to investigate prompting techniques, model architectures, and potential biases, providing empirical insights into how automated language models can reduce the substantial time and cost associated with manual annotation for search and ranking evaluation.

1 item

LLM4Eval@WSDM 2025: Large Language Model for Evaluation in Information Retrieval

LLM4Eval@WSDM 2025: Large Language Model for Evaluation in Information Retrieval

Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L A Clarke, Guglielmo Faggioli, Bhaskar Mitra, Paul Thomas, Emine Yilmaz

OrganizationsAmazonMicrosoftUniversity College LondonUniversity of AmsterdamUniversity of PaduaUniversity of Waterloo

Why you should read this

Presents the scope and shared task for the LLM4Eval workshop at WSDM 2025, detailing how researchers use large language models to automate relevance judgments, evaluate retrieval-augmented generation pipelines, and replace or support human assessments.

Large language models (LLMs) have demonstrated increasing task-solving abilities not present in smaller models. Utilizing the capabilities and responsibilities of LLMs for automated evaluation (LLM4Eval) has recently attracted considerable attention in multiple research communities. For instance, LLM4Eval models have been studied in the context of automated judgments, natural language generation, and retrieval augmented generation systems. We believe that the information retrieval community can significantly contribute to this growing research area by designing, implementing, analyzing, and evaluating various aspects of LLMs with applications to LLM4Eval tasks. The main goal of LLM4Eval workshop is to bring together researchers from industry and academia to discuss various aspects of LLMs for evaluation in information retrieval, including automated judgments, retrieval-augmented generation pipeline evaluation, altering human evaluation, robustness, and trustworthiness of LLMs for evaluation in addition to their impact on real-world applications. We also plan to run an automated judgment challenge prior to the workshop, where participants will be asked to generate labels for a given dataset while maximising correlation with human judgments. The format of the workshop is interactive, including roundtable and keynote sessions and tends to avoid the one-sided dialogue of a mini-conference. This is the second iteration of the workshop. The first version was held in conjunction with SIGIR 2024, attracting over 50 participants.

Added

2026-09-29