Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

LLM4Eval workshop

The LLM4Eval workshop is an academic and industry research venue focused on the development, deployment, and study of large language models for automated evaluation tasks, particularly within information retrieval and related artificial intelligence domains. It serves as a collaborative forum for researchers and practitioners to explore methodologies for using foundation models to assess system outputs, including automated relevance judgments, retrieval-augmented generation pipeline evaluation, and natural language generation quality. In addition to examining the efficiency and scalability of model-based evaluators, the workshop addresses key challenges such as alignment with human judgment, metric trustworthiness, bias, and robustness, often featuring shared benchmark challenges, interactive roundtable discussions, and keynote presentations.

1 item

LLM4Eval@WSDM 2025: Large Language Model for Evaluation in Information Retrieval

LLM4Eval@WSDM 2025: Large Language Model for Evaluation in Information Retrieval

Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L A Clarke, Guglielmo Faggioli, Bhaskar Mitra, Paul Thomas, Emine Yilmaz

OrganizationsAmazonMicrosoftUniversity College LondonUniversity of AmsterdamUniversity of PaduaUniversity of Waterloo

Why you should read this

Presents the scope and shared task for the LLM4Eval workshop at WSDM 2025, detailing how researchers use large language models to automate relevance judgments, evaluate retrieval-augmented generation pipelines, and replace or support human assessments.

Large language models (LLMs) have demonstrated increasing task-solving abilities not present in smaller models. Utilizing the capabilities and responsibilities of LLMs for automated evaluation (LLM4Eval) has recently attracted considerable attention in multiple research communities. For instance, LLM4Eval models have been studied in the context of automated judgments, natural language generation, and retrieval augmented generation systems. We believe that the information retrieval community can significantly contribute to this growing research area by designing, implementing, analyzing, and evaluating various aspects of LLMs with applications to LLM4Eval tasks. The main goal of LLM4Eval workshop is to bring together researchers from industry and academia to discuss various aspects of LLMs for evaluation in information retrieval, including automated judgments, retrieval-augmented generation pipeline evaluation, altering human evaluation, robustness, and trustworthiness of LLMs for evaluation in addition to their impact on real-world applications. We also plan to run an automated judgment challenge prior to the workshop, where participants will be asked to generate labels for a given dataset while maximising correlation with human judgments. The format of the workshop is interactive, including roundtable and keynote sessions and tends to avoid the one-sided dialogue of a mini-conference. This is the second iteration of the workshop. The first version was held in conjunction with SIGIR 2024, attracting over 50 participants.

Added

2026-09-29