Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

expert-curated questions

Expert-curated questions are inquiries crafted, selected, or verified by specialists with advanced knowledge in specific academic, professional, or technical fields to test and benchmark information systems. Unlike general crowd-sourced prompts or automatically generated queries, these questions represent authentic, nuanced, and complex information needs typical of specialized domains such as medicine, law, and the sciences. They are designed to evaluate deep subject-matter understanding, multi-step reasoning, and factual precision, providing rigorous standards for assessing whether automated language models and question-answering technologies can deliver accurate, authoritative, and verifiable information in high-stakes scenarios.

1 item

ExpertQA: Expert-Curated Questions and Attributed Answers

ExpertQA: Expert-Curated Questions and Attributed Answers

Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, Dan Roth

OrganizationsUniversity of PennsylvaniaUniversity of Washington

Why you should read this

Presents a high-quality benchmark of 2,177 domain-specific questions across 32 fields to evaluate the factuality and citation quality of language model responses through direct assessment and revision by verified domain experts.

As language models are adopted by a more sophisticated and diverse set of users, the importance of guaranteeing that they provide factually correct information supported by verifiable sources is critical across fields of study. This is especially the case for high-stakes fields, such as medicine and law, where the risk of propagating false information is high and can lead to undesirable societal consequences. Previous work studying attribution and factuality has not focused on analyzing these characteristics of language model outputs in domain-specific scenarios. In this work, we conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality, by bringing domain experts in the loop. Specifically, we collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. In addition, we ask experts to improve upon responses from language models. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.1

Added

2026-09-26