keyword

PromptBench

PromptBench is an open-source evaluation framework and library designed to assess the performance, robustness, and safety of large language models. It provides a unified environment for researchers and developers to construct prompts, load standardized datasets and models, execute dynamic evaluation protocols, and simulate adversarial prompt attacks. By analyzing how language models respond to various prompt perturbations and diverse task scenarios, the framework facilitates systematic benchmarking, assists in uncovering model vulnerabilities, and supports the development of reliable artificial intelligence systems.

1 item

PromptBench: A Unified Library for Evaluation of Large Language Models

Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, Xing Xie

OrganizationsCarnegie Mellon UniversityInstitute of Automation, Chinese Academy of SciencesMicrosoftUniversity of Science and Technology of China

Why you should read this

Introduces PromptBench, an open-source evaluation framework that unifies model benchmarking, prompt engineering, multi-level adversarial attacks, and dynamic testing protocols to rigorously assess large language models.

The evaluation of large language models (LLMs) is crucial to assess their performance and mitigate potential security risks. In this paper, we introduce PromptBench, a unified library to evaluate LLMs. It consists of several key components that can be easily used and extended by researchers: prompt construction, prompt engineering, dataset and model loading, adversarial prompt attack, dynamic evaluation protocols, and analysis tools. PromptBench is designed as an open, general, and flexible codebase for research purpose. It aims to facilitate original study in creating new benchmarks, deploying downstream applications, and designing new evaluation protocols. The code is available at: https://github.com/microsoft/promptbench and will be continuously supported.

Source

http://jmlr.org/papers/volume25/24-0023/24-0023.pdf

Added

2026-10-03