Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

Robustness testing

Robustness testing is a quality evaluation method used to assess how well a computational system, software application, or machine learning model maintains its performance, stability, and correctness when subjected to unexpected, noisy, or perturbed inputs. In artificial intelligence and data science, this process involves systematically exposing models to edge cases, distribution shifts, adversarial modifications, and variations in input prompts or environmental parameters to determine if their outputs remain reliable. Unlike standard performance evaluations that test behavior under typical or idealized conditions, robustness testing identifies failure modes, behavioral inconsistencies, and vulnerabilities under stress, ensuring that systems can handle unpredictable real-world scenarios safely and dependably.

1 item

Quantifying the Persona Effect in LLM Simulations

Quantifying the Persona Effect in LLM Simulations

Tiancheng Hu, Nigel Collier

OrganizationsUniversity of Cambridge

Why you should read this

Quantifies the limits and efficacy of persona prompting across subjective NLP tasks, establishing that demographic variables explain under ten percent of annotation variance yet enable large language models to recover most predictable human variation when strong correlations exist.

Large language models (LLMs) have shown remarkable promise in simulating human language and behavior. This study investigates how integrating persona variables—demographic, social, and behavioral factors—impacts LLMs’ ability to simulate diverse perspectives. We find that persona variables account for <10% variance in annotating existing subjective NLP datasets. Nonetheless, incorporating persona variables via prompting in LLMs provides modest but statistically significant improvements. Persona prompting is most effective in samples where many annotators disagree, but their disagreements are relatively minor. Notably, we find a linear relationship in our setting: the stronger the correlation between persona variables and human annotations, the more accurate the LLM predictions are using persona prompting. In a zero-shot setting, a powerful 70b model with persona prompting captures 81% of the annotation variance achievable by linear regression trained on ground truth annotations. However, for most subjective NLP datasets, where persona variables have limited explanatory power, the benefits of persona prompting are limited.

Added

2026-09-30