Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

toxicity metrics

Toxicity metrics are quantitative evaluation measures used in natural language processing and content moderation to assess the degree to which text contains harmful, abusive, offensive, or disrespectful language. These metrics typically rely on automated machine learning classifiers, scoring algorithms, or specialized evaluation tools that analyze generated or human-authored text to assign numerical probabilities or categorical classifications indicating the presence and severity of toxic content. Commonly applied in artificial intelligence safety benchmarking and online platform governance, toxicity metrics help researchers and developers evaluate the effectiveness of alignment techniques, track harmful outputs across model iterations, and detect specific categories of undesirable content such as hate speech, harassment, insults, and profanity.

1 item

On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research

On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research

Luiza Pozzobon, Beyza Ermis, Patrick Lewis, Sara Hooker

OrganizationsCohereRecod.aiSchool of Electrical and Computer EngineeringUniversity of Campinas

Why you should read this

Reveals how unannounced updates to commercial toxicity evaluation APIs like Perspective alter benchmark leaderboards and invalidate prior research conclusions, while establishing concrete guidelines to ensure reproducible evaluations over time.

Perception of toxicity evolves over time and often differs between geographies and cultural backgrounds. Similarly, black-box commercially available APIs for detecting toxicity, such as the Perspective API, are not static, but frequently retrained to address any unattended weaknesses and biases. We evaluate the implications of these changes on the reproducibility of findings that compare the relative merits of models and methods that aim to curb toxicity. Our findings suggest that research that relied on inherited automatic toxicity scores to compare models and techniques may have resulted in inaccurate findings. Rescoring all models from HELM, a widely respected living benchmark, for toxicity with the recent version of the API led to a different ranking of widely used foundation models. We suggest caution in applying apples-to-apples comparisons between studies and lay recommendations for a more structured approach to evaluating toxicity over time. 1

Added

2026-09-26