Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

text annotation

Text annotation is the process of attaching informative labels, tags, or metadata to raw text data to make it understandable and usable for computational analysis and machine learning algorithms. This practice typically involves identifying and assigning specific characteristics to words, phrases, or whole documents, such as parts of speech, named entities, sentiment, intent, or thematic categories. While traditionally performed manually by human annotators to generate high-quality ground truth datasets, annotation can also be assisted or conducted automatically through supervised classifiers and large language models. These labeled datasets are essential for training, fine-tuning, and evaluating natural language processing models, enabling computational systems to accurately interpret, classify, and generate human language.

1 item

Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

Nick Pangakis, Sam Wolken

OrganizationsUniversity of Pennsylvania

Why you should read this

Demonstrates through twenty-seven private social science tasks that large language model annotations vary unpredictably and diverge from human judgment, proving that human validation remains essential for automated research workflows.

Automated text annotation is a compelling use case for generative large language models (LLMs) in social media research. Recent work suggests that LLMs can achieve strong performance on annotation tasks; however, these studies evaluate LLMs on a small number of tasks and likely suffer from contamination due to a reliance on public benchmark datasets. Here, we test a human-centered framework for responsibly evaluating artificial intelligence tools used in automated annotation. We use GPT-4 to replicate 27 annotation tasks across 11 password-protected datasets from recently published computational social science articles in high-impact journals. For each task, we compare GPT-4 annotations against human-annotated ground-truth labels and against annotations from separate supervised classification models fine-tuned on human-generated labels. Although the quality of LLM labels is generally high, we find significant variation in LLM performance across tasks, even within datasets. Our findings underscore the importance of a human-centered workflow and careful evaluation standards: Automated annotations significantly diverge from human judgment in numerous scenarios, despite various optimization strategies such as prompt tuning. Grounding automated annotation in validation labels generated by humans is essential for responsible evaluation.

Added

2026-09-29