Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

GPT-4 annotations

GPT-4 annotations refer to data labels, classifications, or descriptive metadata automatically generated by the GPT-4 large language model to categorize and analyze unstructured information such as text. In computational research, natural language processing, and data science workflows, these annotations are produced by prompting the model with defined guidelines or taxonomic criteria, serving as an automated alternative or supplement to manual human labeling. While they allow practitioners to rapidly scale the coding of large datasets for tasks such as sentiment analysis, topic classification, and content moderation, their quality and alignment with human judgment are typically evaluated against human-annotated ground truth to ensure reliability.

1 item

Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

Nick Pangakis, Sam Wolken

OrganizationsUniversity of Pennsylvania

Why you should read this

Demonstrates through twenty-seven private social science tasks that large language model annotations vary unpredictably and diverge from human judgment, proving that human validation remains essential for automated research workflows.

Automated text annotation is a compelling use case for generative large language models (LLMs) in social media research. Recent work suggests that LLMs can achieve strong performance on annotation tasks; however, these studies evaluate LLMs on a small number of tasks and likely suffer from contamination due to a reliance on public benchmark datasets. Here, we test a human-centered framework for responsibly evaluating artificial intelligence tools used in automated annotation. We use GPT-4 to replicate 27 annotation tasks across 11 password-protected datasets from recently published computational social science articles in high-impact journals. For each task, we compare GPT-4 annotations against human-annotated ground-truth labels and against annotations from separate supervised classification models fine-tuned on human-generated labels. Although the quality of LLM labels is generally high, we find significant variation in LLM performance across tasks, even within datasets. Our findings underscore the importance of a human-centered workflow and careful evaluation standards: Automated annotations significantly diverge from human judgment in numerous scenarios, despite various optimization strategies such as prompt tuning. Grounding automated annotation in validation labels generated by humans is essential for responsible evaluation.

Added

2026-09-29