Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

human-AI partnerships

A human-AI partnership is a collaborative framework in which human workers and artificial intelligence systems work together synergistically to accomplish complex tasks, generate content, or solve problems. In this paradigm, the distinct capabilities of both parties are combined, pairing the computational speed, pattern recognition, and generative scale of AI with human domain expertise, critical reasoning, and oversight. Instead of replacing human involvement through complete automation, these partnerships rely on iterative human-in-the-loop interactions, such as guiding task objectives, validating model outputs, correcting algorithmic errors, and filtering out misaligned results. By integrating direct human intervention and supervision into automated workflows, human-AI partnerships enhance the overall accuracy, diversity, and contextual reliability of outcomes beyond what either humans or autonomous systems could achieve on their own.

1 item

Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions

Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions

John Joon Young Chung, Ece Kamar, Saleema Amershi

OrganizationsMicrosoftUniversity of Michigan

Why you should read this

Demonstrates how combining large language model diversification techniques with targeted human label replacement allows smaller downstream classifiers to outperform few-shot large language models by balancing synthetic text variety and annotation accuracy.

Large language models (LLMs) can be used to generate text data for training and evaluating other models. However, creating high-quality datasets with LLMs can be challenging. In this work, we explore human-AI partnerships to facilitate high diversity and accuracy in LLM-based text data generation. We first examine two approaches to diversify text generation: 1) logit suppression, which minimizes the generation of languages that have already been frequently generated, and 2) temperature sampling, which flattens the token sampling probability. We found that diversification approaches can increase data diversity but often at the cost of data accuracy (i.e., text and labels being appropriate for the target domain). To address this issue, we examined two human interventions, 1) label replacement (LR), correcting misaligned labels, and 2) out-of-scope filtering (OOSF), removing instances that are out of the user’s domain of interest or to which no considered label applies. With oracle studies, we found that LR increases the absolute accuracy of models trained with diversified datasets by 14.4%. Moreover, we found that some models trained with data generated with LR interventions outperformed LLM-based few-shot classification. In contrast, OOSF was not effective in increasing model accuracy, implying the need for future work in human-in-the-loop text data generation.

Added

2026-09-26