Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

representation clustering

Representation clustering is an unsupervised learning technique that groups data points based on their learned feature representations, latent embeddings, or internal neural network activations rather than their raw input forms. By operating within these transformed, high-level geometric spaces, the approach identifies natural structures, semantic relationships, and latent concepts that may not be apparent in the original input data. In machine learning and model analysis, representation clustering is frequently employed to discover latent knowledge, separate contrasting states or concepts, and categorize complex data distributions without requiring manual annotations or explicit ground-truth supervision.

1 item

Discovering Latent Knowledge in Language Models Without Supervision

Discovering Latent Knowledge in Language Models Without Supervision

Collin Burns, Haotian Ye, Dan Klein, Jacob Steinhardt

OrganizationsPeking UniversityUniversity of California Berkeley

Why you should read this

Introduces an unsupervised technique for extracting truthful latent knowledge directly from language model activations by enforcing logical consistency, enabling accurate question answering even when models are prompted to generate false outputs.

Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4\% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.

Added

2026-09-26