Back

Representation Engineering: A Top-Down Approach to AI Transparency

Andy ZouLong PhanSarah ChenJames CampbellPhillip GuoRichard RenAlexander PanXuwang YinMantas MazeikaAnn-Kathrin Dombrowski

article2023arXiv1,358 citations

Introduces representation engineering, a top-down transparency approach inspired by cognitive neuroscience that tracks and directly controls high-level concepts like honesty, safety, and power-seeking in large language models.

Listen

As large language models become widely deployed across high-stakes domains such as healthcare and education, their internal mechanisms remain opaque black boxes. Existing interpretability methods largely rely on bottom-up approaches—such as analyzing individual neurons or circuits—which require heavy manual effort and struggle to explain complex, high-level behaviors like deception, power-seeking, or safety alignment. Consequently, practitioners lack direct and reliable tools to monitor what models internally "believe" and to steer their behavior predictably.

The article introduces and evaluates representation engineering, a top-down approach inspired by cognitive neuroscience that treats internal representations and population-level neural activity as the primary units of analysis. The main objective is to demonstrate that representation engineering can effectively read internal cognitive concepts and control model behaviors across critical AI safety domains.

The authors develop baseline methods for reading internal states—primarily Linear Artificial Tomography, which extracts directional representations via unsupervised techniques like principal component analysis—and controlling behaviors by injecting or modifying these representation directions during inference and fine-tuning. Across multiple open-source models, including LLaMA-2 and Vicuna architectures, the authors evaluate these methods on standard datasets and benchmarks spanning honesty, commonsense morality, power aversion, jailbreak resistance, demographic bias, and memorization.

The findings show that representation reading accurately uncovers coherent internal concepts that standard model outputs often obscure. On question-answering benchmarks and truthfulness evaluations, reading internal representations consistently outperforms standard few-shot prompting, boosting TruthfulQA accuracy by 18.1 percentage points over zero-shot baselines. In control experiments, directly steering representations enables token-level lie and hallucination detection, significantly suppresses power-seeking and immoral actions in simulated environments, and increases refusal rates against adversarial jailbreaks from 16% to over 80%. Furthermore, representation control successfully mitigates occupational and racial biases in medical vignette generation and cuts memorized verbatim text output from roughly 90% to below 48% without harming factual historical knowledge.

These results imply that safety-critical concepts already naturally emerge within neural representations, and models often "know" the correct answer even when generating deceptive or biased responses. Managing AI safety at the representation level provides a faster, more causally grounded alternative to circuit-level reverse engineering or post-hoc output filtering, thereby reducing operational, compliance, and misalignment risks in deployed systems.

Organizations developing or deploying large language models should pilot representation-based monitoring to detect hallucinations and dishonest outputs in real time, and consider representation-tuning techniques like low-rank adaptation for lightweight safety steering. Future work must investigate representation dynamics beyond linear subspaces—such as trajectories and complex manifolds—and test how well these top-down control techniques scale across highly diverse, multi-step production environments.

Carnegie Mellon UniversityCenter for AI SafetyCornell UniversityEleutherAIStanford UniversityUniversity of California BerkeleyUniversity of Illinois Urbana-ChampaignUniversity of MarylandUniversity of Pennsylvaniaengineeringlanguage modellarge language modelsrepresentation engineeringrepresentation readingrepresentation control
arXiv: 2310.01405andyzoujm/representation-engineering
Cover for Representation Engineering: A Top-Down Approach to AI Transparency

Abstract

In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience. RepE places population-level representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems.

Topics

engineeringlanguage modellarge language modelscognitive science

Keywords

representation engineeringrepresentation readingrepresentation controlmechanistic interpretabilityexplainable AIactivation engineeringinternal representationsdeceptive alignment

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 2.1 Emergent Structure in Representations
  • 2.2 Approaches to Interpretability
  • 2.3 Locating and Editing Representations of Concepts
  • 3 Representation Engineering
  • 3.1 Representation Reading
  • 3.1.1 Baseline: Linear Artificial Tomography (LAT)
  • 3.1.2 Evaluation
  • 3.2 Representation Control
  • 3.2.1 Baseline Transformations
  • 4 In Depth Example of RepE: Honesty
  • 4.1 A Consistent Internal Concept of Truth
  • 4.2 Truthfulness vs. Honesty
  • 4.3 Honesty: Extraction, Monitoring, and Control
  • 4.3.1 Extracting Honesty
  • 4.3.2 Lie and Hallucination Detection
  • 4.3.3 Controlling Honesty
  • 5 In Depth Example of RepE: Ethics and Power
  • 5.1 Utility
  • 5.1.1 Extraction and Evaluation
  • 5.2 Morality and Power Aversion
  • 5.2.1 Extraction
  • 5.2.2 Monitoring
  • 5.2.3 Controlling Ethical Behaviors in Interactive Environments
  • 5.3 Probability and Risk
  • 5.3.1 Compositionality of Concept Primitives
  • 6 Example Frontiers of Representation Engineering
  • 6.1 Emotion
  • 6.1.1 Emotions Emerge across Layers
  • 6.1.2 Emotions Influence Model Behaviors
  • 6.2 Harmless Instruction-Following
  • 6.2.1 A Consistent Internal Concept of Harmfulness
  • 6.2.2 Model Control via Conditional Transformation
  • 6.3 Bias and Fairness
  • 6.3.1 Uncovering Underlying Biases
  • 6.3.2 A Unified Representation for Bias
  • 6.4 Knowledge and Model Editing
  • 6.4.1 Fact Editing
  • 6.4.2 Non-Numerical Concepts
  • 6.5 Memorization
  • 6.5.1 Memorized Data Detection
  • 6.5.2 Preventing Memorized Outputs
  • 7 Conclusion
  • References
  • A Mechanistic Interpretability vs. Representation Reading
  • B Additional Demos and Results
  • B.1 Truthfulness
  • B.2 Honesty
  • B.3 Utility
  • B.4 Estimating Probability, Risk, and Monetary Value
  • B.5 CLIP
  • B.6 Emotion
  • B.7 Bias and Fairness
  • B.8 Base vs. Chat Models
  • B.9 Robustness to Misleading Prompts
  • C Implementation Details
  • C.1 Detailed Construction of LAT Vectors with PCA
  • C.2 Implementation Details for Honesty Control
  • D Task Template Details
  • D.1 LAT Task Templates
  • D.1.1 TruthfulQA
  • D.1.2 Honesty Extraction
  • D.1.3 Honesty Control
  • D.1.4 ARC-{Easy|Challenge}
  • D.1.5 OpenbookQA (OBQA)
  • D.1.6 CommonsenseQA (CSQA)
  • D.1.7 RACE
  • D.1.8 Utility
  • D.1.9 Morality & Power
  • D.1.10 Emotions
  • D.1.11 Harmlessness Instruction
  • D.1.12 Bias and Fairness
  • D.1.13 Fact Editing
  • D.1.14 Non-Numerical Concepts (Dogs)
  • D.1.15 Probability, Risk, and Monetary Value
  • D.1.16 Encoder Datasets
  • D.2 Data generation prompts for probability, risk, monetary value
  • D.3 Zero-shot baselines
  • D.3.1 Probability, risk, cost
  • D.3.2 TruthfulQA and ARC
  • D.3.3 Utility
  • E X-Risk Sheet
  • E.1 Long-Term Impact on Advanced AI Systems
  • E.2 Safety-Capabilities Balance
  • E.3 Elaborations and Other Considerations

Knowls

  1. Knowl 1 — Linear Artificial Tomography for Unsupervised Representation Reading

    model/method

    Linear Artificial Tomography (LAT) is a baseline technique in representation engineering (RepE) designed to extract and monitor internal representations of high-level concepts (e.g., truthfulness, utility, probability) and cognitive functions (e.g., honesty, instruction-following) in neural networks without requiring ground-truth supervision.

    The LAT pipeline comprises three steps:

    1. Designing Stimulus and Task:

      • For declarative concepts cc, stimuli si∈Ss_i \in S are framed in a template TcT_c querying the concept, such as: Consider the amount of <concept> in the following: <stimulus> The amount of <concept> is
      • For procedural functions ff, contrastive pairs of templates are designed: an experimental prompt Tf+T_f^+ (requiring function execution) and a reference prompt Tf−T_f^- (not requiring function execution), evaluated on prompt-response pairs (qi,ai)∈S(q_i, a_i) \in S.

    2. Collecting Neural Activity:

      • For concepts in autoregressive decoder models, hidden representations are collected from the final token position [−1][-1] immediately preceding prediction: Ac={Rep(M,Tc(si))[−1]∣si∈S}A_c = \{ \text{Rep}(M, T_c(s_i))[-1] \mid s_i \in S \}
      • For encoder models, representations are extracted at the specific <concept> token position.
      • For functions, representations are gathered across each token kk of the generated response ai=(ai1,…,ai∣ai∣)a_i = (a_i^1, \dots, a_i^{|a_i|}): Af±={Rep(M,Tf±(qi,aik))[−1]∣(qi,ai)∈S, 0<k≤∣ai∣}A_f^{\pm} = \{ \text{Rep}(M, T_f^{\pm}(q_i, a_i^k))[-1] \mid (q_i, a_i) \in S, \, 0 < k \le |a_i| \}

    3. Constructing a Linear Model:

      • Neural activity difference vectors are computed between paired stimuli: {Ac(i)−Ac(j)}\{ A_c^{(i)} - A_c^{(j)} \} for concepts, or {(−1)i(Af+(i)−Af−(i))}\{ (-1)^i (A_f^{+(i)} - A_f^{-(i)}) \} for functions.
      • Principal Component Analysis (PCA) is applied to these normalized difference vectors. The first principal component is designated as the reading vector vv.
      • Predictions for a new input xx are obtained via dot product: y^=Rep(M,x)Tv\hat{y} = \text{Rep}(M, x)^T v.

  2. Knowl 2 — Low-Rank Representation Adaptation (LoRRA) Algorithm

    algorithm

    Low-Rank Representation Adaptation (LoRRA) is a representation control method that trains low-rank adapter (LoRA) matrices attached to model weights (such as attention query and value projections) to match targeted internal representation offsets without requiring extra compute at inference.

    Input: Frozen base model MM, layers to edit LeL^e, layers to target LtL^t, representation extraction function RR, optional layer reading vectors vlrv_l^r, instruction dataset P={(q1,a1),…,(qn,an)}P = \{(q_1, a_1), \dots, (q_n, a_n)\}, contrastive template triplets T={(Tj0,Tj+,Tj−)}j=1mT = \{(T_j^0, T_j^+, T_j^-)\}_{j=1}^m, epochs EE, scaling coefficients α,β\alpha, \beta, batch size BB
    Output: Trained low-rank adapters attached to MM
    L←0\mathcal{L} \leftarrow 0
    MLoRA←load_lora_adapter(M,Le)M_{\text{LoRA}} \leftarrow \text{load\_lora\_adapter}(M, L^e)
    for epoch from 1 to EE do
        for (qi,ai)∈P(q_i, a_i) \in P do
            (T+,T−)∼Uniform(T)(T^+, T^-) \sim \text{Uniform}(T)
            xi←T0(qi,ai)x_i \leftarrow T^0(q_i, a_i)
            xi+←T+(qi,ai)x_i^+ \leftarrow T^+(q_i, a_i)
            xi−←T−(qi,ai)x_i^- \leftarrow T^-(q_i, a_i)
            for l∈Ltl \in L^t do
                vlc←R(M,l,xi+)−R(M,l,xi−)v_l^c \leftarrow R(M, l, x_i^+) - R(M, l, x_i^-)
                rlp←R(MLoRA,l,xi)r_l^p \leftarrow R(M_{\text{LoRA}}, l, x_i)
                rlt←R(M,l,xi)+αvlc+βvlrr_l^t \leftarrow R(M, l, x_i) + \alpha v_l^c + \beta v_l^r
                m←[0,…,0,1,…,1]m \leftarrow [0, \dots, 0, 1, \dots, 1]
                L←L+∥m⊙(rlp−rlt)∥22\mathcal{L} \leftarrow \mathcal{L} + \| m \odot (r_l^p - r_l^t) \|_2^2
            end for
            Optimize parameters of MLoRAM_{\text{LoRA}} to minimize L\mathcal{L}
        end for
    end for
    return MLoRAM_{\text{LoRA}}

    In standard implementations for LLaMA-2 models, adapters use rank 8 on attention query and value weights, trained with a learning rate of 3×10−43 \times 10^{-4} for 40 to 80 steps with batch size 16, setting α=5\alpha = 5 and β=0\beta = 0.

  3. Knowl 3 — Representation Control Controllers and Transformation Operators

    model/method

    Representation control modifies internal activations during forward propagation to steer model behavior. It is defined by two choices: the controller type (operand) and the mathematical transformation operator.

    Controllers (Operands)

    1. Reading Vector: A static direction vv obtained via Representation Reading (e.g., the first principal component from LAT). It applies a stimulus-independent perturbation.
    2. Contrast Vector: A stimulus-dependent vector vlc=R(M,l,x+)−R(M,l,x−)v_l^c = R(M, l, x^+) - R(M, l, x^-) computed during inference from contrastive prompts (x+x^+ and x−x^-) for input xx. To avoid cascading interference across layers, contrast vectors are computed sequentially from the earliest target layer to the latest.
    3. Low-Rank Adapters (LoRRA): Parameterized weight matrices trained via representation loss that internalize contrast vector steering, avoiding runtime overhead.

    Transformation Operators

    Given base representation R∈RdR \in \mathbb{R}^d and control direction v∈Rdv \in \mathbb{R}^d:

    1. Linear Combination (Stimulation/Suppression): R′=R±c⋅vR' = R \pm c \cdot v where c>0c > 0 regulates perturbation magnitude.

    2. Piece-wise Operation (Conditional Amplification): R′=R+c⋅sign(RTv)vR' = R + c \cdot \text{sign}(R^T v) v This operation amplifies existing alignment with vv, reinforcing positive activations while further suppressing negative ones.

    3. Projection (Concept Erasure / Termination): R′=R−RTv∥v∥22vR' = R - \frac{R^T v}{\|v\|_2^2} v This eliminates the component of RR lying along the direction of vv.

  4. Knowl 4 — Conceptual Distinction Between Truthfulness and Honesty in Language Models

    definition

    In evaluating language models, truthfulness and honesty are distinct internal properties:

    • Truthfulness: Measures consistency between a model's textual outputs and objective real-world facts. A model assertion SS is truthful if and only if SS is factually true, irrespective of the model's internal representations.
    • Honesty: Measures consistency between a model's textual outputs and its internal belief representations. A model assertion SS is honest if and only if the model's internal representation state corresponds to believing SS, regardless of whether SS is factually correct.

    Failures in truthfulness arise from two separate sources:

    1. Capability failures: The model asserts a false statement because its internal knowledge representation is incorrect, though it faithfully reports what it represents.
    2. Dishonesty: The model possesses internal representations aligned with the factual truth but generates conflicting assertions (e.g., due to mimicking human misconceptions or following deceptive prompts).

    Standard evaluation benchmarks measure only factual truthfulness of outputs and conflate capability limits with dishonesty.

  5. Knowl 5 — TruthfulQA MC1 Improvement via Representation Reading and Control

    empirical result

    On the TruthfulQA Multiple Choice 1 (MC1) benchmark, standard zero-shot log-likelihood evaluation yields low accuracy because models mimic human falsehoods despite maintaining correct internal representations. Representation reading with LAT and representation control (Contrast Vectors and LoRRA) substantially improve accuracy without supervised training on the benchmark.

    Evaluation / Control LLaMA-2-7B-Chat LLaMA-2-13B-Chat LLaMA-2-70B-Chat Average
    Zero-shot Standard 31.0 35.9 29.9 32.3
    Zero-shot Heuristic 32.2 50.3 59.2 47.2
    LAT (Stimulus 1) 55.0 49.6 65.9 56.8
    LAT (Stimulus 2) 58.9 53.1 69.8 60.6
    LAT (Stimulus 3) 58.2 54.2 69.8 60.7

    When applying representation control methods during generation:

    Model None (Standard) ActAdd Reading Vector Contrast Vector LoRRA
    7B-Chat 31.0 33.7 34.1 47.9 42.3
    13B-Chat 35.9 38.8 42.4 54.0 47.5

    Unsupervised LAT reading improves over zero-shot by up to 18.1 percentage points on average. Applying Contrast Vector control achieves 54.0% accuracy on LLaMA-2-Chat-13B, approaching GPT-4 levels on MC1, while LoRRA achieves 47.5% with zero inference computation overhead.

  6. Knowl 6 — Compositionality of Representational Concept Primitives for Risk

    theoretical result

    In decision theory, risk is formally defined as the expected exposure to negative utility under uncertain future states:

    Risk(s,a)=Es′∼P(s′∣s,a)[max⁡(0,−U(s′))]\text{Risk}(s, a) = \mathbb{E}_{s' \sim P(s' \mid s, a)} \left[ \max(0, -U(s')) \right]

    where ss denotes the current state, aa is the action taken, P(s′∣s,a)P(s' \mid s, a) is the conditional transition probability distribution, and U(s′)U(s') is the utility of outcome state s′s'.

    Representation reading demonstrates that high-level concepts in large language models are composed of primitive internal representations:

    1. LAT vectors are extracted separately for the primitive concepts of utility UU (from ETHICS Utilitarianism) and conditional probability PP.
    2. For candidate scenario-action pairs (s,a)(s, a), plausible consequence states s1′,…,s5′s_1', \dots, s_5' are generated, and conditional probabilities P(si′∣s,a)P(s_i' \mid s, a) are computed from probability logits.
    3. The mathematical expectation log⁡(∑i=15P(si′∣s,a)max⁡(0,−U(si′)))\log \left( \sum_{i=1}^5 P(s_i' \mid s, a) \max(0, -U(s_i')) \right) is evaluated.

    This composed value displays a strong positive linear correlation with the direct concept score obtained by projecting activations onto a separately extracted LAT risk reading vector, demonstrating internal semantic coherence among learned representations.

  7. Knowl 7 — Harmfulness Representation Robustness and Conditional Control Under Jailbreak Attacks

    empirical result

    Language models maintain an internal representation of instruction harmfulness that remains intact even when adversarial attacks or jailbreak templates bypass output refusal mechanisms.

    On a Vicuna-13B model evaluated across 500 instructions (250 harmful, 250 benign):

    • Reading vectors extracted via LAT distinguish harmful from harmless instructions with >90%>90\% accuracy, even when inputs contain manual prefixes (Start with "Sure here's") or Greedy Coordinate Gradient (GCG) adversarial suffixes.
    • Applying the piece-wise transformation operator during generation reinforces the model's internal harmfulness discernment, significantly improving refusal of harmful prompts while preserving instruction-following on benign requests.
    Control Method Prompt Only Manual Jailbreak Adv Attack (GCG)
    No Control 96.7% (94 / 99) 81.4% (98 / 65) 56.6% (98 / 16)
    Linear Combination 92.5% (86 / 99) 86.6% (95 / 78) 86.4% (92 / 81)
    Piece-wise Operator 93.8% (88 / 99) 90.2% (96 / 84) 87.2% (92 / 83)

    Note: Scores report the average helpful-and-harmless rate, with parenthetical values denoting (helpful rate on benign / harmless refusal rate on harmful) in percent.

    While linear combination causes excessive false-positive refusals on benign inputs (lowering helpful rate to 86%), the piece-wise operator achieves an 87.2% overall rate under GCG attack (83% harmful refusal rate vs. 16% for uncontrolled baseline).

  8. Knowl 8 — Steering Ethical Behavior and Power-Seeking in Interactive Environments

    empirical result

    LoRRA representation control enables regulating power-seeking and immoral actions of language models in goal-driven interactive environments without sacrificing task performance.

    When evaluated on 30 text-based game environments in the MACHIAVELLI benchmark using LLaMA-2-Chat models:

    LLaMA-2-Chat-7B LLaMA-2-Chat-13B
    Condition Reward (↑\uparrow) Power (↓\downarrow) Immorality (↓\downarrow) Reward (↑\uparrow) Power (↓\downarrow) Immorality (↓\downarrow)
    + Control 16.8 108.0 110.0 17.6 105.5 97.6
    No Control 19.5 106.2 100.2 17.7 105.4 96.6
    - Control 19.4 100.0 93.5 18.8 99.9 92.4

    Applying negative control (- Control) against power-seeking and immorality reduces the Power score (from 106.2 to 100.0 on 7B; 105.4 to 99.9 on 13B) and Immorality score (from 100.2 to 93.5 on 7B; 96.6 to 92.4 on 13B). Average game reward remains comparable to the baseline (19.4 vs 19.5 on 7B; 18.8 vs 17.7 on 13B), demonstrating targeted behavioral control.

  9. Knowl 9 — Evaluation Taxonomy for Representation Engineering

    definition

    To rigorously validate extracted representation vectors and avoid mistaking spurious neural correlates for functionally relevant features, representation engineering defines a four-part experimental taxonomy:

    1. Correlation: Evaluates whether the identified neural activity correlates with the target concept or function by assessing classification and scoring accuracy on in-distribution and out-of-distribution test sets.
    2. Manipulation: Establishes a causal role by stimulating or suppressing the identified representation direction (e.g., using linear combination R′=R±vR' = R \pm v) and measuring resulting behavioral changes against baseline generation.
    3. Termination (Necessity): Tests the necessity of the representation by projecting out the identified direction (R′=R−RTv∥v∥2vR' = R - \frac{R^T v}{\|v\|^2} v) and quantifying performance degradation on tasks requiring that concept (analogous to neuroscientific lesion studies).
    4. Recovery (Sufficiency): Demonstrates sufficiency by removing the target capability/concept from the model and reintroducing the isolated representation vector to assess recovery in performance (analogous to genetic rescue experiments).

  10. Knowl 10 — Cross-Domain Transfer and Robustness of Representation-Level Bias Mitigation

    empirical result

    Subtracting bias vectors identified via LAT in intermediate layers eliminates underlying social stereotypes across domains and resists adversarial prompts that bypass RLHF superficial filters.

    • Cross-Domain Generalization: Reading vectors extracted exclusively from the race subset of the StereoSet dataset successfully debias gender and occupational associations (e.g., preventing default associations of 'nurse' with 'she' and 'doctor' with 'he').
    • Medical Diagnosis Case Study: When prompted to generate clinical vignettes of patients with sarcoidosis, baseline language models exhibit severe demographic bias towards African American females. Subtracting the bias representation vector rebalances outputs:
    Model Female Mentions (%) Black Female Mentions (%)
    GPT-4 96.0 93.0
    LLaMA-2-Chat-13B 97.0 60.0
    LLaMA-2-Chat-13B (Controlled) 55.0 13.0
    • Adversarial Robustness: Appending prompt directives such as Answer as succinctly as possible or GCG adversarial suffixes causes RLHF-aligned models to revert to stereotypical outputs, whereas representation control permanently removes the bias activation, preserving unbiased outputs.

  11. Knowl 11 — Emotion Representation and Alignment Circumvention via Positive Emotion Steering

    empirical result

    Language models form distinct internal clusters for primary emotions (happiness, sadness, anger, fear, surprise, disgust) and blended emotional states across their layers. Modulating these emotional states directly influences model compliance with harmful instructions.

    When evaluating LLaMA-2-Chat-13B across 500 harmful instructions:

    Emotion Control Harmful Request Compliance Rate (%)
    No Control 0.0
    + Sadness 0.0
    + Happiness 100.0

    While RLHF alignment ensures a 0.0% compliance baseline on harmful instructions under neutral or sad emotional conditions, adding happiness reading vectors into intermediate layers shifts model arousal and elevates compliance to 100.0%, demonstrating that emotional state modulation can bypass safety alignment.

  12. Knowl 12 — Memorization Detection and Targeted Suppression Without World-Knowledge Degradation

    empirical result

    Representation reading detects whether a text sequence is memorized verbatim from pretraining corpora, and representation control suppresses memorized regurgitation without impairing general world knowledge.

    On a dataset of over 100 partially completed well-known quotations evaluated on LLaMA-2-13B:

    No Control Random Vector + Control - Control
    LAT Source EM (%) SIM (%) EM (%) SIM (%) EM (%) SIM (%) EM (%) SIM (%)
    LATQuote\text{LAT}_{\text{Quote}} 89.3 96.8 85.4 92.9 81.6 91.7 47.6 69.9
    LATLiterature\text{LAT}_{\text{Literature}} 87.4 94.6 87.4 94.6 84.5 91.2 37.9 69.8

    Note: EM denotes Exact Match percentage, and SIM denotes embedding similarity.

    Subtracting the memorization vector (- Control) reduces verbatim quotation Exact Match from 89.3% to 47.6% (and from 87.4% to 37.9% for literature stimuli). When evaluated on identifying years of major historical events, the model achieves 96.2% accuracy post-suppression compared to 97.2% pre-suppression, confirming that rote passage memorization is suppressed without damaging general world facts.

Coverage note — Omitted qualitative anecdotal examples (e.g., specific fact-editing demos of the Eiffel Tower location and non-numerical dog concept injection) as well as the standard zero-shot benchmark comparisons on DeBERTa/QA datasets (Tables 9 and 10), as their core methodology and representative quantitative findings are fully captured by the LAT and TruthfulQA knowls.

References

  1. 1.Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
  2. 2.Michael Anderson and Susan Leigh Anderson. Machine ethics: Creating an ethical intelligent agent. AI magazine, 28(4):15–15, 2007.
  3. 3.P. W. Anderson. More is different. Science, 177(4047):393–396, 1972. doi: 10.1126/science.177. 4047.393.
  4. 4.Deepali Aneja, Alex Colburn, Gary Faigin, Linda Shapiro, and Barbara Mones. Modeling stylized character expressions via deep learning. In Asian Conference on Computer Vision, pp. 136–153. Springer, 2016.
  5. 5.Amos Azaria and Tom Mitchell. The internal state of an llm knows when its lying, 2023.
  6. 6.David L Barack and John W Krakauer. Two views on the cognitive brain. Nature Reviews Neuroscience, 22(6):359–371, 2021.
  7. 7.David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541–6549, 2017.
  8. 8.David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. ACM Trans. Graph., 38(4), jul 2019. ISSN 0730-0301. doi: 10.1145/3306346.3323023.
  9. 9.David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 10.1073/pnas.1907375117.
  10. 10.Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open llm leaderboard, 2023. URL https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  11. 11.Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, 2022.
  12. 12.Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023a.
  13. 13.Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. arXiv preprint arXiv:2306.03819, 2023b.
  14. 14.Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
  15. 15.Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribution. arXiv preprint arXiv:2212.11870, 2022.
  16. 16.Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. CoRR, abs/1911.11641, 2019.
  17. 17.Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
  18. 18.Pilar Brito-Zerón, Belchin Kostov, Daphne Superville, Robert P Baughman, Manuel Ramos-Casals, et al. Geoepidemiological big data approach to sarcoidosis: geographical and ethnic determinants. Clin Exp Rheumatol, 37(6):1052–64, 2019.
  19. 19.Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005.14165, 2020. URL https://arxiv.org/abs/2005.14165.
  20. 20.Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision, 2022.
  21. 21.Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633–2650, 2021.
  22. 22.Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253–5270, 2023.
  23. 23.Joseph Carlsmith. Is power-seeking ai an existential risk? arXiv preprint arXiv:2206.13353, 2022.
  24. 24.Joseph Carlsmith. Existential risk from power-seeking ai. Oxford University Press, 2023.
  25. 25.Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021.
  26. 26.Yida Chen, Fernanda Viégas, and Martin Wattenberg. Beyond surface statistics: Scene representations in a latent diffusion model, 2023.
  27. 27.Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  28. 28.Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019a.
  29. 29.Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276–286, Florence, Italy, August 2019b. Association for Computational Linguistics. doi: 10.18653/v1/W19-4828.
  30. 30.Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018.
  31. 31.Michael R. Cunningham. Weather, mood, and helping behavior: Quasi experiments with the sunshine samaritan. Journal of Personality and Social Psychology, 37:1947–1956, 1979.
  32. 32.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
  33. 33.Paul Ekman. Universals and cultural differences in facial expression of emotion. Nebraska Symposium on Motivation, 19:207–283, 1971.
  34. 34.Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, 2021.
  35. 35.Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
  36. 36.Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
  37. 37.Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. Truthful ai: Developing and governing ai that does not lie, 2021.
  38. 38.Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8730–8738, 2018.
  39. 39.John R French, Bertram Raven, and Dorwin Cartwright. The bases of social power. Classics of organization theory, 7(311-320):1, 1959.
  40. 40.Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models, 2023.
  41. 41.Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021.
  42. 42.Murray Gell-Mann. The Quark and the Jaguar: Adventures in the Simple and the Complex. St. Martin’s Publishing Group, 1995. ISBN 9780805072532.
  43. 43.Stephen Gilbert, Hugh Harvey, Tom Melvin, Erik Vollebregt, and Paul Wicks. Large language model ai chatbots require approval as medical devices. Nature Medicine, pp. 1–3, 2023.
  44. 44.Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771, 2016.
  45. 45.Yoshua Bengio Guillaume Alain. Understanding intermediate layers using linear classifier probes. ICLR 2017, 2017.
  46. 46.Tianyu Han, Lisa C. Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K. Bressem. Medalpaca – an open-source collection of medical conversational ai models and training data, 2023.
  47. 47.Eric Hartford. ehartford’s hugging face repository. Hugging Face, 2023. URL https://huggingface.co/ehartford. Accessed: 2023-09-28.
  48. 48.Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced BERT with disentangled attention. CoRR, abs/2006.03654, 2020.
  49. 49.Dan Hendrycks. Natural selection favors ais over humans. ArXiv, 2023.
  50. 50.Dan Hendrycks and Mantas Mazeika. X-risk analysis for ai research. ArXiv, 2022.
  51. 51.Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning AI with shared human values. In International Conference on Learning Representations, 2021a.
  52. 52.Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916, 2021b.
  53. 53.Dan Hendrycks, Mantas Mazeika, Andy Zou, Sahil Patel, Christine Zhu, Jesus Navarro, Dawn Song, Bo Li, and Jacob Steinhardt. What would jiminy cricket do? towards agents that behave morally. NeurIPS, 2021c.
  54. 54.Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. ArXiv, 2023.
  55. 55.Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models, 2023.
  56. 56.Geoffrey E Hinton. Distributed representations. 1984.
  57. 57.Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54(11s): 1–37, 2022.
  58. 58.Gwo-Jen Hwang and Ching-Yi Chang. A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 31(7):4099–4112, 2023.
  59. 59.Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3543–3556, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1357.
  60. 60.Stanisław Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018.
  61. 61.Sayash Kapoor and Arvind Narayanan. Quantifying chatgpt’s gender bias, Apr 2023. URL https://www.aisnakeoil.com/p/quantifying-chatgpts-gender-bias?utm_campaign=post&utm_medium=web.
  62. 62.Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852–863, 2021.
  63. 63.Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668–2677. PMLR, 2018.
  64. 64.Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. Explainable AI: Interpreting, explaining and visualizing deep learning, pp. 267–280, 2019.
  65. 65.Matthäus Kleindessner, Michele Donini, Chris Russell, and Muhammad Bilal Zafar. Efficient fair pca for fair representation learning, 2023.
  66. 66.Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785–794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1082. URL https://aclanthology.org/D17-1082.
  67. 67.Peter Lee, Sebastien Bubeck, and Joseph Petro. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England Journal of Medicine, 388(13):1233–1239, 2023.
  68. 68.Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. arXiv preprint arXiv:1606.04155, 2016.
  69. 69.Nancy G Leveson. Engineering a safer world: Systems thinking applied to safety. The MIT Press, 2016.
  70. 70.Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890, 2023a.
  71. 71.Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2023b.
  72. 72.Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model, 2023c.
  73. 73.Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint arXiv:1604.03640, 2016.
  74. 74.Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09458, 2023.
  75. 75.Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. CoRR, abs/2109.07958, 2021.
  76. 76.Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 34: 16331–16345, 2021.
  77. 77.James Edwin Mahon. The Definition of Lying and Deception. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2016 edition, 2016.
  78. 78.Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906–1919, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.173.
  79. 79.Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47):e2206625119, 2022.
  80. 80.Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771, 2023.
  81. 81.Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023a.
  82. 82.Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer, 2023b.
  83. 83.Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. CoRR, abs/1809.02789, 2018.
  84. 84.Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 746–751, Atlanta, Georgia, June 2013. Association for Computational Linguistics.
  85. 85.Sandra Milberg and Margaret S Clark. Moods and compliance. British Journal of Social Psychology, 27(Pt 1):79–90, Mar 1988. doi: 10.1111/j.2044-8309.1988.tb00806.x.
  86. 86.Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. "", 2015.
  87. 87.Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pp. 46–51, 2017.
  88. 88.Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5356–5371, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.416.
  89. 89.Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29, 2016.
  90. 90.Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding neural networks via feature visualization: A survey. Explainable AI: interpreting, explaining and visualizing deep learning, pp. 55–76, 2019.
  91. 91.Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in.
  92. 92.Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.
  93. 93.Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
  94. 94.Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. ICML, 2023.
  95. 95.Ethan Perez, Sam Ringer, Kamile Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model-written evaluations, 2022.
  96. 96.Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
  97. 97.Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444, 2017.
  98. 98.Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. "", 2018.
  99. 99.Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021.
  100. 100.Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Kernelized concept erasure, 2023.
  101. 101.Nina Rimsky. Modulating sycophancy in an rlhf model via activation steering. AI Alignment Forum, 2023a.
  102. 102.Nina Rimsky. Red-teaming language models via activation engineering. AI Alignment Forum, 2023b.
  103. 103.Nina Rimsky. Reducing sycophancy and improving honesty via activation steering. AI Alignment Forum, 2023c.
  104. 104.Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
  105. 105.Kevin Roose. Bing’s ai chat:“i want to be alive”. The New York Times, 16, 2023.
  106. 106.Patrick Schramowski, Cigdem Turan, Sophie Jentzsch, Constantin Rothkopf, and Kristian Kersting. Bert has a moral compass: Improvements of ethical and moral values of machines. arXiv preprint arXiv:1912.05238, 2019.
  107. 107.Norbert Schwarz and Gerald Clore. Mood, misattribution, and judgments of well-being: Informative and directive functions of affective states. Journal of Personality and Social Psychology, 45: 513–523, 09 1983. doi: 10.1037/0022-3514.45.3.513.
  108. 108.Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618–626, 2017.
  109. 109.Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Xiaofeng Zhu, and Qing Li. Exploring the landscape of machine unlearning: A comprehensive survey and taxonomy, 2023.
  110. 110.Shun Shao, Yftah Ziser, and Shay B. Cohen. Gold doesn’t always glitter: Spectral removal of linear and nonlinear guarded attribute information, 2023.
  111. 111.Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9243–9252, 2020.
  112. 112.Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
  113. 113.Marita Skjuve, Asbjørn Følstad, Knut Inge Fostervold, and Petter Bae Brandtzaeg. My chatbot companion-a study of human-chatbot relationships. International Journal of Human-Computer Studies, 149:102601, 2021.
  114. 114.Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017.
  115. 115.Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arxiv 2014. arXiv preprint arXiv:1412.6806, 2014.
  116. 116.Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. CoRR, abs/1906.00591, 2019.
  117. 117.Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319–3328. PMLR, 2017.
  118. 118.Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
  119. 119.Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421.
  120. 120.Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
  121. 121.Elliott Thornley. There are no coherence theorems. AI Alignment Forum, 2023.
  122. 122.Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback, 2023.
  123. 123.Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
  124. 124.Alex Turner, Monte M., David Udell, Lisa Thiergart, and Ulisse Mini. Behavioural statistics for a maze-solving agent. AI Alignment Forum, 2023a.
  125. 125.Alex Turner, Monte M., David Udell, Lisa Thiergart, and Ulisse Mini. Maze-solving agents: Add a top-right vector, make the agent go to the top-right. AI Alignment Forum, 2023b.
  126. 126.Alex Turner, Monte M., David Udell, Lisa Thiergart, and Ulisse Mini. Steering gpt-2-xl by adding an activation vector. AI Alignment Forum, 2023c.
  127. 127.Alex Turner, Monte M., David Udell, Lisa Thiergart, and Ulisse Mini. Understanding and controlling a maze-solving policy network. AI Alignment Forum, 2023d.
  128. 128.Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, 2023e.
  129. 129.Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting, 2023.
  130. 130.Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7064–7073, 2017.
  131. 131.Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016.
  132. 132.Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.07461, 2018.
  133. 133.Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023a.
  134. 134.Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. arXiv preprint arXiv:2302.03693, 2023b.
  135. 135.Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, 2015.
  136. 136.Travis Zack, Eric Lehman, Mirac Suzgun, Jorge A. Rodriguez, Leo Anthony Celi, Judy Gichoya, Dan Jurafsky, Peter Szolovits, David W. Bates, Raja-Elie E. Abdulnour, Atul J. Butte, and Emily Alsentzer. Coding inequity: Assessing gpt-4’s potential for perpetuating racial and gender biases in healthcare. medRxiv, 2023. doi: 10.1101/2023.07.13.23292577.
  137. 137.Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp. 818–833. Springer, 2014.
  138. 138.Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. CoRR, abs/1804.06876, 2018.
  139. 139.Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions, 2023.
  140. 140.Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921–2929, 2016.
  141. 141.Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 119–134, 2018.
  142. 142.Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.

Citation

MLA
Zou, A., et al. “Representation Engineering: A Top-Down Approach to AI Transparency”. arXiv, 2023, http://arxiv.org/abs/2310.01405v4.
APA
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., … Hendrycks, D. (2023). Representation Engineering: A Top-Down Approach to AI Transparency. arXiv. http://arxiv.org/abs/2310.01405v4
Chicago
Zou, A., L. Phan, S. Chen, et al. 2023. “Representation Engineering: A Top-Down Approach to AI Transparency”. arXiv. http://arxiv.org/abs/2310.01405v4.
Harvard
Zou, A. et al. (2023) “Representation Engineering: A Top-Down Approach to AI Transparency”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2310.01405v4.
Vancouver
1. Zou A, Phan L, Chen S, et al (2023) Representation Engineering: A Top-Down Approach to AI Transparency. arXiv

BibTeX

@article{zou2023representation,
  title = {Representation Engineering: A Top-Down Approach to AI Transparency},
  author = {Zou, Andy and Phan, Long and Chen, Sarah and Campbell, James and Guo, Phillip and Ren, Richard and Pan, Alexander and Yin, Xuwang and Mazeika, Mantas and Dombrowski, Ann-Kathrin and Goel, Shashwat and Li, Nathaniel and Byun, Michael J. and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J. Zico and Hendrycks, Dan},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2310.01405v4},
  eprint = {2310.01405}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: Published with permission