Towards Automated Circuit Discovery for Mechanistic Interpretability

Arthur ConmyAugustine N. Mavor-ParkerAengus LynchStefan HeimersheimAdrià Garriga-Alonso

article2023NeurIPS540 citations

Proposes an automated circuit discovery algorithm that replaces labor-intensive manual patching to efficiently isolate the computational subgraphs responsible for specific behaviors in transformer models.

Listen

Modern artificial intelligence models, such as large transformer language models, are widely deployed across high-stakes domains but operate largely as opaque black boxes. Mechanistic interpretability seeks to demystify these systems by reverse-engineering the internal subgraphs—known as circuits—that execute specific behaviors. Historically, discovering these circuits required labor-intensive, manual experimentation, creating a severe bottleneck that prevents safety and alignment evaluations from keeping pace with rapid model scaling.

The article formalizes the prevailing mechanistic interpretability workflow and introduces an automated method to eliminate the manual burden of circuit isolation. The primary objective is to evaluate whether an automated algorithm can reliably discover the precise internal computational pathways responsible for specific tasks without human intervention.

To accomplish this, the authors develop Automatic Circuit DisCovery (ACDC), an algorithm that systematically traverses a model's computational graph from outputs to inputs, pruning non-essential connections using activation patching and measuring changes against the Kullback-Leibler (KL) divergence. The authors evaluate ACDC across diverse language tasks and models, including GPT-2 Small, toy compiled transformers, and multi-layer attention networks. They benchmark ACDC against adaptations of existing techniques—Subnetwork Probing and Head Importance Score for Pruning—using receiver operating characteristic curves against previously published ground-truth circuits as well as performance metrics on held-out tasks.

The analysis yields four key findings. First, ACDC successfully automates edge-level circuit extraction, outperforming baseline methods across the majority of benchmarked tasks and matching manually identified circuits with high precision. For example, on the Greater-Than task in GPT-2 Small, ACDC accurately selected 68 out of 32,000 candidate edges and recovered all five functional component types identified in prior human-led studies. Second, optimizing for low KL divergence provides a more stable and generalizable pruning objective than maximizing task-specific accuracy metrics. Third, baseline methods show vulnerability to hallucinating non-existent circuits on randomized control networks, whereas ACDC more faithfully reflects genuine algorithmic structure. Fourth, ACDC directly enables new discoveries, uncovering previously unnoticed information flows and token interactions in gendered pronoun prediction in GPT-2 Small.

These results demonstrate that automated circuit discovery is a viable path toward scaling machine learning interpretability and auditing. Automating this pipeline substantially reduces human labor costs and accelerates research into AI safety, risk management, and compliance verification. By isolating exact internal pathways, organizations can better audit models for unwanted behaviors, biases, and vulnerabilities before deployment.

Technical leaders and researchers should adopt automated circuit discovery tools as a screening layer to accelerate model auditing pipelines. However, organizations should avoid relying entirely on ACDC for end-to-end safety guarantees without human oversight. Future work must focus on developing automated methods for the functional interpretation of identified circuits and integrating faster gradient-based approximations to reduce the computational runtime required when scanning large graphs.

Confidence in ACDC’s core pruning logic is high, but stakeholders should note key limitations. Current implementations can miss subtle negative components that dampen specific model behaviors, are sensitive to hyperparameter thresholds and corrupted reference distributions, and struggle to isolate redundant parallel logic such as logical OR gates. Consequently, automated circuit extraction should be treated as an accelerator within a broader testing pipeline rather than a fully autonomous safety proof.

FAR AI University College London University of Cambridge encoder-decoder interpretability automated circuit discovery mechanistic interpretability circuit recovery algorithms

arXiv: 2304.14997 ArthurConmy/Automatic-Circuit-Discovery

Paper: Transformer Feed-Forward Layers Are Key-Value Memories, Mor Geva et al. (2020). It provides foundational insights into reverse-engineering Transformer components by demonstrating how feed-forward layers function as key-value memory networks.
Paper: Learning Important Features Through Propagating Activation Differences, Avanti Shrikumar et al. (2017). It establishes the principle of propagating activation differences to attribute component contributions, which underlies modern activation patching techniques.
Paper: GNNExplainer: Generating Explanations for Graph Neural Networks, Rex Ying et al. (2019). It formalizes the problem of finding minimal explanatory subgraphs within neural computation graphs to explain specific model behaviors.
Paper: Interpretable Explanations of Black Boxes by Meaningful Perturbation, Ruth Fong et al. (2017). It introduces optimization frameworks for identifying minimal necessary components via causal perturbations, a key antecedent to automated causal pruning.
Paper: Network Dissection: Quantifying Interpretability of Deep Visual Representations, David Bau et al. (2017). It develops automated protocols to quantify the interpretability and alignment of individual hidden neural units across model representations.
Paper: Understanding intermediate layers using linear classifier probes, Guillaume Alain et al. (2016). It establishes linear probing to isolate and inspect the functional representations present at intermediate hidden layers of neural architectures.
Paper: BERT Rediscovers the Classical NLP Pipeline, Ian Tenney et al. (2019). It pioneered layerwise edge probing in Transformers to localize specific linguistic sub-tasks across internal computational layers.
Paper: Sanity Checks for Saliency Maps, Julius Adebayo et al. (2018). It formulates essential methodology for rigorously validating whether interpretability algorithms truly reflect learned internal network mechanisms.

Paper: Sparse Autoencoders Find Highly Interpretable Features in Language Models, Hoagy Cunningham et al. (2023). It extends mechanistic interpretability beyond manual or unit-level circuits by using sparse autoencoders to uncover monosemantic feature dictionaries and performing causal mediation patching on task circuits.
Paper: The Dead Salmons of AI Interpretability, Maxime Méloux et al. (2025). It critically examines the statistical fragility and non-identifiability challenges inherent in mechanistic interpretability and circuit discovery methods.
Paper: How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Models, Dun Li Chan et al. (2026). It applies activation patching and mechanistic analysis to evaluate how different types of perturbations propagate through language model components and attention heads.
Paper: Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization, Andy Zeyi Liu et al. (2026). It complements circuit-level mechanistic analysis by introducing spectral diagnostics of activation and gradient geometries to monitor internal language model dynamics.
Paper: There Will Be a Scientific Theory of Deep Learning, Jamie Simon et al. (2026). It synthesizes empirical mechanistic interpretability findings into a broader, emerging scientific theory of learning mechanics in deep neural networks.

Cover for Towards Automated Circuit Discovery for Mechanistic Interpretability

Abstract

Through considerable effort and intuition, several recent works have reverse-engineered nontrivial behaviors of transformer models. This paper systematizes the mechanistic interpretability process they followed. First, researchers choose a metric and dataset that elicit the desired model behavior. Then, they apply activation patching to find which abstract neural network units are involved in the behavior. By varying the dataset, metric, and units under investigation, researchers can understand the functionality of each component. We automate one of the process' steps: to identify the circuit that implements the specified behavior in the model's computational graph. We propose several algorithms and reproduce previous interpretability results to validate them. For example, the ACDC algorithm rediscovered 5/5 of the component types in a circuit in GPT-2 Small that computes the Greater-Than operation. ACDC selected 68 of the 32,000 edges in GPT-2 Small, all of which were manually found by previous work. Our code is available at this https URL.

Topics

encoder-decoder interpretability

Keywords

automated circuit discovery mechanistic interpretability circuit recovery algorithms reverse engineering subnetwork probing IOI circuit model activations internal model components ground-truth circuit

1 Introduction
2 The Mechanistic Interpretability Workflow
2.1 Step 1: Select a behavior, dataset, and metric
2.2 Step 2: Divide the neural network into a graph of smaller units
2.3 Step 3: Patch model activations to isolate the relevant subgraph
2.4 Explaining the circuit components
3 Automating circuit discovery (Step 3)
4 Evaluating Subgraph Recovery Algorithms
4.1 Grounded in previous work: area under ROC curves
4.2 Stand-alone circuit properties with a test metric
5 Related work
6 Conclusion
7 Acknowledgements
References
A Table of contents
B Impact statement
C Discussion of metrics optimized
C.1 Changing the metric in ACDC
C.2 Limitations of logit difference
C.3 Alternatives to minimizing a metric
D Details of Subnetwork Probing and Head Importance Score for Pruning
D.1 Subnetwork Probing
D.2 Head Importance Score for Pruning
E Experimental study of algorithm design
E.1 Minimizing the task-specific metric, rather than the KL divergence
E.2 Activation patching with zeros, instead of corrupted input
E.3 Node-level ROC curve, rather than edge-level ROC curve
F IOI task: details and qualitative evidence
F.1 Further details on the IOI experiments
F.2 The IOI circuit
F.3 Limitations of ACDC in recovering the IOI circuit
G Greater-Than task: details and qualitative evidence
H Docstring task: details and qualitative evidence
H.1 The docstring circuit
H.2 Additional docstring experiments
I Tracr tasks: details and qualitative evidence
I.1 tracr-xproportion
I.2 tracr-reverse
J Induction task: details and qualitative evidence
K Gendered pronoun completion: qualitative evidence
L Reset Network Experiments
M Automated Circuit Discovery and OR gates
N Connection to Causal Scrubbing

Knowls

Knowl 1 — Automatic Circuit DisCovery (ACDC) Algorithm
algorithm
The Automatic Circuit DisCovery (ACDC) algorithm automates the identification of functional subgraphs (circuits) within a neural network computational graph $G = (V, E)$ . Given a dataset of clean inputs $(x_i)_{i=1}^n$ that elicit a specific behavior, a paired dataset of corrupted inputs $(x'_i)_{i=1}^n$ that suppress the behavior, and an edge-pruning threshold $au > 0$ , ACDC iteratively searches for a sparse subnetwork $H \subseteq G$ that retains the original model's behavior.

The algorithm operates by traversing the nodes of $G$ in reverse topological order (from the output towards the inputs). For each node $v$ , it evaluates its incoming edges $(w \to v)$ . To determine if an edge $w \to v$ is necessary, ACDC temporarily removes it from $H$ , meaning the activation transmitted along this edge during the forward pass on $x_i$ is replaced by the corresponding activation computed on the corrupted input $x'_i$ . If the degradation in model behavior—measured by the change in average Kullback-Leibler (KL) divergence between the full model $G$ and the ablated subgraph $H$ —is less than $au$ , the edge is deemed unimportant and permanently removed from $H$ . Otherwise, the edge is retained.
Input: Computational graph $G = (V, E)$ , clean dataset $(x_i)_{i=1}^n$ , corrupted dataset $(x'_i)_{i=1}^n$ , threshold $au > 0$
Output: Discovered subnetwork $H \subseteq G$
$H \leftarrow G$
$H \leftarrow \text{reverse_topological_sort}(H)$
for each node $v \in H$ do
    for each parent node $w$ of $v$ in $G$ do
         $H_{\text{new}} \leftarrow H \setminus \{w \to v\}$
        if $D_{\text{KL}}(G \parallel H_{\text{new}}) - D_{\text{KL}}(G \parallel H) < \tau$ then
             $H \leftarrow H_{\text{new}}$
        end if
    end for
end for
return $H$
The order in which incoming parent edges $w$ are iterated is lexicographical: from later-layer to earlier-layer components, and from higher- to lower-indexed attention heads.
Knowl 2 — Three-Step Workflow for Mechanistic Interpretability
model/method
Mechanistic interpretability aims to explain neural network behaviors by reverse-engineering internal computational subgraphs (circuits) into human-understandable algorithms. This process is structured into a standard three-step workflow:
1. Select behavior, dataset, and metric: A specific, well-isolated behavior of the model is selected. Researchers construct a clean dataset of prompts $(x_i)_{i=1}^n$ that elicit the behavior, a corrupted dataset $(x'_i)_{i=1}^n$ where the relevant information or context is perturbed, and an evaluation metric (such as logit difference or Kullback-Leibler divergence) quantifying task performance.
2. Divide the neural network into a computational graph: The network's internals are formalized as a directed acyclic graph (DAG) of abstract computational units. The granularity of nodes is chosen based on the desired level of detail, ranging from full attention heads and multilayer perceptron (MLP) blocks to individual query ( $Q$ ), key ( $K$ ), value ( $V$ ) projections, distinct token positions, or individual neurons. Edges represent direct additive contributions to the residual stream across both adjacent and non-adjacent layers.
3. Isolate the subgraph via activation patching: Iterative activation patching experiments are conducted by replacing clean intermediate activations with corrupted (or zero) activations to prune non-essential nodes and edges, leaving a minimal faithful subgraph.
Once isolated, researchers formulate and validate mechanistic hypotheses regarding the human-interpretable role played by each component in the resulting circuit.
Knowl 3 — Quantitative Subgraph Evaluation Metric via Kullback-Leibler Divergence
equation
Given a neural network modeled as a computational graph $G$ , a candidate circuit subgraph $H \subseteq G$ , a clean input $x_i$ , and a corrupted input $x'_i$ , the ablated subgraph computation $H(x_i, x'_i)$ is defined as the forward pass on $x_i$ where all edge activations for edges in $G \setminus H$ are replaced with their activations computed on $x'_i$ .

The difference in behavior between the full model $G$ and the ablated subgraph $H$ over a dataset of $n$ prompts is quantified using the dataset-averaged Kullback-Leibler (KL) divergence:

$D_{\text{KL}}(G \parallel H) = \frac{1}{n} \sum_{i=1}^n D_{\text{KL}}(G(x_i) \parallel H(x_i, x'_i)) = \frac{1}{n} \sum_{i=1}^n \sum_{y \in \mathcal{Y}} G(x_i)(y) \log\left(\frac{G(x_i)(y)}{H(x_i, x'_i)(y)}\right)$

where $\mathcal{Y}$ denotes the model vocabulary and $G(x_i)(y)$ is the probability assigned to token $y$ by the unperturbed model.

The edge removal condition in circuit discovery evaluates whether:

$D_{\text{KL}}(G \parallel H_{\text{new}}) - D_{\text{KL}}(G \parallel H) < \tau$

where $\tau > 0$ is a user-defined threshold and $H_{\text{new}} = H \setminus \{w \to v\}$ . Minimizing KL divergence is preferred over task-specific metrics (such as logit difference) because $D_{\text{KL}} \ge 0$ , it prevents over-optimization where a pruned subnetwork produces larger logit differences than the original model, and it applies generally to next-token prediction without requiring prespecified target output labels.

Knowl 4 — Empirical Benchmark Evaluation of Circuit Discovery Algorithms

data/table

The performance of Automated Circuit DisCovery (ACDC), Head Importance Score for Pruning (HISP), and Subnetwork Probing (SP) was evaluated by comparing automatically extracted circuits against human-curated ground-truth circuits across five benchmark tasks: Indirect Object Identification (IOI), Greater-Than, Docstring, Tracr-Proportion, and Tracr-Reverse. Performance is reported as the Area Under the Receiver Operating Characteristic Curve (AUC) for edge-level classification (E) and node-level classification (N) under two ablation regimes: random/corrupted activations (interchange intervention) and zero activations.

Metric	Task	ACDC(E)	HISP(E)	SP(E)	ACDC(N)	HISP(N)	SP(N)
Corrupted Activations (Random Ablation)
KL	Docstring	0.982	0.805	0.937	0.950	0.881	0.928
	Greaterthan	0.853	0.693	0.806	0.890	0.642	0.827
	IOI	0.869	0.789	0.823	0.880	0.668	0.842
Loss	Docstring	0.972	0.821	0.942	0.938	0.889	0.941
	Greaterthan	0.461	0.706	0.812	0.766	0.631	0.811
	IOI	0.589	0.836	0.707	0.777	0.728	0.797
	Tracr-Proportion	0.679	0.679	0.525	0.750	0.909	0.818
	Tracr-Reverse	0.200	0.577	0.193	0.312	0.750	0.375
Zero Activations (Zero Ablation)
KL	Docstring	0.906	0.805	0.428	0.837	0.881	0.420
	Greaterthan	0.701	0.693	0.163	0.887	0.642	0.134
	IOI	0.539	0.792	0.486	0.458	0.671	0.605
Loss	Docstring	0.929	0.821	0.482	0.825	0.889	0.398
	Greaterthan	0.491	0.706	0.639	0.783	0.631	0.522
	IOI	0.447	0.836	0.393	0.424	0.728	0.479
	Tracr-Proportion	1.000	0.679	0.829	1.000	0.909	1.000
	Tracr-Reverse	1.000	0.577	0.801	1.000	0.750	1.000

When optimizing KL divergence with corrupted activations, ACDC consistently achieves the highest edge-level AUC on language tasks (0.982 on Docstring, 0.853 on Greater-Than, and 0.869 on IOI). Under zero ablation with the task loss, ACDC achieves perfect recovery (AUC = 1.000) on both compiled Tracr models. However, when optimizing task-specific losses directly on language tasks, ACDC performance degrades significantly (e.g., AUC dropping to 0.461 on Greater-Than and 0.589 on IOI), demonstrating that ACDC is sensitive to metric specification.

Knowl 5 — Adaptations of Subnetwork Probing and HISP for Circuit Discovery
model/method
To benchmark Automated Circuit DisCovery (ACDC), two gradient-based model pruning methods—Subnetwork Probing (SP) and Head Importance Score for Pruning (HISP)—are generalized for mechanistic circuit discovery:
1. Subnetwork Probing (SP): Originally designed to find subnetworks that preserve linear probe accuracy, SP is modified by: (i) omitting the downstream linear probe, (ii) setting the training objective to minimize the KL divergence $D_{\text{KL}}(G \parallel H)$ between the full model $G$ and masked subgraph $H$ with $L_0$ regularization (parameter $\lambda$ ), and (iii) modifying the continuous mask $Z$ to linearly interpolate between clean activations $C(x_i)$ (when $Z=1$ ) and corrupted activations $C(x'_i)$ (when $Z=0$ ). After training, the mask is thresholded to binary values, and edges between unmasked components define the circuit.
2. Head Importance Score for Pruning (HISP): Originally ranking only multi-head attention outputs under zero ablation, HISP is generalized to all differentiable internal components $C$ (including query, key, value heads, and MLP layer outputs) under corrupted activations. Component importance scores $I_C$ are computed as:
$I_C = \frac{1}{n} \sum_{i=1}^n \left| (C(x_i) - C(x'_i))^T \frac{\partial F(x_i)}{\partial C(x_i)} \right|$

where $F(x_i)$ is the objective metric (e.g., KL divergence or task loss). For zero ablation, the term $-C(x'_i)$ is omitted. Pruning all but the top $k$ highest-scoring components yields subgraphs across varying sparsity levels $k$ .
Knowl 6 — Suite of Canonical Benchmark Tasks for Transformer Circuit Discovery
experimental setup
Automated circuit discovery algorithms are evaluated across six benchmark language and synthetic algorithmic tasks where ground-truth mechanisms have been established:
1. Indirect Object Identification (IOI): In GPT-2 Small, prompts like "When John and Mary went to the store, Mary gave a bottle of milk to" require predicting " John". Evaluated using logit difference between target and non-target names over $N=50$ examples. Corrupted inputs use random alternative names (ABC template). The canonical circuit contains 26 attention heads (including Previous Token, Duplicate Token, Induction, S-Inhibition, and Name Mover heads) comprising 1,041 QKV/MLP edges.
2. Greater-Than: In GPT-2 Small, prompts like "The war lasted from 1517 to 15" require predicting two-digit years $>17$ . Evaluated via probability difference on $N=100$ examples. Corrupted inputs use year-neutral contexts. The canonical circuit consists of 262 edges.
3. Docstring: In a 4-layer attention-only model, prompts provide Python function docstrings requiring the correct parameter name completion. Evaluated via logit difference on $N=50$ examples. The canonical circuit consists of 37 edges across 8 attention heads.
4. Tracr-xproportion: A compiled transformer computing the cumulative fraction of occurrences of token 'x'. Metric is $L_2$ distance, with corrupted inputs being derangements (permutations without fixed points). Canonical circuit: 10 edges.
5. Tracr-reverse: A 3-layer compiled transformer reversing input sequences. Evaluated via $L_2$ distance on one-hot outputs, using deranged input sequences as corruptions. Canonical circuit: 15 edges.
6. Induction: A 2-layer, 8-head attention-only transformer evaluated on 40 sequences of 300 tokens from OpenWebText containing repeated sub-sequences $[A, B, \dots, A, B]$ , measuring KL divergence on the prediction of the second $B$ token.
Knowl 7 — Systematic Omission of Negative Components in Metric-Based Circuit Pruning
limitation
Circuit discovery methods that optimize a single scalar objective (such as task loss, logit difference, or dataset-averaged KL divergence) systematically fail to recover "negative" components in neural networks. Negative components are internal attention heads or MLPs whose direct causal effect reduces the model's confidence in the correct prediction (such as the Negative Name Mover heads 10.7 and 11.10 in GPT-2 Small's Indirect Object Identification circuit).

When optimizing logit difference or probability difference, removing a negative head increases the performance metric. Consequently, greedy pruning algorithms (such as ACDC) classify the edge as unimportant and permanently ablate it. When optimizing KL divergence, negative heads are only retained at extremely small threshold values $\tau$ (e.g., $\tau = 0.00398$ ), but at these thresholds, the algorithm also includes numerous false-positive, extraneous heads (such as heads 9.2 and 11.11), significantly lowering precision.
Knowl 8 — Inability of Edge and Node Pruning Methods to Recover Disjunctive Logic (OR Gates)
limitation
Greedy edge-pruning algorithms like ACDC, as well as gradient-based pruning methods (HISP and Subnetwork Probing), fail to recover both inputs to disjunctive logic gates (OR gates) embedded in neural networks. In a neural network, a logical OR function over binary inputs $x, y \in \{0, 1\}$ can be implemented as $1 - \text{ReLU}(1 - x - y)$ .

When both inputs are active ( $x=1, y=1$ ), the output is 1. When ACDC evaluates the first incoming edge representing input $x$ , setting $x=0$ leaves $y=1$ , meaning the gate still outputs $1 - \text{ReLU}(0) = 1$ . Because the model output does not change, ACDC judges the edge for $x$ as completely redundant and removes it. It then evaluates the second edge for $y$ , finds that setting $y=0$ (with $x$ already pruned to 0) breaks the output ( $1 - \text{ReLU}(1) = 0$ ), and retains only $y$ . As a result, ACDC only discovers one branch of the OR gate.

Similarly, gradient-based methods like HISP fail to detect the attention heads feeding the OR gate because the saturating non-linearity results in vanishing local gradients at the clean operating point.
Knowl 9 — Reset Network Methodology for Testing Circuit Discovery Hallucinations
experimental setup
To test whether circuit discovery methods identify true computational mechanisms rather than hallucinating functional circuits from overparameterization, candidate algorithms are evaluated on "reset networks". A reset network is constructed by randomly permuting the head dimensions of each layer's query ( $Q$ ), key ( $K$ ), and value ( $V$ ) weight matrices, as well as the MLP bias vectors.

This permutation destroys the trained algorithmic logic of the model while preserving broad statistical activation characteristics (such as the mean activation magnitude per layer). Because reset networks cannot perform the task, an ideal circuit discovery algorithm should fail to find any compact, performant subgraph. If an algorithm successfully extracts a sparse circuit with low KL divergence on a reset network (as observed with zero-ablated HISP and Subnetwork Probing), it provides strong evidence that the method is susceptible to finding spurious subgraphs (false positives).
Knowl 10 — Automated Identification of the Gendered Pronoun Prediction Circuit in GPT-2 Small
empirical result
Applying ACDC (with threshold $\tau = 0.05$ ) to GPT-2 Small on a gendered pronoun completion task (e.g., "So Dave is a really great friend, isn't" $\to$ " he") uncovered an end-to-end circuit exhibiting a three-stage "summarization motif" where MLPs play a dominant role compared to attention heads:
1. Early Name Encoding: Layer 0 attention head 0.4 and MLP0 process the subject name token embedding, transmitting gender information through intermediate MLPs (notably MLP3 and MLP5 at the name position) via key- and value-composition.
2. Information Routing to Punctuation/Verb: Attention heads 4.3 and 6.0 attend back to the name token position to compose with head 0.4 and MLP0, routing the representation to MLP7 located at the " is" token position, which serves as a central information hub with the largest number of incoming edges.
3. Late-Layer Readout: Attention heads 9.7 and 10.9 attend to the " is" token position, composing with MLP7 to feed into late-layer MLPs 10 and 11, which exert the strongest direct effect on the output logits predicting the gendered pronoun.

Coverage note — None omitted; all primary algorithms, interpretability workflows, baseline adaptations, benchmark evaluations, validation methods, and stated limitations are fully represented.

References

1.Alishahi, Afra, Grzegorz Chrupała, and Tal Linzen (2019). “Analyzing and interpreting neural networks for NLP: A report on the first BlackboxNLP workshop”. In: Natural Language Engineering 25.4, pp. 543–557. DOI: 10.1017/S135132491900024X.
2.Bau, David, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba (2020). Rewriting a Deep Generative Model. URL: https://arxiv.org/abs/2007.15646.
3.Bills, Steven, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders (2023). Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html.
4.Brundage, Miles, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, et al. (2018). “The malicious use of artificial intelligence: Forecasting, prevention, and mitigation”. In: arXiv preprint arXiv:1802.07228.
5.Buhrmester, Vanessa, David Münch, and Michael Arens (2021). “Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey”. In: 3.4, pp. 966–989. ISSN: 2504-4990. DOI: 10.3390/make3040048. URL: https://www.mdpi.com/2504-4990/3/4/48.
6.Cammarata, Nick, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah (2021). “Curve Circuits”. In: Distill. https://distill.pub/2020/circuits/curve-circuits. DOI: 10 . 23915 / distill.00024.006.
7.Cao, Steven, Victor Sanh, and Alexander Rush (2021). “Low-Complexity Probing via Finding Subnetworks”. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Online: Association for Computational Linguistics, pp. 960–966. DOI: 10.18653/v1/2021.naacl-main.74. URL: https://aclanthology.org/2021.naacl-main.74.
8.Chan, Lawrence, Adria Garriga-Alonso, Nix Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas (2022). Causal scrubbing: A method for rigorously testing interpretability hypotheses. Alignment Forum. URL: https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-forrigorously-testing.
9.Chughtai, Bilal, Lawrence Chan, and Neel Nanda (2023). A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations. URL: https://arxiv.org/abs/2302.03025.
10.Cuadros, Xavier Suau, Luca Zappella, and Nicholas Apostoloff (2022). “Self-conditioning pre-trained language models”. In: International Conference on Machine Learning. PMLR, pp. 4455–4473.
11.Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova (2019). “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, pp. 4171–4186. DOI: 10.18653/v1/N19-1423. URL: https://aclanthology.org/N19-1423.
12.Elhage, Nelson, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. (2022). “Toy Models of Superposition”. In: arXiv preprint arXiv:2209.10652.
13.Elhage, Nelson, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah (2021). “A Mathematical Framework for Transformer Circuits”. In: Transformer Circuits Thread. URL: https://transformer-circuits.pub/2021/framework/index.html.
14.Fawcett, Tom (2006). “An introduction to ROC analysis”. In: Pattern Recognition Letters 27.8. ROC Analysis in Pattern Recognition, pp. 861–874. ISSN: 0167-8655. DOI: https://doi.org/10.1016/j.patrec.2005.10.010. URL: https://www.sciencedirect.com/science/article/pii/S016786550500303X.
15.Feder, Amir, Nadav Oved, Uri Shalit, and Roi Reichart (2021). “CausaLM: Causal Model Explanation Through Counterfactual Language Models”. In: Computational Linguistics 47.2, pp. 333–386. DOI: 10.1162/coli_a_00404. URL: https://aclanthology.org/2021.cl-2.13.
16.Fong, Ruth C. and Andrea Vedaldi (2017). “Interpretable Explanations of Black Boxes by Meaningful Perturbation”. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, pp. 3449–3457. DOI: 10.1109/ICCV.2017.371. URL: https://doi.org/10.1109/ICCV.2017.371.
17.Frantar, Elias and Dan Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot.
18.Fu, Daniel Y, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re (2023). “Hungry Hungry Hippos: Towards Language Modeling with State Space Models”. In: The Eleventh International Conference on Learning Representations. URL: https://openreview.net/forum?id=COZDy0WYGg.
19.Geiger, Atticus, Hanson Lu, Thomas Icard, and Christopher Potts (2021). Causal Abstractions of Neural Networks. URL: https://arxiv.org/abs/2106.02997.
20.Geva, Mor, Roei Schuster, Jonathan Berant, and Omer Levy (2021). “Transformer Feed-Forward Layers Are Key-Value Memories”. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, pp. 5484–5495. DOI: 10.18653/v1/2021.emnlp-main.446. URL: https://aclanthology.org/2021.emnlp-main.446.
21.Gokaslan, Aaron, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex (2019). OpenWebText Corpus. URL: https://Skylion007.github.io/OpenWebTextCorpus.
22.Goldowsky-Dill, Nicholas, Chris MacLeod, Lucas Sato, and Aryaman Arora (2023). Localizing Model Behavior with Path Patching. arXiv: 2304.05969 [cs.LG].
23.Grover, Jasdeep Singh, Bhavesh Gawri, and Ruskin Raj Manku (2022). “DeepCuts: Single-Shot Interpretability based Pruning for BERT”. In.
24.Gurnee, Wes, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas (2023). Finding Neurons in a Haystack: Case Studies with Sparse Probing. arXiv: 2305.01610 [cs.LG].
25.Hanna, Michael, Ollie Liu, and Alexandre Variengien (2023). How does GPT-2 compute greaterthan?: Interpreting mathematical abilities in a pre-trained language model. arXiv: 2305.00586 [cs.CL].
26.Hassibi, Babak and David Stork (1992). “Second order derivatives for network pruning: Optimal brain surgeon”. In: 5.
27.Heimersheim, Stefan and Jett Janiak (2023). A circuit for Python docstrings in a 4-layer attention-only transformer. URL: https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/acircuit-for-python-docstrings-in-a-4-layer-attention-only.
28.Hendrycks, Dan and Mantas Mazeika (2022). X-Risk Analysis for AI Research. arXiv: 2206.05862 [cs.CY].
29.Hendrycks, Dan, Mantas Mazeika, and Thomas Woodside (2023). An Overview of Catastrophic AI Risks. arXiv: 2306.12001 [cs.CY].
30.Hernandez, Evan, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas (2022). Natural Language Descriptions of Deep Visual Features. arXiv: 2201.11114 [cs.CV].
31.Hubinger, Evan (2020). An overview of 11 proposals for building safe advanced AI. arXiv: 2012.07532 [cs.LG].
32.Jacovi, Alon and Yoav Goldberg (2020). “Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?” In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, pp. 4198–4205. DOI: 10.18653/v1/2020.acl- main.386. URL: https://aclanthology.org/2020.acl-main.386.
33.Jang, Eric, Shixiang Gu, and Ben Poole (2017). Categorical Reparameterization with GumbelSoftmax. arXiv: 1611.01144 [stat.ML].
34.Kaddour, Jean, Aengus Lynch, Qi Liu, Matt J. Kusner, and Ricardo Silva (2022). Causal Machine Learning: A Survey and Open Problems. URL: https://arxiv.org/abs/2206.15475.
35.Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei (2020). Scaling Laws for Neural Language Models. arXiv: 2001.08361 [cs.LG].
36.Kurtic, Eldar, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh (2022). “The optimal BERT surgeon: Scalable and accurate second-order pruning for large language models”. In.
37.LeCun, Yann, John Denker, and Sara Solla (1989). “Optimal brain damage”. In: 2.
38.Lindner, David, János Kramár, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik (2023). “Tracr: Compiled Transformers as a Laboratory for Interpretability”. In.
39.Lipton, Zachary C. (2016). The Mythos of Model Interpretability. URL: https://arxiv.org/abs/1606.03490.
40.Louizos, Christos, Max Welling, and Diederik P. Kingma (2018). “Learning Sparse Neural Networks through L_0 Regularization”. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. URL: https://openreview.net/forum?id=H1Y8hhg0b.
41.Mathwin, Chris, Guillaume Corlouer, Esben Kran, Fazl Barez, and Neel Nanda (2023). Identifying a Preliminary Circuit for Predicting Gendered Pronouns in GPT-2 Small. URL: https://itch.io/jam/mechint/rate/1889871.
42.McDougall, Callum, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda (2023). Copy Suppression: Comprehensively Understanding an Attention Head. arXiv: 2310.04625 [cs.LG].
43.Meng, Kevin, David Bau, Alex J Andonian, and Yonatan Belinkov (2022). “Locating and editing factual associations in GPT”. In: Advances in Neural Information Processing Systems.
44.Michel, Paul, Omer Levy, and Graham Neubig (2019). “Are Sixteen Heads Really Better than One?” In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. Ed. by Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, pp. 14014–14024. URL: https://proceedings.neurips.cc/paper/2019/hash/2c601ad9d2ff9bc8b282670cdd54f69f-Abstract.html.
45.Mu, Jesse and Jacob Andreas (2021). Compositional Explanations of Neurons. arXiv: 2006.14032 [cs.LG].
46.Nanda, Neel (2022). TransformerLens. URL: https : / / github . com / neelnanda - io / TransformerLens.
47.– (2023). Attribution Patching: Activation Patching At Industrial Scale. URL: https : / / www . neelnanda.io/mechanistic-interpretability/attribution-patching.
48.Nanda, Neel, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt (2023). “Progress measures for grokking via mechanistic interpretability”. In: The Eleventh International Conference on Learning Representations. URL: https://openreview.net/forum?id=9XFSbDPmdW.
49.Olah, Chris (2022). Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases. https://www.transformer-circuits.pub/2022/mech-interp-essay.
50.Olah, Chris, Alexander Mordvintsev, and Ludwig Schubert (2017). “Feature Visualization”. In: Distill. https://distill.pub/2017/feature-visualization. DOI: 10.23915/distill.00007.
51.Olsson, Catherine, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. (2022). In-context learning and induction heads. URL: https://transformer-circuits.pub/2022/in-context-learning-andinduction-heads/index.html.
52.OpenAI (2023). GPT-4 Technical Report. arXiv: 2303.08774 [cs.CL].
53.Pearl, Judea (2009). Causality. Models, Reasoning, and Inference. 2nd ed. Cambridge University Press. ISBN: 978-0-521-89560-6. DOI: 10.1017/CBO9780511803161.
54.Radford, Alec, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever (2019). “Language Models are Unsupervised Multitask Learners”. In.
55.Ramanujan, Vivek, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari (2020). “What’s Hidden in a Randomly Weighted Neural Network?” In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. IEEE, pp. 11890–11899. DOI: 10.1109/CVPR42600.2020.01191. URL: https://doi.org/10.1109/CVPR42600.2020.01191.
56.Räuker, Tilman, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell (2022). Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks. URL: https://arxiv.org/abs/2207.13243.
57.Sanh, Victor, Thomas Wolf, and Alexander M. Rush (2020). “Movement Pruning: Adaptive Sparsity by Fine-Tuning”. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Ed. by Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin. URL: https : / / proceedings . neurips . cc / paper / 2020 / hash / eae15aabaa768ae4a5993a8a4f4fa6e4-Abstract.html.
58.Syed, Aaquib, Can Rager, and Arthur Conmy (2023). Attribution Patching Outperforms Automated Circuit Discovery. arXiv: 2310.10348 [cs.LG].
59.Tigges, Curt, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda (2023). Linear Representations of Sentiment in Large Language Models. arXiv: 2310.15154 [cs.LG].
60.Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin (2017). “Attention is All you Need”. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. Ed. by Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, pp. 5998–6008. URL: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
61.Veit, Andreas, Michael J. Wilber, and Serge J. Belongie (2016). “Residual Networks Behave Like Ensembles of Relatively Shallow Networks”. In: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain. Ed. by Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, pp. 550–558. URL: https://proceedings.neurips.cc/paper/2016/hash/37bc2f75bf1bcfe8450a1a41c200364c-Abstract.html.
62.Vig, Jesse, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber (2020). Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias. URL: https://arxiv.org/abs/2004.12265.
63.Wang, Kevin Ro, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt (2023). “Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small”. In: The Eleventh International Conference on Learning Representations. URL: https://openreview.net/forum?id=NpsVSN6o4ul.
64.Wang, Ziheng, Jeremy Wohlwend, and Tao Lei (2020). “Structured Pruning of Large Language Models”. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, pp. 6151–6162. DOI: 10.18653/v1/2020.emnlp-main.496. URL: https://aclanthology.org/2020.emnlpmain.496.
65.Wei, Jason, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus (2022). Emergent Abilities of Large Language Models. arXiv: 2206.07682 [cs.CL].
66.Wu, Zhengxuan, Atticus Geiger, Christopher Potts, and Noah D. Goodman (2023). Interpretability at Scale: Identifying Causal Mechanisms in Alpaca. arXiv: 2305.08809 [cs.CL].
67.Wu, Zhengxuan, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, and Noah Goodman (2022). “Causal Distillation for Language Models”. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle, United States: Association for Computational Linguistics, pp. 4288–4295. DOI: 10 . 18653 / v1 / 2022 . naacl - main . 318. URL: https ://aclanthology.org/2022.naacl-main.318.
68.Zhang, Kelly and Samuel Bowman (2018). “Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis”. In: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Brussels, Belgium: Association for Computational Linguistics, pp. 359–361. DOI: 10.18653/v1/W18-5448. URL: https://aclanthology.org/W18-5448.
69.Zhang, Yu, Peter Tino, Aleš Leonardis, and Ke Tang (2021). “A survey on neural network inter- ˇ pretability”. In: 5.5, pp. 726–742. DOI: 10.1109/TETCI.2021.3100641.
70.Zhu, Xunyu, Jian Li, Yong Liu, Can Ma, and Weiping Wang (2023). A Survey on Model Compression for Large Language Models. arXiv: 2308.07633 [cs.CL].

Citation

MLA

Conmy, A., et al. “Towards Automated Circuit Discovery for Mechanistic Interpretability”. arXiv, 2023, http://arxiv.org/abs/2304.14997v4.

APA

Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., & Garriga-Alonso, A. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. arXiv. http://arxiv.org/abs/2304.14997v4

Chicago

Conmy, A., A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. 2023. “Towards Automated Circuit Discovery for Mechanistic Interpretability”. arXiv. http://arxiv.org/abs/2304.14997v4.

Harvard

Conmy, A. et al. (2023) “Towards Automated Circuit Discovery for Mechanistic Interpretability”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2304.14997v4.

Vancouver

1. Conmy A, Mavor-Parker AN, Lynch A, Heimersheim S, Garriga-Alonso A (2023) Towards Automated Circuit Discovery for Mechanistic Interpretability. arXiv

BibTeX

@article{conmy2023towards,
  title = {Towards Automated Circuit Discovery for Mechanistic Interpretability},
  author = {Conmy, Arthur and Mavor-Parker, Augustine N. and Lynch, Aengus and Heimersheim, Stefan and Garriga-Alonso, Adrià},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2304.14997v4},
  eprint = {2304.14997}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Towards Automated Circuit Discovery for Mechanistic Interpretability

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Automatic Circuit DisCovery (ACDC) Algorithm

Knowl 2 — Three-Step Workflow for Mechanistic Interpretability

Knowl 3 — Quantitative Subgraph Evaluation Metric via Kullback-Leibler Divergence

Knowl 4 — Empirical Benchmark Evaluation of Circuit Discovery Algorithms

Knowl 5 — Adaptations of Subnetwork Probing and HISP for Circuit Discovery

Knowl 6 — Suite of Canonical Benchmark Tasks for Transformer Circuit Discovery

Knowl 7 — Systematic Omission of Negative Components in Metric-Based Circuit Pruning

Knowl 8 — Inability of Edge and Node Pruning Methods to Recover Disjunctive Logic (OR Gates)

Knowl 9 — Reset Network Methodology for Testing Circuit Discovery Hallucinations

Knowl 10 — Automated Identification of the Gendered Pronoun Prediction Circuit in GPT-2 Small

References

Citation

BibTeX

Source Code

Access the Paper