Locating and Editing Factual Associations in GPT

Kevin Meng$^{*}$
MIT CSAIL

David Bau$^{*}$
Northeastern University

Alex Andonian
MIT CSAIL

Yonatan Belinkov$^{\dagger}$
Technion – IIT

$^{*}$ Equal contribution. Correspondence to [email protected], [email protected].
$^{\dagger}$ Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion.

Abstract

We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task. We also evaluate ROME on a new dataset of difficult counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available at https://rome.baulab.info/.

Executive Summary: The research paper examines how autoregressive transformer language models such as GPT store and retrieve factual associations, addressing the practical challenge of correcting or updating specific facts in deployed models without broad retraining. Accurate localization of knowledge is important because current editing techniques often produce either superficial changes that fail to generalize or unintended side effects that degrade performance on related content.

The work set out to identify the internal computations responsible for factual recall and to test whether those computations could be directly modified to insert new associations while preserving model behavior elsewhere.

The authors first applied causal mediation analysis to GPT-2 XL and larger variants, running controlled interventions on hidden states across thousands of factual prompts to measure each component’s contribution to correct predictions. They then introduced Rank-One Model Editing (ROME), a closed-form method that treats a chosen mid-layer feed-forward module as a linear associative memory and inserts a new key-value pair with a single rank-one weight update. Performance was measured on the standard zsRE benchmark and on a new COUNTERFACT dataset of 21,919 difficult counterfactual statements, with metrics for efficacy, paraphrase generalization, neighborhood specificity, fluency, and semantic consistency.

The analysis shows that factual associations are mediated primarily by middle-layer MLP modules processing the final subject token; restoring clean activations at these sites recovers correct predictions far more effectively than attention modules or other positions. ROME achieves near-perfect efficacy and strong generalization on both benchmarks while maintaining high specificity on unrelated facts, outperforming fine-tuning, constrained fine-tuning, and hypernetwork baselines that typically trade off one property for another. Human raters confirm that ROME-generated text aligns more consistently with the inserted fact than the next-best baseline, although minor fluency reductions appear in some cases.

These results indicate that factual knowledge in GPT-style models resides in localized, editable computations rather than being diffusely distributed, which opens a route to precise, low-cost model updates. For practical use, the approach could support targeted corrections in production systems where full retraining is prohibitive, provided edits remain limited in number.

Further work is needed to extend the method to simultaneous multi-fact edits and to other forms of knowledge such as logical or numerical relations. The main limitations are that ROME currently handles one association at a time, that edited models can still generate plausible but unsupported statements, and that the study examined only autoregressive transformers of moderate size. Confidence in the core localization and editing findings is high because they are corroborated across multiple model scales, corruption schemes, and evaluation protocols.

1. Introduction

Section Summary: This paper investigates how GPT-like language models store factual knowledge, presenting evidence that such associations are handled through localized computations in specific neural network components that can be directly modified. Using causal analysis of internal activations, the authors identify middle-layer feedforward modules as critical for recalling facts about a subject when processing its final token. They then introduce a simple parameter-editing technique called ROME that alters these modules, demonstrating strong performance on both standard benchmarks and new counterfactual tests by achieving better generalization and specificity than prior fine-tuning or meta-learning approaches.

Where does a large language model store its facts? In this paper, we report evidence that factual associations in GPT correspond to a localized computation that can be directly edited.

Large language models can predict factual statements about the world ([1, 2, 3]). For example, given the prefix "The Space Needle is located in the city of, " GPT will reliably predict the true answer: "Seattle" (Figure 1 a). Factual knowledge has been observed to emerge in both autoregressive GPT models ([4, 5]) and masked BERT models ([6]).

In this paper, we investigate how such factual associations are stored within GPT-like autoregressive transformer models. Although many of the largest neural networks in use today are autoregressive, the way that they store knowledge remains under-explored. Some research has been done for masked models ([1, 2, 7, 8, 9, 10]), but GPT has architectural differences such as unidirectional attention and generation capabilities that provide an opportunity for new insights.

We use two approaches. First, we trace the causal effects of hidden state activations within GPT using causal mediation analysis ([11, 12]) to identify the specific modules that mediate recall of a fact about a subject (Figure 1). Our analysis reveals that feedforward MLPs at a range of middle layers are decisive when processing the last token of the subject name (Figure 1 b, Figure 2 b, Figure 3).

Second, we test this finding in model weights by introducing a Rank-One Model Editing method (ROME) to alter the parameters that determine a feedfoward layer's behavior at the decisive token. Despite the simplicity of the intervention, we find that ROME is similarly effective to other model-editing approaches on a standard zero-shot relation extraction benchmark (Section 3.2).

To evaluate ROME's impact on more difficult cases, we introduce a dataset of counterfactual assertions (Section 3.3) that would not have been observed in pretraining. Our evaluations (Section 3.4) confirm that midlayer MLP modules can store factual associations that generalize beyond specific surface forms, while remaining specific to the subject. Compared to previous fine-tuning ([13]), interpretability-based ([9]), and meta-learning ([14, 10]) methods, ROME achieves good generalization and specificity simultaneously, whereas previous approaches sacrifice one or the other.

2. Interventions on Activations for Tracing Information Flow

Section Summary: Researchers trace how facts are stored inside a transformer by measuring the causal impact of its internal hidden states on next-word predictions. They do this by running the model three times on a factual prompt—once normally, once with noise added to corrupt the subject, and once with selected clean states restored at specific layers and tokens—to see which states can recover the correct answer despite the corruption. This approach reveals that certain middle-layer states at the subject’s final token and late-layer states at the last token play especially strong roles in recalling the fact.

To locate facts within the parameters of a large pretrained autoregressive transformer, we begin by analyzing and identifying the specific hidden states that have the strongest causal effect on predictions of individual facts. We represent each fact as a knowledge tuple $t = \left( s, r, o \right)$ containing the subject $s$, object $o$, and relation $r$ connecting the two. Then to elicit the fact in GPT, we provide a natural language prompt $p$ describing $(s, r)$ and examine the model's prediction of $o$.

An autoregressive transformer language model $G: \mathcal{X} \rightarrow \mathcal{Y}$ over vocabulary $V$ maps a token sequence $x = [x_1, ..., x_T] \in \mathcal{X}$, $x_i \in V$ to a probability distribution $\smash{y \in \mathcal{Y} \subset \mathbb{R}^{|V|}}$ that predicts next-token continuations of $x$. Within the transformer, the $i$th token is embedded as a series of hidden state vectors $\smash{h^{(l)}_i}$, beginning with $\smash{h^{(0)}i = \mathrm{emb}(x_i) + \mathrm{pos}(i) \in \mathbb{R}^{H}}$. The final output $\smash{y = \mathrm{decode}(h^{(L)}{T})}$ is read from the last hidden state.

We visualize the internal computation of $G$ as a grid (Figure 1 a) of hidden states $\smash{h^{(l)}_i}$ in which each layer $l$ (left $\rightarrow$ right) adds global attention $\smash{a^{(l)}_i}$ and local MLP $\smash{m^{(l)}_i}$ contributions computed from previous layers, and where each token $i$ (top $\rightarrow$ bottom) attends to previous states from other tokens. Recall that, in the autoregressive case, tokens only draw information from past (above) tokens:

$ \begin{aligned} \notag h^{(l)}_i = h^{(l-1)}_i + a^{(l)}_i &+ m^{(l)}_i \ a^{(l)}_i &= \mathrm{attn}^{(l)}\left(h^{(l-1)}_1, h^{(l-1)}_2, \dots, h^{(l-1)}i\right) \ \notag m^{(l)}i &= W{proj}^{(l)}, \sigma\left( W{fc}^{(l)} \gamma\left( a^{(l)}_i + h^{(l-1)}_i \right) \right). \end{aligned} $

Each layer's MLP is a two-layer neural network parameterized by matrices $\smash{W_{proj}^{(l)}}$ and $\smash{W_{fc}^{(l)}}$, with rectifying nonlinearity $\sigma$ and normalizing nonlinearity $\gamma$. For further background on transformers, we refer to [15].[^4]

[^4]: Equation 1 calculates attention sequentially after the MLP module as in [5]. Our methods also apply to GPT variants such as [16] that put attention in parallel to the MLP.

2.1 Causal Tracing of Factual Associations

The grid of states (Figure 1) forms a causal graph ([17]) describing dependencies between the hidden variables. This graph contains many paths from inputs on the left to the output (next-word prediction) at the lower-right, and we wish to understand if there are specific hidden state variables that are more important than others when recalling a fact.

As [12] have shown, this is a natural case for causal mediation analysis, which quantifies the contribution of intermediate variables in causal graphs ([11]). To calculate each state's contribution towards a correct factual prediction, we observe all of $G

• In the clean run, we pass a factual prompt $x$ into $G$ and collect all hidden activations $\left{ \smash{h^{(l)}_i} \mid i \in[1,T], l\in[1,L] \right}$. Figure 1 a provides an example illustration with the prompt: "The Space Needle is in downtown $\rule{1cm}{0.15mm}$ ", for which the expected completion is $o =$ "Seattle".
• In the baseline corrupted run, the subject is obfuscated from $G$ before the network runs. Concretely, immediately after $x$ is embedded as $\left[ \smash{h^{(0)}{1}}, \smash{h^{(0)}{2}}, \dots, \smash{h^{(0)}{T}} \right]$, we set $\smash{h^{(0)}{i} := h^{(0)}{i}} + \epsilon$ for all indices $i$ that correspond to the subject entity, where $\epsilon \sim \mathcal{N}(0;\nu)$[^1]; . $G$ is then allowed to continue normally, giving us a set of corrupted activations $\left{ \smash{h^{(l)}{i*}} \mid i \in[1,T], l\in[1,L] \right}$. Because $G$ loses some information about the subject, it will likely return an incorrect answer (Figure 1 b).
• The corrupted-with-restoration run, lets $G$ run computations on the noisy embeddings as in the corrupted baseline, except at some token $\smash{\hat{i}}$ and layer $\smash{\hat{l}}$. There, we hook $G$ so that it is forced to output the clean state $\smash{h^{(\smash{\hat{l}})}_{\hat{i}}}$; future computations execute without further intervention. Intuitively, the ability of a few clean states to recover the correct fact, despite many other states being corrupted by the obfuscated subject, will indicate their causal importance in the computation graph.

[^1]: We select $\nu$ to be $3$ times larger than the empirical standard deviation of embeddings; see Appendix B.1 for details, and see Appendix B.4 for an analysis of other corruption rules.

Let $\mathbb{P}[o]$, $\mathbb{P}*[o]$, and $\mathbb{P}{,\smash{\text{ clean } h_i^{(l)}}}[o]$ denote the probability of emitting $o$ under the clean, corrupted, and corrupted-with-restoration runs, respectively; dependence on the input $x$ is omitted for notational simplicity. The total effect (TE) is the difference between these quantities: $\text{TE} = \mathbb{P}[o] - \mathbb{P}_[o]$. The indirect effect (IE) of a specific mediating state $\smash{h_i^{(l)}}$ is defined as the difference between the probability of $o$ under the corrupted version and the probability when that state is set to its clean version, while the subject remains corrupted: $\text{IE} = \mathbb{P}{*,\smash{\text{ clean } h_i^{(l)}}}[o] - \mathbb{P}*[o]$. Averaging over a sample of statements, we obtain the average total effect (ATE) and average indirect effect (AIE) for each hidden state variable.[^5]

[^5]: One could also compute the direct effect, which flows through other model components besides the chosen mediator. However, we found this effect to be noisy and uninformative, in line with results by [12].

2.2 Causal Tracing Results

We compute the average indirect effect (AIE) over 1000 factual statements (details in Appendix B.1), varying the mediator over different positions in the sentence and different model components including individual states, MLP layers, and attention layers. Figure 2 plots the AIE of the internal components of GPT-2 XL (1.5B parameters). The ATE of this experiment is 18.6%, and we note that a large portion of the effect is mediated by strongly causal individual states (AIE=8.7% at layer 15) at the last subject token. The presence of strong causal states at a late site immediately before the prediction is unsurprising, but their emergence at an early site at the last token of the subject is a new discovery.

Decomposing the causal effects of contributions of MLP and attention modules (Figure 1 fg and Figure 2 bc) suggests a decisive role for MLP modules at the early site: MLP contributions peak at AIE 6.6%, while attention at the last subject token is only AIE 1.6%; attention is more important at the last token of the prompt. Appendix B.2 further discusses this decomposition.

Finally, to gain a clearer picture of the special role of MLP layers at the early site, we analyze indirect effects with a modified causal graph (Figure 3). (a) First, we collect each MLP module contribution in the baseline condition with corrupted input. (b) Then, to isolate the effects of MLP modules when measuring causal effects, we modify the computation graph to sever MLP computations at token $i$ and freeze them in the baseline corrupted state so that they are unaffected by the insertion of clean state for $\smash{h_i^{(l)}}$. This modification is a way of probing path-specific effects ([11]) for paths that avoid MLP computations. (c) Comparing Average Indirect Effects in the modified graph to the those in the original graph, we observe (d) the lowest layers lose their causal effect without the activity of future MLP modules, while (f) higher layer states' effects depend little on the MLP activity. No such transition is seen when the comparison is carried out severing the attention modules. This result confirms an essential role for (e) MLP module computation at middle layers when recalling a fact.

Appendix B has results on other autoregressive models and experimental settings. In particular, we find that Causal Tracing is more informative than gradient-based salience methods such as integrated gradients ([18]) (Figure 16) and is robust under different noise configurations.

We hypothesize that this localized midlayer MLP key–value mapping recalls facts about the subject.

2.3 The Localized Factual Association Hypothesis

Based on causal traces, we posit a specific mechanism for storage of factual associations: each midlayer MLP module accepts inputs that encode a subject, then produces outputs that recall memorized properties about that subject. Middle layer MLP outputs accumulate information, then the summed information is copied to the last token by attention at high layers.

This hypothesis localizes factual association along three dimensions, placing it (i) in the MLP modules (ii) at specific middle layers (iii) and specifically at the processing of the subject's last token. It is consistent with the [8] view that MLP layers store knowledge, and the [19] study showing an information-copying role for self-attention. Furthermore, informed by the [20] finding that transformer layer order can be exchanged with minimal change in behavior, we propose that this picture is complete. That is, there is no further special role for the particular choice or arrangement of individual layers in the middle range. We conjecture that any fact could be equivalently stored in any one of the middle MLP layers. To test our hypothesis, we narrow our attention to a single MLP module at a mid-range layer $l^*$, and ask whether its weights can be explicitly modified to store an arbitrary fact.

3. Interventions on Weights for Understanding Factual Association Storage

Section Summary: Researchers hypothesize that transformer MLP layers function like linear associative memories, storing factual associations as key-value pairs in their weights. To test this idea, they introduce Rank-One Model Editing (ROME), a method that identifies a subject-derived key vector and an optimized value vector for a new fact, then inserts the pair into a chosen MLP layer through a precise rank-one weight update. Successful edits that generalize appropriately while preserving unrelated knowledge would confirm that factual information is localized and directly editable in these specific modules.

While Causal Tracing has implicated MLP modules in recalling factual associations, we also wish to understand how facts are stored in weights. [8] observed that MLP layers (Figure 4 cde) can act as two-layer key–value memories, [^6] where the neurons of the first layer $\smash{W^{(l)}{fc}}$ form a key, with which the second layer $\smash{W^{(l)}{proj}}$ retrieves an associated value. We hypothesize that MLPs can be modeled as a linear associative memory; note that this differs from [8]'s per-neuron view.

[^6]: Unrelated to keys and values in self-attention.

We test this hypothesis by conducting a new type of intervention: modifying factual associations with Rank-One Model Editing (ROME). Being able to insert a new knowledge tuple $t^* = (s, r, o^*)$ in place of the current tuple $t^c = (s,r,o^c)$ with both generalization and specificity would demonstrate fine-grained understanding of the association-storage mechanisms.

3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory

We view $\smash{W^{(l)}{proj}}$ as a linear associative memory ([21, 22]). This perspective observes that any linear operation $W$ can operate as a key–value store for a set of vector keys $K = [ k_1 \mid k_2 \mid \dots ]$ and corresponding vector values $V = [v_1 \mid v_2 \mid \dots]$, by solving $WK \approx V$, whose squared error is minimized using the Moore-Penrose pseudoinverse: $W = VK^{+}$. [23] observed that a new key–value pair $(k, v_)$ can be inserted optimally into the memory by solving a constrained least-squares problem. In a convolutional network, [23] solve this using an optimization, but in a fully-connected layer, we can derive a closed form solution:

$ \begin{aligned} \text{minimize} ; \lVert \hat{W}K &- V \rVert ; \text{such that} ; \hat{W}k_* = v_* \quad \text{by setting} ; \hat{W} = W + \Lambda (C^{-1}k_*)^T. \end{aligned} $

Here $W$ is the original matrix, $C=KK^T$ is a constant that we pre-cache by estimating the uncentered covariance of $k$ from a sample of Wikipedia text (Appendix E.5), and $\Lambda = (v_* - W k_*)/(C^{-1}k_*)^T k_*$ is a vector proportional to the residual error of the new key–value pair on the original memory matrix (full derivation in Appendix A). Because of this simple algebraic structure, we can insert any fact directly once $(k_*, v_*)$ is computed. All that remains is to choose the appropriate $k_*$ and $v_*$.

Step 1: Choosing k* to Select the Subject. Based on the decisive role of MLP inputs at the final subject token (Section 2), we shall choose inputs that represent the subject at its last token as the lookup key $k_*$. Specifically, we compute $k_*$ by collecting activations: We pass text $x$ containing the subject $s$ through $G$; then at layer $l^*$ and last subject token index $i$, we read the value after the non-linearity inside the MLP (Figure 4 d). Because the state will vary depending on tokens that precede $s$ in text, we set $k_*$ to an average value over a small set of texts ending with the subject $s$:

$ k_* = \frac{1}{N} \sum_{j=1}^N k(x_j + s), ; \text{where} ; k(x) = \sigma\left( W_{fc}^{(l^*)} ; \gamma(a^{(l^*)}{[x],i} + h^{(l^*-1)}{[x],i}) \right). $

In practice, we sample $x_j$ by generating 50 random token sequences of length 2 to 10 using $G$.

Step 2: Choosing v* to Recall the Fact. Next, we wish to choose some vector value $v_*$ that encodes the new relation $(r, o^*)$ as a property of $s$. We set $v_* = \operatorname*{argmin}_z \mathcal{L}(z)$, where the objective $\mathcal{L}(z)$ is:

$ \frac{1}{N} \sum_{j=1}^N \underbrace{-\log\mathbb{P}{G(m^{(l^*)}{i}:=z)}\left[ o^* \mid x_j + p, \right]}\text{(a) Maximizing $o^*$ probability} ; + ; \underbrace{D{\mathrm{KL}}\left(\mathbb{P}{G(m^{(l^*)}{i^\prime}:=z)}\left[ x \mid p^\prime \right] \big\Vert \mathbb{P}{G}\left[ x \mid p^\prime \right] \right)}\text{(b) Controlling essence drift}. $

The first term Equation (4 a) seeks a vector $z$ that, when substituted as the output of the MLP at the token $i$ at the end of the subject (notated $\smash{G(m^{(l^*)}{i}:=z)}$), will cause the network to predict the target object $o^*$ in response to the factual prompt $p$. The second term Equation (4 b) minimizes the KL divergence of predictions for the prompt $p^\prime$ (of the form "subject is a") to the unchanged model, which helps preserve the model's understanding of the subject's essence. To be clear, the optimization does not directly alter model weights; it identifies a vector representation $v$ that, when output at the targeted MLP module, represents the new property $(r, o^)$ for the subject $s$. Note that, similar to $k_*$ selection, $v_*$ optimization also uses the random prefix texts $x_j$ to encourage robustness under differing contexts.

Step 3: Inserting the Fact. Once we have computed the pair ($k_*$, $v_*$) to represent the full fact $(s, r, o^*)$, we apply Equation 2, updating the MLP weights $\smash{W^{(l)}_{proj}}$ with a rank-one update that inserts the new key–value association directly. For full implementation details, see Appendix E.5.

3.2 Evaluating ROME: Zero-Shot Relation Extraction (zsRE)

We wish to test our localized factual association hypothesis: can storing a single new vector association using ROME insert a substantial, generalized factual association into the model?

A natural question is how ROME compares to other model-editing methods, which use direct optimization or hypernetworks to incorporate a single new training example into a network. For baselines, we examine Fine-Tuning (FT), which applies Adam with early stopping at one layer to minimize $-\log \mathbb{P}\left[ o^* \mid x \right]$. Constrained Fine-Tuning (FT+L) ([13]) additionally imposes a parameter-space $L_\infty$ norm constraint on weight changes. We also test two hypernetworks: Knowledge Editor (KE) ([10]) and MEND ([14]), both of which learn auxiliary models to predict weight changes to $G$. Further details are described in Appendix E.

::: {caption="Table 1: zsRE Editing Results on GPT-2 XL."}

:::

We first evaluate ROME on the Zero-Shot Relation Extraction (zsRE) task used in [14] and [10]. Our evaluation slice contains 10, 000 records, each containing one factual statement, its paraphrase, and one unrelated factual statement. "Efficacy" and "Paraphrase" measure post-edit accuracy $\smash{\mathbb{I}\big[o^* = \mathrm{argmax}o \mathbb{P}{G^\prime}\left[ o \right]\big]}$ of the statement and its paraphrase, respectively, while "Specificity" measures the edited model's accuracy on an unrelated fact. Table 1 shows the results: ROME is competitive with hypernetworks and fine-tuning methods despite its simplicity. We find that it is not hard for ROME to insert an association that can be regurgitated by the model. Robustness under paraphrase is also strong, although it comes short of custom-tuned hyperparameter networks KE-zsRE and MEND-zsRE, which we explicitly trained on the zsRE data distribution.[^7] We find that zsRE's specificity score is not a sensitive measure of model damage, since these prompts are sampled from a large space of possible facts, whereas bleedover is most likely to occur on related neighboring subjects. Appendix C has additional experimental details.

[^7]: Out-of-the-box, they are trained on a WikiText generation task ([14, 10]).

3.3 Evaluating ROME: Our CounterFact Dataset

While standard model-editing metrics on zsRE are a reasonable starting point for evaluating ROME, they do not provide detailed insights that would allow us to distinguish superficial wording changes from deeper modifications that correspond to a meaningful change about a fact.

In particular, we wish to measure the efficacy of significant changes. [24] observed that standard model-editing benchmarks underestimate difficulty by often testing only proposals that the model previously scored as likely. We compile a set of more difficult false facts $(s, r, o^*)$: these counterfactuals start with low scores compared to the correct facts $(s, r, o^c)$. Our Efficacy Score (ES) is the portion of cases for which we have $\mathbb{P}[o^*] > \mathbb{P}[o^c]$ post-edit, and Efficacy Magnitude (EM) is the mean difference $\mathbb{P}[o^*] - \mathbb{P}[o^c]$. Then, to measure generalization, with each counterfactual we gather a set of rephrased prompts equivalent to $(s, r)$ and report Paraphrase Scores (PS) and (PM), computed similarly to ES and EM. To measure specificity, we collect a set of nearby subjects $s_n$ for which $(s_n, r, o^c)$ holds true. Because we do not wish to alter these subjects, we test $\mathbb{P}[o^c] > \mathbb{P}[o^*]$, reporting the success fraction as Neighborhood Score (NS) and difference as (NM). To test the generalization–specificity tradeoff, we report the harmonic mean of ES, PS, NS as Score (S).

We also wish to measure semantic consistency of $G^\prime

In order to facilitate the above measurements, we introduce $\textsc{CounterFact}$, a challenging evaluation dataset for evaluating counterfactual edits in language models. Containing 21, 919 records with a diverse set of subjects, relations, and linguistic variations, $\textsc{CounterFact}$ 's goal is to differentiate robust storage of new facts from the superficial regurgitation of target words. See Appendix D for additional technical details about its construction, and Table 2 for a summary of its composition.

3.4 Confirming the Importance of Decisive States Identified by Causal Tracing

In Section 2, we used Causal Tracing to identify decisive hidden states. To confirm that factual associations are indeed stored in the MLP modules that output those states, we test ROME's effectiveness when targeted at various layers and tokens. Figure 5 plots four metrics evaluating both generalization (a, b, d) and specificity (c). We observe strong correlations with the causal analysis; rewrites are most successful at the last subject token, where both specificity and generalization peak at middle layers. Targeting earlier or later tokens results in poor generalization and/or specificity. Furthermore, the layers at which edits generalize best correspond to the middle layers of the early site identified by Causal Tracing, with generalization peaking at the 18th layer. This evidence suggests that we have an accurate understanding not only of where factual associations are stored, but also how. Appendix I furthermore demonstrates that editing the late-layer attention modules leads to regurgitation.

::: {caption="Table 3: Quantitative Editing Results. 95% confidence intervals are in parentheses. Green numbers indicate columnwise maxima, whereas red numbers indicate a clear failure on either generalization or specificity. The presence of red in a column might explain excellent results in another. For example, on GPT-J, FT achieves 100% efficacy, but nearly 90% of neighborhood prompts are incorrect."}

:::

Table 3 showcases quantitative results on GPT-2 XL (1.5B) and GPT-J (6B) over 7, 500 and 2, 000-record test sets in $\textsc{CounterFact}$, respectively. In this experiment, in addition to the baselines tested above, we compare with a method based on neuron interpretability, Knowledge Neurons (KN) ([9]), which first selects neurons associated with knowledge via gradient-based attribution, then modifies MLP weights at corresponding rows by adding scaled embedding vectors. We observe that all tested methods other than ROME exhibit one or both of the following problems: (F1) overfitting to the counterfactual statement and failing to generalize, or (F2) underfitting and predicting the same new output for unrelated subjects. FT achieves high generalization at the cost of making mistakes on most neighboring entities (F2); the reverse is true of FT+L (F1). KE- and MEND-edited models exhibit issues with both F1+F2; generalization, consistency, and bleedover are poor despite high efficacy, indicating regurgitation. KN is unable to make effective edits (F1+F2). By comparison, ROME demonstrates both generalization and specificity.

3.5 Comparing Generation Results

Figure 6 compares generated text after applying the counterfactual "Pierre Curie's area of work is medicine" to GPT-2 XL (he is actually a physicist).

Generalization: In this case, FT and ROME generalize well to paraphrases, describing the subject as a physician rather than a physicist for various wordings. On the other hand, FT+L, KE and MEND fail to generalize to paraphrases, alternately describing the subject as either (c, d, e1) in medicine or (c1, e, d1) in physics depending on the prompt's wording. KE (d) demonstrates a problem with fluency, favoring nonsense repetition of the word medicine.

Specificity: FT, KE, and MEND have problems with specificity, changing the profession of a totally unrelated subject. Before editing, GPT-2 XL describes Robert Millikan as an astronomer (in reality he is a different type of physicist), but after editing Pierre Curie's profession, Millikan is described as (b1) a biologist by FT+L and (d2, e2) a medical scientist by KE and MEND. In contrast, ROME is specific, leaving Millikan's field unchanged. See Appendix for additional examples.

3.6 Human evaluation

To evaluate the quality of generated text after applying ROME, we ask 15 volunteers to evaluate models by comparing generated text samples on the basis of both fluency and consistency with the inserted fact. Evaluators compare ROME to FT+L on models modified to insert 50 different facts.

We find that evaluators are 1.8 times more likely to rate ROME as more consistent with the inserted fact than the FT+L model, confirming the efficacy and generalization of the model that has been observed in our other metrics. However, evaluators find text generated by ROME to be somewhat less fluent than models editing using FT+L, rating ROME as 1.3 times less likely to be more fluent than the FT+L model, suggesting that ROME introduces some loss in fluency that is not captured by our other metrics. Further details of the human evaluation can be found in Appendix J.

3.7 Limitations

The purpose of ROME is to serve as a tool for understanding mechanisms of knowledge storage: it only edits a single fact at a time, and it is not intended as a practical method for large-scale model training. Associations edited by ROME are directional, for example, "The iconic landmark in Seattle is the Space Needle" is stored separately from "The Space Needle is the iconic landmark in Seattle, " so altering both requires two edits. A scalable approach for multiple simultaneous edits built upon the ideas in ROME is developed in [26].

ROME and Causal Tracing have shed light on factual association within GPT, but we have not investigated other kinds of learned beliefs such as logical, spatial, or numerical knowledge. Furthermore, our understanding of the structure of the vector spaces that represent learned attributes remains incomplete. Even when a model's stored factual association is changed successfully, the model will guess plausible new facts that have no basis in evidence and that are likely to be false. This may limit the usefulness of a language model as a source of facts.

4. Related Work

Section Summary: Researchers have explored what language models learn by probing their internal representations, often using classifiers to detect encoded properties, though these methods can miss true causal influences on behavior. More reliable approaches use causal interventions, such as mediation analysis, to trace how specific neurons or hidden states affect outputs like biased predictions or factual recall. Other work evaluates and edits stored knowledge through prompt-based testing and targeted weight modifications in transformer layers, with methods like ROME aiming to match or improve on prior editing techniques in accuracy and reliability.

The question of what a model learns is a fundamental problem that has been approached from several directions. One line of work studies which properties are encoded in internal model representations, most commonly by training a probing classifier to predict said properties from the representations ([27, 28, 29, 30, 31, 32] inter alia). However, such approaches suffer from various limitations, notably being dissociated from the network's behavior ([33]). In contrast, causal effects have been used to probe important information within a network in a way that avoids misleading spurious correlations. [12, 34] introduced the use of causal mediation analysis to identify individual neurons that contribute to biased gender assumptions, and [35] have used a similar methodology to investigate mechanisms of syntactic agreement in language models. [36] described a framework that applies interventions on representations and weights to understand the causal structure of models. [37] proposed erasing specific information from a representation in order to measure its causal effect. Extending these ideas, our Causal Tracing method introduces paired interventions that allow explicit measurement of causal indirect effects ([11]) of individual hidden state vectors.

Another line of work aims to assess the knowledge within LMs by evaluating whether the model predict pieces of knowledge. A common strategy is to define a fill-in-the-blank prompt, and let a masked LM complete it ([1, 38]). Later work showed that knowledge extraction can be improved by diversifying the prompts ([2, 39]), or by fine-tuning a model on open-domain textual facts ([3]). However, constructing prompts from supervised knowledge extraction data risks learning new knowledge instead of recalling existing knowledge in an LM ([39]). More recently, [7] introduced ParaRel, a curated dataset of paraphrased prompts and facts. We use it as a basis for constructing $\textsc{CounterFact}$, which enables fine-grained measurements of knowledge extraction and editing along multiple dimensions. Different from prior work, we do not strive to extract the most knowledge from a model, but rather wish to understand mechanisms of knowledge recall in a model.

Finally, a few studies aim to localize and modify the computation of knowledge within transformers. [8] identify the MLP layers in a (masked LM) transformer as key–value memories of entities and information associated with that entity. Building on this finding, [9] demonstrate a method to edit facts in BERT by writing the embedding of the object into certain rows of the MLP matrix. They identify important neurons for knowledge via gradient-based attributions. [10] train a hyper-network to predict a weight update at test time, which will alter a fact. They experiment with BERT and BART ([40]), a sequence-to-sequence model, and focus on models fine-tuned for question answering. [14] presents a hyper-network method that learns to transform the decomposed terms of the gradient in order to efficiently predict a knowledge update, and demonstrates the ability to scale up to large models including T5 ([41]) and GPT-J ([16]). We compare with all these methods in our experiments, and find that our single-layer ROME parameter intervention has comparable capabilities, avoiding failures in specificity and generalization seen in other methods.

5. Conclusion

Section Summary: In this conclusion, the researchers explain that they have mapped out how information moves through autoregressive transformer models when recalling knowledge, leading them to create a straightforward editing tool named ROME. Their work reveals details about where and how facts are stored in these models and shows that it's possible to directly adjust the inner workings of large pretrained AI systems. However, the current approach handles only one fact at a time, with efforts to expand it to multiple facts addressed in related studies, and all their code and resources have been made publicly available.

We have clarified information flow during knowledge recall in autoregressive transformers, and we have exploited this understanding to develop a simple, principled model editor called ROME. Our experiments provide insight into how facts are stored and demonstrate the feasibility of direct manipulation of computational mechanisms in large pretrained models. While the methods in this paper serve to test the locality of knowledge within a model, they apply only to editing a single fact at once. Adapting the approach to scale up to many more facts is the subject of other work such as [26].

Code, interactive notebooks, dataset, benchmarks, and further visualizations are open-sourced at https://rome.baulab.info.

6. Ethical Considerations

Section Summary: This research on understanding and editing knowledge inside large language models could make the systems more transparent and cheaper to fix when they make mistakes. At the same time, the ability to change what a model knows opens the door to harmful misuse, such as inserting false information or bias. The authors therefore caution against treating these models as reliable sources of facts in any high-stakes setting.

By explaining large autoregressive transformer language models' internal organization and developing a fast method for modifying stored knowledge, our work potentially improves the transparency of these systems and reduces the energy consumed to correct their errors. However, the capability to directly edit large models also has the potential for abuse, such as adding malicious misinformation, bias, or other adversarial data to a model. Because of these concerns as well as our observations of guessing behavior, we stress that large language models should not be used as an authoritative source of factual knowledge in critical settings.

Acknowledgements

Section Summary: The authors thank Antonio Torralba, Martin Wattenberg, and Bill Ferguson for their helpful discussions, financial support, and encouragement that made the project possible. The work also received funding from an AI Alignment grant by Open Philanthropy, along with grants from DARPA for some of the researchers. One author received additional support from the Israel Science Foundation and an Azrieli Foundation fellowship.

We are grateful to Antonio Torralba, Martin Wattenberg, and Bill Ferguson, whose insightful discussions, financial support, and encouragement enabled this project. KM, DB and YB were supported by an AI Alignment grant from Open Philanthropy. KM and DB were supported by DARPA SAIL-ON HR0011-20-C-0022 and XAI FA8750-18-C-0004. YB was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20) and an Azrieli Foundation Early Career Faculty Fellowship.

Appendix

Section Summary: The appendix first derives an algebraic solution for updating a weight matrix to satisfy a new input-output constraint while preserving a prior least-squares fit; it defines a Lagrangian, solves the resulting linear system in block form, and arrives at a simple formula for the multiplier vector Lambda that produces the rank-one update. It then details causal-tracing experiments on GPT-2 XL, describing how factual prompts are selected, how subject-token embeddings are corrupted with noise, and how individual or grouped hidden states are restored layer by layer to measure their causal impact on correct predictions. A further subsection explains the procedure for isolating the contributions of MLP versus attention modules by restoring runs of ten consecutive activations.

A. Solving for Lambda Algebraically

Here we present the detailed derivation of Equation 2, including the linear system that is used to calculate $\Lambda$ from $v_*$, $C$, and $k_*$. This derivation is included for clarity and completeness and is a review of the classical solution of least-squares with equality constraints as applied to our setting, together with the rank-one update rule that was proposed in [23].

We assume that $W$ is the optimal least-squares solution for memorizing a mapping from a previous set of keys $K$ to values $V$; this solution can be written using the normal equations as follows.

$ \begin{aligned} \text{the }W\text{ that minimizes} \quad &|| W K - V ||_F^2 \ \text{solves} \quad & W K K^T = V K^T \end{aligned} $

Here the Frobenius norm is used to write the total square error since the variable being optimized is a matrix $W$ rather than a vector $x$ as in the classical textbook presentation of least squares.

We wish to find a new matrix $\hat{W}$ that solves the same least squares problem with an additional equality constraint as written in Equation 2:

$ \hat{W}k_* = v_* $

This is the well-studied problem of least squares with a linear equality constraint. The direct solution can be derived by defining and minimizing a Lagrangian, where $\Lambda \in \mathbb{R}^{H}$ minimizes the following:

$ \begin{aligned} \text{define}\quad L(\hat{W}, \Lambda) &= \frac{1}{2} ||\hat{W}K-V||F^2 - \Lambda^T(\hat{W}k-v_) \ &=\frac{1}{2} (\hat{W}K)(\hat{W}K)^T - V(\hat{W}K)^T + \frac{1}{2}VV^T - \Lambda^T(\hat{W}k_*-v_*) \ \text{setting} \quad 0 = \frac{\partial L}{\partial \hat{W}} &= \hat{W}(KK^T) - VK^T - \Lambda k_*^T\ \hat{W}KK^T &= VK^T + \Lambda k_*^T \end{aligned} $

Subtracting Equation 5 from Equation 7, most terms cancel, and we obtain the update rule:

$ \begin{aligned} (\hat{W}-W) KK^T & = \Lambda k_^T \ \hat{W} & = W + \Lambda (C^{-1}k_)^T \end{aligned} $

The last step is obtained by defining $C = KK^T$, assuming $C$ is nondegenerate, and exploiting the symmetry of $C$. Here we also write the row vector term as $u^T = (C^{-1}k_*)^T \in \mathbb{R}^{D}$, so we can write simply (rearranging Equation 2 and 8):

$ \begin{aligned} \hat{W}I - \Lambda u^T &= W \end{aligned} $

To solve for $\Lambda$, we note that Equation 9 and 6 form a linear system that allows both $\hat{W}$ and $\Lambda$ to be solved simultaneously if written together in block form.

$ \begin{aligned} \left[\begin{array}{c|c} \ \quad \hat{W} \quad, & \Lambda \ , \end{array}\right] \left[\begin{array}{c|c} \ \quad; I \quad;; & k_* \ \ \hline -u^T \rule{0pt}{2.2ex} & 0 \end{array}\right] & = \left[\begin{array}{c|c} \mathstrut \ \quad W \quad, & v_* \ \mathstrut \end{array}\right] \end{aligned} $

That is equivalent to substituting Equation 8 into Equation 6 and calculating the following:

$ \begin{aligned} \hat{W} k_* & = (W + \Lambda u^T) k_* = W k_* + \Lambda (u^T k_*) = v_* \ \Lambda & = \frac{v_* - W k_*}{u^T k_*} = \frac{v_* - W k_*}{(C^{-1} k_*)^T k_*} \end{aligned} $

B. Causal Tracing

B.1 Experimental Settings

Note that, in by-layer experimental results, layers are numbered from 0 to $L-1$ rather than $1$ to $L$.

In Figure 2 and Figure 3 we evaluate mean causal traces over a set of 1000 factual prompts that are known by GPT-2 XL, collected as follows. We perform greedy generation using facts and fact templates from $\textsc{CounterFact}$, and we identify predicted text that names the correct object $o^c$ before naming any other capitalized word. We use the text up to but not including the object $o^c$ as the prompt, and we randomly sample 1000 of these texts. In this sample of known facts, the predicted probability of the correct object token calculated by GPT-2 XL averages 27.0%.

In the corrupted run, we corrupt the embeddings of the token naming the subject $s$ by adding Gaussian noise $\epsilon \sim \mathcal{N}(0; \nu)$, where $\nu = 3\sigma_t$ is set to be three times larger than the observed standard deviation $\sigma_t$ of token embeddings as sampled over a body of text. For each run of text, the process is repeated ten times with different samples of corruption noise. On average, this reduces the correct object token score to 8.47%, less than one third the original score.

When we restore hidden states from the original run, we substitute the originally calculated values from the same layer and the same token, and then we allow subsequent calculations to proceed without further intervention. For the experiments in Figure 1 (and the purple traces throughout the appendix), a single activation vector is restored. Naturally, restoring the last vector on the last token will fully restore the original predicted scores, but our plotted results show that there are also earlier activation vectors at a second location that also have a strong causal effect: the average maximum score seen by restoring the most impactful activation vector at the last token of the subject is 19.5%. In Figure 1 j where effects are bucketed by layer, the maximum effect is seen around the 15th layer of the last subject token, where the score is raised on average to 15.0%.

B.2 Separating MLP and Attn Effects

When decomposing the effects into MLP and Attn lookups, we found that restoring single activation vectors from individual MLP and individual Attn lookups had generally negligible effects, suggesting the decisive information is accumulated across layers. Therefore for MLP and Attn lookups, we restored runs of ten values of $\smash{m^{(l)}i}$ (and $\smash{a^{(l)}i}$, respectively) for an interval of layers ranging from $[l* - 4, ..., l* + 5]$ (clipping at the edges), where the results are plotted at layer $l_*$. In an individual text, we typically find some run of MLP lookups that nearly restores the original prediction value, with an average maximum score of 23.6%. Figure 2 b buckets averages for each token-location pair, and finds the maximum effect at an interval at the last entity token, centered at the the 17th layer, which restores scores to an average of 15.0%. For Attn lookups (Figure 2 c), the average maximum score over any location is 19.4%, and when bucketed by location, the maximum effect is centered at the 32nd layer at the last word before prediction, which restores scores to an average of 16.5%.

Figure 7 shows mean causal traces as line plots with 95% confidence intervals, instead of heatmaps.

B.3 Traces of EleutherAI GPT-NeoX (20B) and GPT-J (6B) and smaller models

We conduct the causal trace experiment using on GPT-NeoX (20 billion parameters) as well as GPT-J (6 billion parameters). For GPT-NeoX we adjust the injected noise to $\nu = 0.03$ and in GPT-J we use $\nu = 0.025$ to match embedding magnitudes. We use the same factual prompts as GPT-2 XL, eliminating cases where the larger models would have predicted a different word for the object. Results are shown in Figure 8. GPT-NeoX and GPT-J differ from GPT-2 because they have has fewer layers (44 and 28 layers instead of 48), and a slightly different residual structure across layers. Nevertheless, the causal traces look similar, with an early site with causal states concentrated at the last token of the subject, a large role for MLP states at that site. Again, attention dominates at the last token before prediction.

There are some differences compared to GPT-2. The importance of attention at the first layers of the last subject token is more apparent in GPT-Neo and GPT-J compared to GPT-2, suggesting that the attention parameters may be playing a larger role in storing factual associations. This concentration of attention at the beginning may also be due to fewer layers in the Eleuther models: attending to the subject name must be done in a concentrated way at just a layer or two, because there are not enough layers to spread out that computation in the shallower model. The similarity between the GPT-NeoX and GPT-J and GPT-2 XL traces helps us to understand why ROME continues to work well with higher-parameter models, as seen on our experiments in altering parameters of GPT-J.

To examine effects over a wide range of scales, we also compare causal traces for smaller models GPT-2 Medium and GPT-2 Large. These smaller models are compared to NeoX-20B in Figure 9. We find that across sizes and architectural variations, early-site MLP modules continue to have high indirect causal effects at the last subject token, although the layers where effects peak are different from one model to another.

B.4 Causal Tracing Examples and Further Insights

We include further examples of phenomena that can be observed in causal traces. Figure 10 shows typical examples across different facts. Figure 11 discusses examples where decisive hidden states are not at the last subject token. Figure 14 examines traces at an individual token in more detail.

We note that causal tracing depends on a corruption rule to create baseline input for a model that does not contain all the information needed to make a prediction. Therefore we ask: are Causal Tracing results fragile if the exact form of the corruption changes? We test this by expanding the corruption rule: even when additional tokens after the subject name are also corrupted, we find that the results are substantially the same. Figure 12 shows causal traces with the expanded corruption rule. Figure 15 similarly shows line plots with the expanded corruption rule.

We do find that the noise must be large enough to create large average total effects. For example, if noise with variance that is much smaller is used (for example if we set $\sigma=\sigma_t$), average total effects become very small, and the small gap in the behavior between clean runs and corrupted run makes it difficult discern indirect effects of mediators. Similarly, if we use a uniform distribution where components range in $\pm 3\sigma$, effects large enough for causl tracing but smaller than a Gaussian distribution.

If instead of using spherical Gaussian noise, we draw noise from $\mathcal{N}(mu, \Sigma)$ where we set $\mu = \mu_t$ and $\Sigma_ = \Sigma_t$ to match the observed distribution over token embeddings, average total effects are also strong enough to perform causal traces. This is shown in Figure 13.

Furthermore, we investigate whether Integrated Gradients (IG) ([18]) provides the same insights as Causal Tracing. We find that IG is very sensitive to local features but does not yield the same insights about large-scale global logic that we have been able to obtain using causal traces. Figure 16 compares causal traces to IG saliency maps.

{width=100%}

{width=100%}

{width=100%}

{width=100%}

{width=100%}

C. Details on the zsRE Evaluation Task

Dataset Details. The zsRE question answering task ([42]) was first used for factual knowledge evaluation by [10], later being extended and adopted by [14]. In our study, we use the same train/test splits as [14]; note that non-hypernetwork methods (including ROME) do not require training, so the corresponding dataset split is discarded in those cases. Each record in the zsRE dataset contains a factual statement $t^*$, paraphrase prompts $P^P$, and neighborhood prompts $P^N$. $t^*$ and $P^N$ were included in the original version of zsRE, whereas $P^N$ was added by [14] via sampling of a random dataset element. See Figure 22 for an example record.

Additional Baselines. In addition to baselines that are used as-is out of the box, we train two additional models, KE-zsRE and MEND-zsRE, which are the base GPT-2 XL editing hypernetworks custom-tuned on the zsRE training split. This is done to ensure fair comparison; the original pre-trained KE and MEND models were created using a WikiText generation task ([10, 14]), rather than zsRE.

D. Details on the CounterFact Dataset

$\textsc{CounterFact}$ is designed to enable distinction between superficial changes in model word choices from specific and generalized changes in underlying factual knowledge. Table 2 summarizes statistics about $\textsc{CounterFact}$ 's composition.

Each record in $\textsc{CounterFact}$ is derived from a corresponding entry in $\textsc{ParaRel}$ ([7]) containing a knowledge tuple $t^c = (s,r,o^c)$ and hand-curated prompt templates $\mathcal{T}(r)$, where all subjects, relations, and objects exist as entities in WikiData. Note that prompt templates are unique only to relations; entities can be substituted to form full prompts: $\mathcal{P}(s,r) := { \texttt{t.format(s)} \mid \texttt{t} \in \mathcal{T}(r) }$, where $\texttt{.format()}$ is string substitution. For example, a template for $(r=\text{plays sport professionally})$ might be " plays the sport of, " where "LeBron James" substitutes for "".

Solely using the $\textsc{ParaRel}$ entry, we derive two elements. A requested rewrite is represented as ${s, r, o^c, o^*, p^* }$, where $p^* \sim\mathcal{P}(s, r)$ is the sole rewriting prompt, and $o^*$ is drawn from a weighted sample of all $\textsc{ParaRel}$ tuples with the predicate $(r, \cdot)$. Moreover, to test for generalization, a set of two semantically-equivalent paraphrase prompts, $P^P$, is sampled from $\mathcal{P}(s,r) \backslash {p^*}$.

To test for specificity, we execute a WikiData SPARQL query^2 to collect a set of entities that share a predicate with $s$: $\mathcal{E} ={ s^\prime \mid (s^\prime, r, o^c) }$; e.g., for $(s=\textit{Eiffel Tower}, r=\textit{city location}, o^c=\textit{Paris})$, $\mathcal{E}$ might contain entities like the Champs-Élysées or Louvre. We then construct a set of prompts ${\mathcal{P}(s^\prime, r) \mid s^\prime \in \mathcal{E}}$ and sample ten to get our neighborhood prompts, $P^N$. Our rationale for employing this strategy over random sampling is that the $s^\prime$ we select are close to $s$ in latent space and thus more susceptible to bleedover when editing $s$ using linear methods. Comparing the Drawdown column in Table 1 with the Neighborhood Scores and Magnitudes in Table 3, we observe the improved resolution of $\textsc{CounterFact}$ 's targeted sampling.

Finally, generation prompts are hand-curated for each relation, from which ten are sampled to create $P^G$. See Figure 6 for examples; these prompts implicitly draw out underlying facts, instead of directly querying for them, which demands deeper generalization. For evaluating generations, we provide reference texts $RT$, which are Wikipedia articles for a sample of entities from ${s^\prime \mid (s^\prime, r, o^*) }$; intuitively, these contain $n$-gram statistics that should align with generated text.

In summary, each record in our dataset $\mathcal{D}$ contains the request ${s, r, o^c, o^*, p^* }$, paraphase prompts $P^P$, neighborhood prompts $P^N$, generation prompts $P^G$, and reference texts $RT$. See Figure 21 for an example record. Compared to other evaluation benchmarks, $\textsc{CounterFact}$ provides several new types of tests that allow precise evaluation of knowledge editing (Table 2).

E. Method Implementation Details

E.1 [GPT-2 XL, GPT-J] Fine-Tuning (FT), Constrained Fine-Tuning (FT+L)

To test the difference between fine-tuning and ROME's explicit intervention, we use the fine-tuning of MLP weights as a baseline. Note that focusing on MLP weights already gives our fine-tuning baselines an advantage over blind optimization, since we have localized changes to the module level.

For basic Fine-Tuning (FT), we use Adam [43] with early stopping to minimize $-\log \mathbb{P}{G^\prime}\left[ o^* \mid p \right]$, changing only $\mathrm{mlp}{proj}$ weights at one layer. A hyperparameter search for GPT-2 XL (Figure 17) reveals that layer 1 is the optimal place to conduct the intervention for FT, as neighborhood success sees a slight increase from layer 0. Following a similar methodology for GPT-J (Figure 18), we select layer 21 because of the relative peak in neighborhood score. For both models, we use a learning rate of $5\times 10^{-4}$ and early stop at a 0.03 loss.

For constrained fine-tuning (FT+L), we draw from [13] by adding an $L_\infty$ norm constraint: $\lVert \theta_G - \theta_{G^\prime} \rVert_\infty \leq \epsilon$. This is achieved in practice by clamping weights $\theta_G^\prime$ to the $\theta_G \pm \epsilon$ range at each gradient step. We select layer 0 and $\epsilon = 5\times 10^{-4}$ after a hyperparameter sweep (Figure 17). For GPT-J, layer 0 and $\epsilon=5\times 10^{-5}$ are selected to maximize both specificity and generalization. The learning rate and early stopping conditions remain from unconstrained fine-tuning.

E.2 [GPT-2 XL only] Knowledge Neurons (KN)

The method by [9] first selects neurons that are associated with knowledge expression via gradient-based attributions, and then modifies $\smash{\mathrm{mlp}_{proj}^{(l)}}$ at the rows corresponding to those neurons by adding scaled embedding vectors. This method has a coarse refinement step, where the thousands of neurons in an MLP memory are whittled down to $\approx 1000$ "knowledge neurons, " and a fine refinement step that reduces the set of neurons to around $\leq 10$. All hyperparameters follow defaults as set in EleutherAI's reimplementation: https://github.com/EleutherAI/knowledge-neurons.

E.3 [GPT-2 XL only] Knowledge Editor (KE)

[10] learn an LSTM sequence model that uses gradient information to predict rank-1 weight changes to $G$. Because the official code does not edit GPT-2, we use [14]'s re-implementation in their study. To encourage fair comparison on both zsRE and $\textsc{CounterFact}$ tasks, we additionally train KE-zsRE and KE-CF models on size-10, 000 subsets of the respective training sets. Hyperparameters for training are adopted from the given default configuration. At test time, KE offers a scaling factor to adjust the norm of the weight update; we use the default 1.0.

E.4 [GPT-2 XL, GPT-J] Model Editor Networks with Gradient Decomposition (MEND)

[14] learn a rank-1 decomposition of the negative log likelihood gradient with respect to some subset of $\theta_G$ (in practice, this amounts to several of the last few layers of the transformer network). Again, for fair comparison, we train new versions of MEND (MEND-zsRE, MEND-CF) on the same sets that KE-zsRE and KE-CF were trained on. Similar to KE, hyperparameters for training and test-time inference are adopted from default configurations.

E.5 [GPT-2 XL, GPT-J] Rank-One Model Editing (ROME)

ROME's update (Section 3.1) consists of key selection Equation (3), value optimization Equation (4), and $v$ insertion (Appendix A). We perform the intervention at layer 18. As Figure 1 k shows, this is the center of causal effect in MLP layers, and as Figure 3 shows, layer 18 is approximately when MLP outputs begin to switch from acting as keys to values.

Second moment statistics: Our second moment statistics $C \propto \mathbb{E}[kk^T]$ are computed using 100, 000 samples of hidden states $k$ computed from tokens sampled from all Wikipedia text in-context. Notice that sampling is not restricted to only special subject words; every token in the text is included in the statistic. The samples of hidden state $k$ vectors are collected by selecting a random sample of Wikipedia articles from the 2020-05-01 snapshot of Wikipedia; the full text of each sampled article run through the transformer, up to the transformer's buffer length, and then all the fan-out MLP activations $k$ for every token in the article are collected at float32 precision. The process is repeated (sampling from further Wikipedia articles without replacement) until 100,000 $k$ vectors have been sampled. This sample of vectors is used to compute second moment statistics.

Key Selection: We sample 20 texts to compute the prefix ($x_j$ in Equation 3): ten of length 5 and ten of length 10. The intention is to pick a $k_*$ that accounts for the different contexts in which $s$ could appear. Note that we also experimented with other $x_j$ sampling methods:

• No prefix. This baseline option performed worse ($\mathrm{S}^\prime = 86.1$ compared to $\mathrm{S}=89.2$).
• Longer prefixes. Using ten of length 5, ten of length 10, and ten of length 50 did not help performance much ($\mathrm{S}^\prime = 89.3$).
• More same-length prefixes. Using thirty of length 5 and thirty of length 10 did not help performance much ($\mathrm{S}^\prime = 89.2$).

Value Optimization: $v_*$ is solved for using Adam with a learning rate of $0.5$ and $1.5\times 10^{-3}$ weight decay. The KL divergence scaling factor, denoted $\lambda$ in Equation 4, is set to $1\times 10^{2}$. The minimization loop is run for a maximum of 20 steps, with early stopping when $\mathcal{L}(z)$ reaches $5\times 10^{-2}$.

The entire ROME edit takes approximately $2\mathrm{s}$ on an NVIDIA A6000 GPU for GPT-2 XL. Hypernetworks such as KE and MEND are much faster during inference (on the order of $100\mathrm{ms}$), but they require hours-to-days of additional training overhead.

F. Extended Quantitative Results

::: {caption="Table 4: Extended Quantitative Editing Results. Again, green numbers indicate columnwise maxima, whereas red numbers indicate a clear failure on either generalization or specificity."}

:::

::: {caption="Table 5: Extended zsRE Editing Results. Drawdown is measured with respect to the vanilla GPT-2 model. Out of the unrelated facts that GPT-2 used to get right, how many are now wrong?"}

:::

To demonstrate that ROME is also effective on smaller autoregressive language models, we perform $\textsc{CounterFact}$ and zsRE evaluations on both GPT-2 Medium (345M) and GPT-2 Large (774M). As Table 4 and Table 5 reflect, ROME outperforms the next-best baseline as measured on GPT-2 XL (FT+L).

G. Generation Examples

G.1 GPT-2 XL (1.5B) Generation Examples

We select four additional cases from $\textsc{CounterFact}$ to examine qualitatively, selecting representative generations to display. Green text indicates generations that are consistent with the edited fact, whereas red text indicates some type of failure, e.g. essence drift, fluency breakage, or poor generalization. Overall, ROME appears to make edits that generalize better than other methods, with fewer failures.

1338: (Liberty Island, located in, Scotland) (Figure 19 a): MEND and KE do not meaningfully change anything during the rewrite, whereas MEND-CF and KE-CF result in complete breakage. ROME, FT, and FT+L produce the most interesting generations. Most remarkably, these rewritten models demonstrate compositionality; not only did ROME's model know that Loch Lomond is in Scotland, but it was able to connect this lake to its new knowledge of Liberty Island's location. Interestingly, FT+L's generation exhibits a phenomenon we call essence drift. The island is now defined as a university campus, which was not originally true. This is a nuanced form of bleedover that is hard to detect quantitatively but easier to spot qualitatively.

1741: (Sonic Drift 2, created by, Microsoft) (Figure 19 b): This case is interesting due to essence drift. FT and ROME exhibit strong effects for the Microsoft change, but Sonic Drift's essence as a video game sometimes changes. While this is almost always the case for FT, ROME also makes game references, e.g. Playdead. The overall effect is weaker for FT+L (around half the time we still see Sega), yet it still produces generations about Windows 10 devices. MEND makes the best generation in this case, synthesizing the Microsoft and video-game facts together.

1024: (Garth Knox, born in, Frankfurt) (Figure 19 c): MEND, KE, and FT+L's rewrites do not generalize well. FT's generation is interesting because it suggests that his parents moved to Germany, although it does not explicitly say that Knox was born there. ROME's generation is straightforward and correct.

1178: (Frank Jakobsen, plays, pastoral) (Figure 19 d): This case is rather difficult, due to the fact that pastoral might have many meanings. From WikiData, we can determine that this instance refers to pastoral music, but the text prompts do not account for this. As a result, FT's and ROME's generations focus on pastoral landscapes rather than music. FT+L, KE, and MEND do not exhibit much change. Note that ROME produces a slight glitch with two *pastoral*s in a row.

G.2 GPT-J (6B) Generation Examples

We also provide generation samples on GPT-J (6B). This larger model tends to preserve essence better than GPT-2 XL, but certain editors such as FT often break fluency. Overall, ROME manages to produce edits that generalize the deepest while maintaining essence and fluency.

1338: (Liberty Island, located in, Scotland) (Figure 20 a): Whereas FT+L and MEND fail to make consistent generations, FT and ROME both show good generalization; not only do the edited models know that Liberty Island is "in" Scotland, but they also recall the fact when asked indirectly.

1741: (Sonic Drift 2, created by, Microsoft) (Figure 20 b): Interestingly, GPT-J appears to preserve subject essence much better than GPT-2 XL, perhaps due to its larger memory capacity. Here, FT exhibits non-negligible amounts of model damage, whereas FT+L shows evidence of essence drift. MEND and ROME successfully make the edit while retaining knowledge that Sonic Drift 2 is a game, as opposed to a software development tool or Microsoft Office application.

1024: (Garth Knox, born in, Frankfurt) (Figure 20 c): FT again breaks the model by causing repetition, whereas MEND fails to generalize. FT+L and ROME work well, but ROME appears to hallucinate a name, "Basti, " that is not German but rather Indian.

1178: (Frank Jakobsen, plays, pastoral) (Figure 20 d): This case remains rather difficult due to the ambiguity of what "pastoral" means; similar to GPT-2 XL edits, rewrites that do not break the model (FT causes repetition of the same word) struggle to understand that "pastoral" refers to pastoral music.

H. Dataset Samples

See Figure 21 for a sample record in $\textsc{CounterFact}$, complete with tests for all 5 rewrite success criteria. Figure 22 shows a record of the zsRE dataset.

I. Are Attention Weight Interventions Effective?

{width=80%}

Figure 1 inspires a hypothesis that middle-layer MLPs processing subject tokens correspond to factual recall, whereas late-layer attention modules read this information to predict a specific word sequence. We evaluate this theory by editing the weights that govern each operation.

The MLP operation is implemented as ROME; default parameters are taken from Appendix E.5. The attention operation is called AttnEdit, which applies constrained fine-tuning on the $\smash{W_i^Q, W_i^K, W_i^V}$ weights of all heads $i$ at some layer of the network.[^3] This layer is chosen to be 33, the center of high causal effect in the attention causal trace (Figure 1 l). To determine the $L_\infty$ norm constraint on fine-tuning, we run a grid search (Figure 23):

[^3]: See [15] for additional details on attention; the $W_i^Q, W_i^K, W_i^V$ notation is lifted from there.

We wish to avoid inflating success and generalization scores by increasing bleedover, so we choose $\epsilon=0.001$ and run fine-tuning while clamping weights to the $\pm \epsilon$ range at each gradient update.

Examination of generation text supports our hypothesis. Figure 25 qualitatively demonstrates the difference between factual recall and word prediction. Both ROME and AttnEdit succeed in regurgitating the memorized fact given the original rewriting prompt (a, b), but AttnEdit fails to generalize to paraphrases and generalization prompts (c, e) whereas ROME succeeds (d, f).

J. Human Evaluation

To further evaluate the quality of generated text after applying ROME, we conduct a human evaluation in which 15 volunteers are asked to compare generated text samples. 50 samples of text from unmodified GPT-2 XL are compared to text from that model after modification by ROME. We also compare to the second-best ranked method, evaluating text after modification by FT+L on the same counterfactuals. Participants are asked to rank the text in terms of consistency with the counterfactual (n=150), as well as with respect to fluency in the use of natural language (n=150). Results are summarized in Figure 26, and randomly-sampled examples are shown in Figure 27, Figure 28, Figure 29.

Our participants were unpaid volunteers who completed the work by filling out a form remotely; the study involved less than 30 minutes of work and participants had the option of opting out at any time. Figure 30 shows the full instructions.

We observe that ROME is much more successful than FT+L at generating text that is consistent with the counterfactual; this finding is consistent results in Table 3 that show that ROME generalizes better than FT+L. Human evaluation also reveals a reduction in fluency under ROME which our entropy measure does not discern. Some of the differences are subtle: examples of fluency losses detected by human raters can be seen in Figure 27, Figure 28.

References

Section Summary: This section compiles a numbered list of academic citations that support the paper's discussion of language models and their handling of knowledge. The references include foundational and recent studies on how models like transformers encode facts, methods for editing or probing that information, and related techniques from causality and interpretability research. They draw from major conferences such as EMNLP and NeurIPS as well as key publications on models including BERT and GPT variants.

[1] Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463–2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi:10.18653/v1/D19-1250. URL https://aclanthology.org/D19-1250.

[2] Jiang, Z., Xu, F. F., Araki, J., and Neubig, G. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438, 2020. doi:10.1162/tacl_a_00324. URL https://aclanthology.org/2020.tacl-1.28.

[3] Roberts, A., Raffel, C., and Shazeer, N. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418–5426, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.437. URL https://aclanthology.org/2020.emnlp-main.437.

[4] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, pp. 9, 2019.

[5] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877–1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

[6] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi:10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.

[7] Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., and Goldberg, Y. Measuring and Improving Consistency in Pretrained Language Models. Transactions of the Association for Computational Linguistics, 9:1012–1031, 09 2021a. ISSN 2307-387X. doi:10.1162/tacl_a_00410. URL https://doi.org/10.1162/tacl_a_00410.

[8] Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484–5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.emnlp-main.446.

[9] Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493–8502, 2022.

[10] De Cao, N., Aziz, W., and Titov, I. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491–6506, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.emnlp-main.522.

[11] Pearl, J. Direct and indirect effects. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, pp. 411–420, 2001.

[12] Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. M. Investigating gender bias in language models using causal mediation analysis. In NeurIPS, 2020b.

[13] Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F., and Kumar, S. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.

[14] Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D. Fast model editing at scale. In International Conference on Learning Representations, 2021.

[15] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017.

[16] Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021.

[17] Pearl, J. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X.

[18] Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319–3328. PMLR, 2017.

[19] Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A mathematical framework for transformer circuits. https://transformer-circuits.pub/2021/framework/index.html, December 2021.

[20] Zhao, S., Pascual, D., Brunner, G., and Wattenhofer, R. Of non-linearity and commutativity in BERT. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE, 2021.

[21] Kohonen, T. Correlation matrix memories. IEEE transactions on computers, 100(4):353–359, 1972.

[22] Anderson, J. A. A simple neural network generating an interactive memory. Mathematical biosciences, 14(3-4):197–220, 1972.

[23] Bau, D., Liu, S., Wang, T., Zhu, J.-Y., and Torralba, A. Rewriting a deep generative model. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.

[24] Hase, P., Diab, M., Celikyilmaz, A., Li, X., Kozareva, Z., Stoyanov, V., Bansal, M., and Iyer, S. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint arXiv:2111.13654, 2021.

[25] Zhang, Y., Galley, M., Gao, J., Gan, Z., Li, X., Brockett, C., and Dolan, W. B. Generating informative and diverse conversational responses via adversarial information maximization. In NeurIPS, 2018.

[26] Meng, K., Sen Sharma, A., Andonian, A., Belinkov, Y., and Bau, D. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022.

[27] Ettinger, A., Elgohary, A., and Resnik, P. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pp. 134–139, Berlin, Germany, August 2016. Association for Computational Linguistics. doi:10.18653/v1/W16-2524. URL https://aclanthology.org/W16-2524.

[28] Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., and Goldberg, Y. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations (ICLR), April 2017.

[29] Hupkes, D., Veldhoen, S., and Zuidema, W. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907–926, 2018.

[30] Conneau, A., Kruszewski, G., Lample, G., Barrault, L., and Baroni, M. What you can cram into a single

[31] Belinkov, Y., Durrani, N., Dalvi, F., Sajjad, H., and Glass, J. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 861–872, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi:10.18653/v1/P17-1080. URL https://aclanthology.org/P17-1080.

[32] Belinkov, Y. and Glass, J. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72, March 2019. doi:10.1162/tacl_a_00254. URL https://aclanthology.org/Q19-1004.

[33] Belinkov, Y. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, pp. 1–13, 11 2021. ISSN 0891-2017. doi:10.1162/coli_a_00422. URL https://doi.org/10.1162/coli_a_00422.

[34] Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Sakenis, S., Huang, J., Singer, Y., and Shieber, S. Causal mediation analysis for interpreting neural NLP: The case of gender bias. arXiv preprint arXiv:2004.12265, 2020a.

[35] Finlayson, M., Mueller, A., Gehrmann, S., Shieber, S., Linzen, T., and Belinkov, Y. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1828–1843, Online, August 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.acl-long.144. URL https://aclanthology.org/2021.acl-long.144.

[36] Feder, A., Oved, N., Shalit, U., and Reichart, R. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333–386, 2021.

[37] Elazar, Y., Ravfogel, S., Jacovi, A., and Goldberg, Y. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, 2021b.

[38] Petroni, F., Lewis, P., Piktus, A., Rocktäschel, T., Wu, Y., Miller, A. H., and Riedel, S. How context affects language models' factual predictions. In Automated Knowledge Base Construction, 2020.

[39] Zhong, Z., Friedman, D., and Chen, D. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5017–5033, Online, June 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.naacl-main.398. URL https://aclanthology.org/2021.naacl-main.398.

[40] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880, Online, July 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.703. URL https://aclanthology.org/2020.acl-main.703.

[41] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.

[42] Levy, O., Seo, M., Choi, E., and Zettlemoyer, L. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 333–342, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi:10.18653/v1/K17-1034. URL https://aclanthology.org/K17-1034.

[43] Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980.