Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

precise model editing

Precise model editing is a technique in machine learning for updating, correcting, or inserting specific factual knowledge within a trained neural network without retraining the entire architecture or degrading unrelated capabilities. In transformer-based language models, this approach isolates and alters the specific internal components responsible for storing factual associations, such as feed-forward network layers, while accounting for the distinct information flows contributed by self-attention mechanisms and residual connections. By decoupling factual updates from general pattern-extraction pathways and localizing parameter adjustments, precise model editing ensures that targeted facts are effectively modified with high specificity and minimal interference to the broader reasoning and language generation performance of the model.

1 item

PMET: Precise Model Editing in a Transformer

PMET: Precise Model Editing in a Transformer

Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, Jie Yu

OrganizationsNational University of Defense Technology

Why you should read this

Proposes PMET, a model editing framework that jointly optimizes attention and feed-forward hidden states while selectively updating feed-forward weights, preventing the transfer of irrelevant attention signals for more accurate factual updates in large language models.

Model editing techniques modify a minor proportion of knowledge in Large Language Models (LLMs) at a relatively low cost, which have demonstrated notable success. Existing methods assume Transformer Layer (TL) hidden states are values of key-value memories of the Feed-Forward Network (FFN). They usually optimize the TL hidden states to memorize target knowledge and use it to update the weights of the FFN in LLMs. However, the information flow of TL hidden states comes from three parts: Multi-Head Self-Attention (MHSA), FFN, and residual connections. Existing methods neglect the fact that the TL hidden states contains information not specifically required for FFN. Consequently, the performance of model editing decreases. To achieve more precise model editing, we analyze hidden states of MHSA and FFN, finding that MHSA encodes certain general knowledge extraction patterns. This implies that MHSA weights do not require updating when new knowledge is introduced. Based on above findings, we introduce PMET, which simultaneously optimizes Transformer Component (TC, namely MHSA and FFN) hidden states, while only using the optimized TC hidden states of FFN to precisely update FFN weights. Our experiments demonstrate that PMET exhibits state-of-the-art performance on both the COUNTERFACT and zsRE datasets. Our ablation experiments substantiate the effectiveness of our enhancements, further reinforcing the finding that the MHSA encodes certain general knowledge extraction patterns and indicating its storage of a small amount of factual knowledge. Our code is available at https://github.com/xpq-tech/PMET.

Added

2026-09-26