keyword
parametric memory
Parametric memory is the implicit store of factual knowledge and linguistic patterns encoded directly within the learned weights and parameters of a neural network during training. Unlike non-parametric memory systems that retrieve information dynamically from explicit external databases, document indexes, or memory tables, parametric memory relies entirely on internal mathematical representations formed through optimization processes like pre-training and fine-tuning. This allows a model to generate text, answer questions, and perform complex tasks autonomously without querying an external data source, though modifying, expanding, or correcting the stored information generally requires updating or retraining the underlying network parameters.
2 items
-mem: Efficient Online Memory for Large Language Models
Jingdi Lei, Di Zhang, Junxian Li, Weida Wang, Kaixuan Fan, Xiang Liu, Qihan Liu, Xiaoteng Ma, Baian Chen, Soujanya Poria
Why you should read this
Introduces -mem, a lightweight, compact online memory mechanism that significantly enhances large language models' ability to use historical information in long-term tasks through low-rank attention corrections, achieving substantial performance gains on memory-heavy benchmarks without extensive fine-tuning or context expansion.
Large language models increasingly need to accumulate and reuse historical information in long-term assistants and agent systems. Simply expanding the context window is costly and often fails to ensure effective context utilization. We propose -mem, a lightweight memory mechanism that augments a frozen full-attention backbone with a compact online state of associative memory. -mem compresses past information into a fixed-size state matrix updated by delta-rule learning, and uses its readout to generate low-rank corrections to the backbone's attention computation during generation. With only an online memory state, -mem improves the average score to that of the frozen backbone and that of the strongest non--mem memory baseline. It achieves larger gains on memory-heavy benchmarks, reaching on MemoryAgentBench and on LoCoMo, while largely preserving general capabilities. These results show that effective memory can be realized through a compact online state directly coupled with attention computation, without full fine-tuning, backbone replacement, or explicit context extension.
Added
2026-05-13
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela
Why you should read this
Proposes the definitive retrieval-augmented architecture linking a pre-trained retriever with a sequence-to-sequence generator trained end-to-end to mitigate hallucinations.
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.
Added
2026-02-14