Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

Differentiable Search Index

A Differentiable Search Index is an information retrieval framework that encodes an entire collection of documents directly within the parameters of a single neural network and maps input queries directly to relevant document identifiers. Unlike traditional search systems that rely on external data structures, such as inverted indexes or vector databases for nearest-neighbor retrieval, this architecture unifies indexing, document representation, and ranking into an end-to-end trainable model. By treating retrieval as a generative sequence-to-sequence task, the network memorizes the corpus during training and generates the appropriate document IDs for input queries, enabling the full retrieval process to be optimized using standard gradient-based machine learning methods.

1 item

Transformer Memory as a Differentiable Search Index

Transformer Memory as a Differentiable Search Index

Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Prakash Gupta, Tal Schuster, William W. Cohen, Donald Metzler

OrganizationsGoogle

Why you should read this

Proposes the Differentiable Search Index, demonstrating that a single Transformer can internalize an entire text corpus within its parameters to map queries directly to document identifiers, outperforming standard dual-encoder retrieval systems.

In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.

Added

2026-09-26