Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

vector approximation file

A vector approximation file is a high-dimensional indexing structure designed to accelerate similarity search by quantizing vector data into compact, bit-encoded representations. In high-dimensional spaces where traditional tree-based indexes deteriorate due to the curse of dimensionality, this method divides the data space into a grid of cells and assigns each vector a short bit string indicating its cell location. During query execution, such as nearest neighbor search, the system performs a rapid sequential scan through the small approximation file to compute lower and upper distance bounds for each item. These bounds allow the algorithm to filter out the vast majority of non-matching vectors without accessing the full dataset on disk, drastically reducing input and output operations so that only a small candidate set of original, uncompressed vectors must be retrieved and evaluated.

1 item

A Quantitative Analysis and Performance Study for Similarity-Search Methods in High-Dimensional Spaces

A Quantitative Analysis and Performance Study for Similarity-Search Methods in High-Dimensional Spaces

Roger Weber, Hans-J. Schek, Stephen Blott

OrganizationsBell LaboratoriesETH ZurichInstitute of Information Systems

Why you should read this

Proves that conventional tree-based indexing structures degenerate into linear scans beyond ten dimensions and introduces the Vector Approximation File to dramatically accelerate similarity search in high-dimensional vector spaces.

For similarity search in high-dimensional vector spaces (or ‘HDVSs’), researchers have proposed a number of new methods (or adaptations of existing methods) based, in the main, on data-space partitioning. However, the performance of these methods generally degrades as dimensionality increases. Although this phenomenon—known as the ‘dimensional curse’ is well known, little or no quantitative analysis of the phenomenon is available. In this paper, we provide a detailed analysis of partitioning and clustering techniques for similarity search in HDVSs. We show formally that these methods exhibit linear complexity at high dimensionality, and that existing methods are outperformed on average by a simple sequential scan if the number of dimensions exceeds around 10. Consequently, we come up with an alternative organization based on approximations to make the unavoidable sequential scan as fast as possible. We describe a simple vector approximation scheme, called VA-file, and report on an experimental evaluation of this and of two tree-based index methods (an R*-tree and an X-tree).

Added

2026-09-18