Cross-modal discrete representation learning is a machine learning paradigm that maps continuous data from multiple distinct modalities, such as audio, vision, and text, into a shared vocabulary of discrete tokens or codebooks. By applying quantization techniques such as vector quantization, continuous multi-modal feature spaces are compressed into a common categorical structure where individual discrete codes capture shared semantic concepts across modalities. This unified discrete space facilitates fine-grained semantic alignment, cross-modal retrieval, and unsupervised concept localization, allowing models to discover correspondences between granular multi-modal elements such as visual objects, audio segments, and textual words.
