Stochastic mask regularization is a training technique in discrete representation learning and vector quantization that selectively introduces randomness into the quantization process by applying a stochastic mask over latent token positions. In this approach, a random binary mask determines which latent vectors are quantized via stochastic sampling from predicted categorical distributions and which are assigned deterministically to the closest codebook embeddings. By blending deterministic selection with probabilistic sampling during model training, stochastic mask regularization helps reduce the discrepancy between deterministic training and probabilistic inference in downstream generative models, while simultaneously preventing the reconstruction objective from becoming overly corrupted by unconstrained stochastic noise.
