keyword
position encodings
Position encodings are numerical vectors injected into machine learning models, particularly transformer-based architectures, to provide information about the sequential or spatial order of elements within an input dataset. Because standard self-attention mechanisms process input tokens or data patches simultaneously and are inherently permutation invariant, they cannot determine the order or arrangement of elements on their own. Position encodings overcome this limitation by representing absolute locations or relative distances between tokens, which are added to or integrated into token embeddings and attention calculations. These representations can be constructed using deterministic mathematical formulas, such as sinusoidal functions across multiple frequencies, or learned as trainable parameters during model training, thereby allowing neural networks to model structural relationships across text sequences, visual patches, and other multidimensional data.
2 items
Transformer in Transformer
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, Yunhe Wang
Why you should read this
Introduces the Transformer in Transformer architecture to model fine-grained interactions within local image sub-patches alongside global patch representations, improving image classification accuracy with minimal computational overhead.
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 1616) as "visual sentences" and present to further divide them into smaller patches (e.g., 44) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at this https URL, and the MindSpore code is available at this https URL.
Added
2026-09-16
Self-Attention with Relative Position Representations
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani
Why you should read this
Enhances self-attention by incorporating relative distances between tokens directly into the attention graph, fixing the rigidity of absolute positioning.
Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs.
Added
2026-02-11