Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

CrossFormer

CrossFormer is a vision transformer architecture designed to capture and integrate visual features across multiple spatial scales for computer vision applications. Unlike conventional vision transformers that rely on uniform-scale token embeddings or discard fine-grained details to reduce computational costs, CrossFormer uses a cross-scale embedding layer that blends image patches of different sizes into unified representations. It processes these multi-scale features using a long-short distance attention mechanism, which divides self-attention into local interactions for fine details and long-range interactions for broader context. Coupled with a dynamic position bias that allows the model to flexibly adapt to variable input image dimensions, the architecture serves as a versatile backbone for tasks such as image classification, object detection, and semantic segmentation.

1 item

CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention

CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention

Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, Wei Liu

OrganizationsColumbia UniversityTencentZhejiang University

Why you should read this

Proposes CrossFormer, a vision transformer that establishes cross-scale feature interactions using multi-scale patch embeddings and long-short distance attention to achieve superior performance across image classification, object detection, and segmentation benchmarks.

Transformers have made great progress in dealing with computer vision tasks. However, existing vision transformers do not yet possess the ability of building the interactions among features of different scales, which is perceptually important to visual inputs. The reasons are two-fold: (1) Input embeddings of each layer are equal-scale, so no cross-scale feature can be extracted; (2) to lower the computational cost, some vision transformers merge adjacent embeddings inside the self-attention module, thus sacrificing small-scale (fine-grained) features of the embeddings and also disabling the cross-scale interactions. To this end, we propose Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). On the one hand, CEL blends each embedding with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the embeddings. Through the above two designs, we achieve cross-scale attention. Besides, we put forward a dynamic position bias for vision transformers to make the popular relative position bias apply to variable-sized images. Hinging on the cross-scale attention module, we construct a versatile vision architecture, dubbed CrossFormer, which accommodates variable-sized inputs. Extensive experiments show that CrossFormer outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks. The code has been released: this https URL.

Added

2026-09-26