Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

Flexible Vision Transformer

A Flexible Vision Transformer is a deep learning architecture designed to process and generate visual content across unrestricted resolutions and arbitrary aspect ratios by representing images as dynamically sized sequences of tokens rather than rigid, fixed-resolution grids. Unlike traditional vision transformers that constrain inputs to uniform dimensions through standard cropping and resizing, this framework adjusts to varying sequence lengths during both training and inference phases. This structural flexibility mitigates cropping-induced biases, supports seamless adaptation to diverse aspect ratios, and facilitates resolution generalization and extrapolation, particularly within generative modeling frameworks such as diffusion transformers.

1 item

FiT: Flexible Vision Transformer for Diffusion Model

FiT: Flexible Vision Transformer for Diffusion Model

Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, Lei Bai

OrganizationsShanghai Artificial Intelligence LaboratoryShanghai Jiao Tong UniversityTsinghua UniversityUniversity of Hong KongUniversity of Sydney

Why you should read this

Proposes a flexible vision transformer architecture that treats images as variable-length token sequences using 2D rotary positional embeddings, enabling diffusion models to generate high-fidelity images across arbitrary resolutions and aspect ratios without cropping.

Nature is infinitely resolution-free. In the context of this reality, existing diffusion models, such as Diffusion Transformers, often face challenges when processing image resolutions outside of their trained domain. To overcome this limitation, we present the Flexible Vision Transformer (FiT), a transformer architecture specifically designed for generating images with unrestricted resolutions and aspect ratios. Unlike traditional methods that perceive images as static-resolution grids, FiT conceptualizes images as sequences of dynamically-sized tokens. This perspective enables a flexible training strategy that effortlessly adapts to diverse aspect ratios during both training and inference phases, thus promoting resolution generalization and eliminating biases induced by image cropping. Enhanced by a meticulously adjusted network structure and the integration of training-free extrapolation techniques, FiT exhibits remarkable flexibility in resolution extrapolation generation. Comprehensive experiments demonstrate the exceptional performance of FiT across a broad range of resolutions. Repository available at https://github.com/whlzy/FiT.

Added

2026-09-26