Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

FiT architecture

The FiT architecture, short for Flexible Vision Transformer, is a neural network design for diffusion-based image generation that accommodates unrestricted image resolutions and aspect ratios. Rather than treating images as static grids of a fixed size, this architecture conceptualizes visual inputs as sequences of dynamically sized tokens. By managing variable-length token sequences using specialized attention masking and multidimensional positional encodings, it allows models to train and perform inference across diverse image dimensions without the need for fixed cropping or standard resizing. This design eliminates cropping-induced spatial biases and improves the network's ability to extrapolate and generate high-quality visual content beyond its original training resolution.

1 item

FiT: Flexible Vision Transformer for Diffusion Model

FiT: Flexible Vision Transformer for Diffusion Model

Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, Lei Bai

OrganizationsShanghai Artificial Intelligence LaboratoryShanghai Jiao Tong UniversityTsinghua UniversityUniversity of Hong KongUniversity of Sydney

Why you should read this

Proposes a flexible vision transformer architecture that treats images as variable-length token sequences using 2D rotary positional embeddings, enabling diffusion models to generate high-fidelity images across arbitrary resolutions and aspect ratios without cropping.

Nature is infinitely resolution-free. In the context of this reality, existing diffusion models, such as Diffusion Transformers, often face challenges when processing image resolutions outside of their trained domain. To overcome this limitation, we present the Flexible Vision Transformer (FiT), a transformer architecture specifically designed for generating images with unrestricted resolutions and aspect ratios. Unlike traditional methods that perceive images as static-resolution grids, FiT conceptualizes images as sequences of dynamically-sized tokens. This perspective enables a flexible training strategy that effortlessly adapts to diverse aspect ratios during both training and inference phases, thus promoting resolution generalization and eliminating biases induced by image cropping. Enhanced by a meticulously adjusted network structure and the integration of training-free extrapolation techniques, FiT exhibits remarkable flexibility in resolution extrapolation generation. Comprehensive experiments demonstrate the exceptional performance of FiT across a broad range of resolutions. Repository available at https://github.com/whlzy/FiT.

Added

2026-09-26