Frequency-Aware Transformer for Learned Image Compression

Han LiShaohui LiWenrui DaiChenglin LiJunni ZouHongkai Xiong

article2024ICLR87 citations

Introduces a frequency-aware transformer architecture for learned image compression that captures directional image details through multiscale frequency decomposition, outperforming the VTM-12.1 standard codec by more than 13% in BD-rate across benchmark datasets.

Listen

Modern digital workflows generate massive volumes of imagery, making efficient data compression critical for cutting bandwidth costs and storage demands. While neural network-based compression methods have advanced rapidly, existing solutions rely on uniform processing windows that fail to separate fine directional details from broader background structures, leading to redundant data encoding.

The article evaluates whether integrating multiscale directional frequency analysis into transformer architectures can optimize learned image compression. Specifically, it demonstrates a new framework that decomposes natural image features across orientations and adaptively compresses them.

The researchers developed the Frequency-aware Transformer-based learned Image Compression model, which introduces anisotropic window attention to process low, high, horizontal, and vertical image frequencies simultaneously. They also incorporated a frequency-modulation feed-forward network to adjust frequency weights dynamically and a transformer-based channel-wise autoregressive model to eliminate redundancy across data channels. The system was trained on standard public datasets totaling over a million images and evaluated across three established benchmarks: Kodak, Tecnick, and CLIC.

The evaluation yielded several key findings. First, the proposed framework achieved state-of-the-art compression efficiency, reducing average bitrate by 14.5% on Kodak, 15.1% on Tecnick, and 13.0% on CLIC compared to the latest standardized benchmark codec, VTM-12.1. Second, it surpassed existing deep-learning methods in preserving fine edge details and directional textures. Third, ablation testing confirmed that directional window attention accounts for the majority of the performance gains without incurring the computational penalties of larger uniform windows. Finally, the framework maintained practical inference speeds, achieving an encoding latency of 125 milliseconds and a decoding latency of 242 milliseconds per image.

These results demonstrate that frequency-aware attention enables higher visual fidelity at significantly lower file sizes than both conventional standards and earlier neural approaches. For organizations handling large-scale image storage and transmission, adopting these methods offers substantial reductions in bandwidth expenses and infrastructure loads without compromising visual quality.

Organizations should consider evaluating frequency-aware neural compression in pilot deployments for high-throughput image storage pipelines. For decision-makers planning implementations, trade-offs between hardware encoding speed and transmission bandwidth savings should be weighed. Future engineering should expand the underlying attention mechanisms to capture diagonal directions and adapt the framework to spatial-temporal video compression.

The conclusions are supported by rigorous benchmarking across standard datasets; however, evaluations were confined to static natural images on workstation-class graphics processors. Performance under edge-device hardware constraints or with non-standard visual data remains an area for cautious validation.

Shanghai Jiao Tong University Tsinghua University different frequency encoder-decoder rate distortion performance frequency-aware transformer learned image compression transformer-based image compression

arXiv: 2310.16387 qingshi9974/ICLR2024-FTIC

Paper: ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding, Dailan He et al. (2022). Introduces state-of-the-art space-channel contextual adaptive coding for learned image compression that motivates FAT's transformer-based channel-wise autoregressive design.
Paper: Joint Autoregressive and Hierarchical Priors for Learned Image Compression, David Minnen et al. (2018). Establishes joint autoregressive and hierarchical entropy priors for learned image compression upon which modern latent context modeling relies.
Paper: Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules, Zhengxue Cheng et al. (2020). Demonstrates the integration of attention modules with flexible likelihood models to capture spatial redundancy in learned image compression.
Paper: Variational image compression with a scale hyperprior, Johannes Ballé et al. (2018). Pioneers the variational scale hyperprior architecture that serves as the foundation for modern learned image compression autoencoders.
Paper: End-to-end Optimized Image Compression, Johannes Ballé et al. (2016). Provides the seminal end-to-end rate-distortion optimization framework and continuous quantization relaxation used across neural compression models.
Paper: Cross Aggregation Transformer for Image Restoration, Zheng Chen et al. (2022). Presents directional and rectangular window aggregation mechanisms in vision transformers that directly relate to directional frequency attention.

No sufficiently relevant recommendations were found.

Cover for Frequency-Aware Transformer for Learned Image Compression

Abstract

Learned image compression (LIC) has gained traction as an effective solution for image storage and transmission in recent years. However, existing LIC methods are redundant in latent representation due to limitations in capturing anisotropic frequency components and preserving directional details. To overcome these challenges, we propose a novel frequency-aware transformer (FAT) block that for the first time achieves multiscale directional ananlysis for LIC. The FAT block comprises frequency-decomposition window attention (FDWA) modules to capture multiscale and directional frequency components of natural images. Additionally, we introduce frequency-modulation feed-forward network (FMFFN) to adaptively modulate different frequency components, improving rate-distortion performance. Furthermore, we present a transformer-based channel-wise autoregressive (T-CA) model that effectively exploits channel dependencies. Experiments show that our method achieves state-of-the-art rate-distortion performance compared to existing LIC methods, and evidently outperforms latest standardized codec VTM-12.1 by 14.5%, 15.1%, 13.0% in BD-rate on the Kodak, Tecnick, and CLIC datasets.

Topics

different frequency encoder-decoder rate distortion performance feed-forward network

Keywords

frequency-aware transformer learned image compression transformer-based image compression anisotropic window attention frequency-modulation feed-forward network frequency decomposition rate-distortion performance nonlinear transforms autoregressive entropy model

1 Introduction
2 Related Work
2.1 Transformer-based Image Compression
2.2 Autoregressive Entropy Modeling
2.3 Frequency Decomposition in Learned Image Compression
3 Methods
3.1 Overview
3.2 Frequency-aware transformer block
3.2.1 Frequency-Decomposed Window Attention
3.2.2 Frequency-Modulation Feed-forward Network
3.3 Transformer-based Channel-wise Autoregressive (T-CA) Entropy Model
4 Experiments
4.1 Experimental Setup
4.2 Rate-Distortion Performance
4.3 Ablation Studies and Analysis
5 Conclusion
References
A Model Architecture
A.1 Details of overall framework
A.2 Details of Shift-Window Operation
A.3 Details of Entropy Parameters Network
B Training
C Additional Experimental Results
C.1 R-D Performance on Tecnick and CLIC Professional Validation Datesets
C.2 Comparison on Coding Complexity
C.3 Comparison on Training Speed and GPU memory requirement
C.4 Visualization of channel attention weights in T-CA
C.4.1 Visual Quality Results
D Limitation and future work

Knowls

Knowl 1 — FTIC Framework and Rate-Distortion Objective
model/method
The Frequency-aware Transformer-based learned Image Compression (FTIC) framework constructs nonlinear analysis and synthesis transforms using Frequency-Aware Transformer (FAT) blocks to capture multiscale and directional frequency components. Given an input raw image $x$ , the analysis transform $g_a(\cdot)$ extracts a latent representation $y = g_a(x)$ . A uniform quantizer $Q(\cdot)$ discretizes $y$ into quantized latent $\hat{y} = Q(y)$ .

To compress $\hat{y}$ , a hyperprior path extracts hyper-latents $z = h_a(y)$ , which are quantized to $\hat{z} = Q(z)$ , and decoded by the hyper-decoder to yield context $\phi = h_s(\hat{z})$ . The quantized latent $\hat{y}$ is divided along the channel dimension into $n_s$ slices $\{\hat{y}_1, \hat{y}_2, \dots, \hat{y}_{n_s}\}$ . A Transformer-based Channel-wise Autoregressive (T-CA) entropy model predicts Gaussian distribution parameters (mean $\mu_i$ , scale $\sigma_i$ ) and a latent quantization residual $r_i$ for each slice $i \in \{1, \dots, n_s\}$ conditioned on previously decoded slices $\hat{y}_{<i}$ and hyperprior context $\phi$ :

$r_i, \mu_i, \sigma_i = \text{T-CA}(\phi, \hat{y}_{<i}), \quad 1 \le i \le n_s$

A refined latent representation $\bar{y}$ is formed by adding the accumulated residual $r = \text{Concat}(r_1, r_2, \dots, r_{n_s})$ to $\hat{y}$ :

$\bar{y} = \hat{y} + r$

The reconstructed image $\hat{x}$ is synthesized via $\hat{x} = g_s(\bar{y})$ . The model is optimized end-to-end via a rate-distortion Lagrangian loss:

$\mathcal{L} = R(\hat{y}) + R(\hat{z}) + \lambda \cdot D(x, \hat{x})$

where $R(\hat{y})$ and $R(\hat{z})$ denote the estimated bitrate for $\hat{y}$ and $\hat{z}$ , $D(x, \hat{x})$ denotes distortion measured by Mean Squared Error (MSE) or Multiscale Structural Similarity (MS-SSIM), and $\lambda$ controls the rate-distortion trade-off.
Knowl 2 — Frequency-Decomposed Window Attention
model/method
Frequency-Decomposition Window Attention (FDWA) decomposes hidden representations into multiscale isotropic and anisotropic directional frequency components within a single attention layer. Standard self-attention acts as a low-pass filter, where smaller windows extract fine-grained high frequencies and larger windows extract low frequencies. Square windows, however, are isotropic and fail to capture directional orientations.

Given an intermediate feature tensor $X \in \mathbb{R}^{C \times H \times W}$ with $C$ channels and spatial dimensions $H \times W$ , the linear projection maps $X$ into $K$ attention heads. The $K$ heads are partitioned into four equal groups of $K/4$ heads, each executing local window self-attention using a distinct window geometry (height $\times$ width) based on a base scale parameter $s$ :
1. Low-Frequency Window Attention (LL-WA): Window size $4s \times 4s$ , capturing low-frequency isotropic features.
2. High-Frequency Window Attention (HH-WA): Window size $s \times s$ , capturing fine-grained high-frequency isotropic details.
3. Vertical Frequency Window Attention (HL-WA): Window size $s \times 4s$ , capturing vertically oriented high frequencies.
4. Horizontal Frequency Window Attention (LH-WA): Window size $4s \times s$ , capturing horizontally oriented high frequencies.
For a specific window attention type (e.g., LH-WA), the feature $X$ is partitioned into $M = \frac{H \times W}{4s \times s}$ non-overlapping windows $[X^1, \dots, X^M]$ with $X^i \in \mathbb{R}^{C \times 4s \times s}$ . For head $k$ with head dimension $d_k = C/K$ and projection matrices $W_k^Q, W_k^K, W_k^V \in \mathbb{R}^{C \times d_k}$ :

$Y_k^i = \text{Attention}(X^i W_k^Q, X^i W_k^K, X^i W_k^V)$

$\text{LH-WA}_k(X) = [Y_k^1, Y_k^2, \dots, Y_k^M]$

The outputs of all $K$ heads across the four groups are concatenated and linearly projected via $W^O \in \mathbb{R}^{C \times C}$ to facilitate cross-frequency interaction:

$\text{FDWA}(X) = \text{Concat}[\text{head}_1, \dots, \text{head}_K] W^O$

$\text{head}_k = \begin{cases} \text{LL-WA}_k(X), & 1 \le k \le K/4 \\ \text{HH-WA}_k(X), & K/4 < k \le K/2 \\ \text{HL-WA}_k(X), & K/2 < k \le 3K/4 \\ \text{LH-WA}_k(X), & 3K/4 < k \le K \end{cases}$

In consecutive FAT blocks, the first block applies regular FDWA and the second applies shifted-window FDWA, where windows are shifted by half their respective height and width dimensions.
Knowl 3 — Frequency-Modulation Feed-Forward Network
model/method
The Frequency-Modulation Feed-Forward Network (FMFFN) adaptively modulates and filters frequency components produced by self-attention, dynamically amplifying or suppressing specific subbands according to target bitrates.

Given an input feature representation $X$ , FMFFN first applies standard pointwise convolutions with a GELU activation to produce an expanded intermediate feature $X_{\text{ffn}}$ :

$X_{\text{ffn}} = \text{Conv}_{1\times 1}(\text{GELU}(\text{Conv}_{1\times 1}(X)))$

To make frequency modulation independent of varying input image resolutions, a spatial block partitioning operator $B(\cdot)$ partitions $X_{\text{ffn}}$ into non-overlapping blocks of size $4s \times 4s$ (matching the maximum window size in FDWA). A 2D Fast Fourier Transform $\mathcal{F}(\cdot)$ converts each block into the frequency domain, where it undergoes element-wise multiplication $\odot$ with a learnable filter tensor $W$ of fixed size $4s \times 4s$ across channels:

$X_{\text{fm}} = \mathcal{F}(B(X_{\text{ffn}})) \odot W$

The modulated frequency representation $X_{\text{fm}}$ is converted back to the spatial domain using 2D Inverse Fast Fourier Transform $\mathcal{F}^{-1}(\cdot)$ , and non-overlapping blocks are merged back to the original feature map shape via $B^{-1}(\cdot)$ :

$X_{\text{out}} = B^{-1}\left(\mathcal{F}^{-1}(X_{\text{fm}})\right)$
Knowl 4 — Transformer-Based Channel-Wise Autoregressive Entropy Model
model/method
The Transformer-based Channel-wise Autoregressive (T-CA) entropy model captures dynamic inter-slice and intra-slice channel correlations using causal masked channel self-attention rather than fixed-weight CNNs.

The quantized latent representation $\hat{y} \in \mathbb{R}^{M \times H \times W}$ is partitioned along channels into $n_s$ slices $\{\hat{y}_1, \dots, \hat{y}_{n_s}\}$ , where each slice contains $M_s = M / n_s$ channels. Each slice is linearly expanded to $r \cdot M_s$ channels (where $r > 1$ is a projection ratio) using a $1 \times 1$ group convolution with $n_g = n_s$ groups.

The transformed slices are processed through $L$ consecutive transformer layers structured as follows:
1. Masked-Slice Channel Attention: Self-attention is performed across the channel dimension ( $C = r \cdot M$ ). To enforce causal decoding, an upper-triangular slice-wise mask matrix $M_{\text{mask}} \in \mathbb{R}^{C \times C}$ is applied to the channel attention maps, preventing tokens in subsequent slices from attending to future latent slices. A learnable pseudo start slice $y_s$ provides context for the initial slice $\hat{y}_1$ .
2. Group-wise Operations: Standard LayerNorm and linear projections are replaced with GroupNorm and $1 \times 1$ group convolutions with $n_g = n_s$ groups, isolating channel mixing per slice and reducing computational parameter overhead.
The final layer output $y_{\text{out}} \in \mathbb{R}^{4M \times H \times W}$ is reshaped to $\mathbb{R}^{M \times 4 \times H \times W}$ and concatenated with the reshaped hyperprior context $\phi \in \mathbb{R}^{M \times 2 \times H \times W}$ to form $y_{\text{concat}} \in \mathbb{R}^{6M \times H \times W}$ . This concatenated tensor is passed into an Entropy Parameters Network composed of three $3 \times 3$ group convolutional layers ( $n_g = n_s$ ) with GELU activations to output the Gaussian distribution parameters $\mu_i, \sigma_i$ and quantization residual $r_i$ for each slice $i$ .
Knowl 5 — Two-Stage Training Strategy for Autoregressive Learned Image Compression
algorithm
In single-Gaussian LIC models, encoding $\lceil y - \mu \rfloor$ rather than $\lceil y \rfloor$ provides performance advantages. However, in autoregressive models, predicting $\mu_i$ requires prior quantized latent slices $\hat{y}_{<i}$ , leading to train-test mismatch if trained directly on centered representations. FTIC resolves this via a two-stage training strategy.
Input: Training datasets Flickr2W and ImageNet-1k, Adam optimizer, loss function weights $\lambda$
Output: Fully converged FTIC model parameters
# Stage 1: Pretraining transforms with hyperprior only
Initialize analysis transform $g_a$ , synthesis transform $g_s$ , hyper-transforms $h_a, h_s$
Set learning rate $\eta = 10^{-4}$ , crop size $= 256 \times 256$ , batch size $= 8$
for step $= 1$ to $2{,}000{,}000$ do
    Sample batch $x$
    Compute latents $y = g_a(x)$ , $z = h_a(y)$ , $\hat{z} = Q(z)$ , $\phi = h_s(\hat{z})$
    Predict mean $\mu$ and scale $\sigma$ solely from hyperprior context $\phi$
    Encode symbols as $\lceil y - \mu \rfloor$ and restore as $\hat{y} = \lceil y - \mu \rfloor + \mu$
    Compute reconstructed image $\hat{x} = g_s(\hat{y})$
    Update $g_a, g_s, h_a, h_s$ by minimizing $R(\hat{y}) + R(\hat{z}) + \lambda D(x, \hat{x})$
end for
# Stage 2: Joint autoregressive fine-tuning
Load trained weights of $g_a, g_s, h_a, h_s$ from Stage 1
Initialize T-CA entropy model randomly
Set learning rate $\eta = 10^{-4}$ , crop size $= 256 \times 256$ , batch size $= 8$
for step $= 1$ to $1{,}000{,}000$ do
    Sample batch $x$
    Compute latents $y = g_a(x)$ , $\hat{y} = Q(y)$ , $z = h_a(y)$ , $\hat{z} = Q(z)$ , $\phi = h_s(\hat{z})$
    Predict slice parameters $\mu_i, \sigma_i, r_i = \text{T-CA}(\phi, \hat{y}_{<i})$
    Directly encode $\lceil y \rfloor$ without mean subtraction; restore $\bar{y} = \hat{y} + r$
    Compute reconstructed image $\hat{x} = g_s(\bar{y})$
    Update all parameters by minimizing $R(\hat{y}) + R(\hat{z}) + \lambda D(x, \hat{x})$
end for
# Stage 2 Fine-tuning on larger crop size
Set learning rate $\eta = 10^{-5}$ , crop size $= 384 \times 384$
for step $= 1$ to $200{,}000$ do
    Execute joint autoregressive optimization step with direct $\lceil y \rfloor$ encoding
end for
Knowl 6 — Rate-Distortion Compression Performance Across Standard Benchmarks
empirical result
The FTIC compression model was evaluated on three standard benchmark datasets: Kodak (24 images of resolution $768 \times 512$ ), Tecnick (100 images of resolution $1200 \times 1200$ ), and the CLIC Professional Validation dataset (41 images up to 2K resolution). Performance was assessed using PSNR and MS-SSIM against bitrates measured in bits per pixel (BPP), with BD-rate computed relative to the standardized video codec VVC test model anchor VTM-12.1 (where negative BD-rate indicates bitrate savings at equal reconstruction quality).

FTIC achieved state-of-the-art BD-rate improvements over VTM-12.1 across all evaluated datasets:
- Kodak dataset: $-14.5\%$ BD-rate saving relative to VTM-12.1.
- Tecnick dataset: $-15.1\%$ BD-rate saving relative to VTM-12.1.
- CLIC Professional Validation dataset: $-13.0\%$ BD-rate saving relative to VTM-12.1.
FTIC consistently outperformed prior learned image compression methods across all operational bitrates, including transformer-based architectures (Liu et al., 2023; Zou et al., 2022; Zhu et al., 2022) and CNN-based autoregressive models (He et al., 2022; Cheng et al., 2020; Minnen et al., 2018).

Knowl 7 — Ablation Analysis of FAT Block Components

data/table

Ablation studies on the Kodak dataset evaluated the architectural contributions of the Frequency-Decomposition Window Attention (FDWA) and Frequency-Modulation Feed-Forward Network (FMFFN) against a baseline utilizing standard $4 \times 4$ isotropic window self-attention with a conventional FFN. Computational complexity was quantified by Multiply-Accumulate Operations (GMACs) for the analysis transform $g_a(\cdot)$ , overall parameter counts, and BD-rate savings relative to VTM-12.1.

Methods	Larger Window	FDWA	FMFFN	GMACs	#Params	BD-rate
Baseline				77.3	70.29M	-9.3%
Variant 1	✓			91.3	70.50M	-11.2%
Variant 2		✓		82.2	70.36M	-13.3%
Variant 3 (FAT Block)		✓	✓	83.1	70.97M	-14.5%
VTM-12.1	-	-	-	-	-	0%

Expanding window size to $16 \times 16$ (Variant 1) improved BD-rate by $1.9\%$ over Baseline but increased GMACs by $18.1\%$ . Introducing FDWA (Variant 2) achieved a larger gain of $4.0\%$ BD-rate saving over Baseline with only a modest GMAC increase (82.2 vs. 77.3 GMACs), demonstrating that frequency-decomposed isotropic and anisotropic window geometry is more effective than scaling isotropic receptive fields. Adding FMFFN (Variant 3) yielded an additional $1.2\%$ BD-rate improvement (-14.5% total) with negligible parameter and computational increase.

Knowl 8 — Parameter and Architectural Ablation of T-CA Entropy Model

data/table

The Transformer-based Channel-wise Autoregressive (T-CA) entropy model was evaluated against the CNN-based channel-wise autoregressive baseline CHARM (Minnen & Singh, 2020) and across various parameterizations of transformer layers $L$ and slice counts $n_s$ . Models were evaluated on the Kodak dataset using pretrained transforms from mbt2018-mean and fine-tuned for 1M batches, with BPG codec used as the BD-rate anchor.

$L$	$n_s$	#Params	BD-rate
4	5	14.5M	-17.6%
8	5	22.4M	-18.3%
12	5	30.4M	-19.2%
16	5	38.4M	-19.0%
12	4	37.8M	-19.0%
12	5	30.4M	-19.2%
12	8	19.3M	-18.0%
12	10	15.6M	-18.4%
BPG	-	-	0%

T-CA with 5 slices outperformed 10-slice CHARM by $3.3\%$ in BD-rate while requiring fewer parameters (30.4M vs 34.8M). Varying the number of transformer layers showed that performance peaked at $L=12$ (-19.2%), with $L=16$ providing no additional benefit. Unlike CHARM, where parameter counts grow linearly with slice count $n_s$ , increasing $n_s$ in T-CA reduces the parameter count of group convolutions and the overall entropy model due to shared transformer weights, with $n_s=5$ achieving optimal rate-distortion efficiency.

Knowl 9 — Coding Complexity and Latency Benchmark

data/table

Computational complexity and inference latency were benchmarked on the Kodak dataset using an NVIDIA GeForce RTX 4090 GPU (24 GB memory). Inference latency represents model forward GPU runtime excluding CPU entropy arithmetic encoding/decoding. BD-rate is calculated against the VTM-12.1 anchor.

Model	GMACs		Inference Latency (ms)		#Params	BD-rate
Model	Enc.	Dec.	Enc.	Dec.	#Params	BD-rate
Cheng et al. (2020)	154	229	>1000	>1000	26.60M	3.6%
Minnen Singh (2020)	101	100	56	43	55.13M	1.1%
Zhu et al. (2022)	116	116	110	99	32.71M	-3.3%
Zou et al. (2022)	143	161	97	101	99.58M	-4.3%
Liu et al. (2023)	317	453	255	322	76.57M	-11.9%
Ours (FTIC)	141	349	125	242	70.97M	-14.5%
VTM-12.1	-	-	-	-	-	0%

While spatial autoregressive methods (Cheng et al., 2020) suffer from latencies exceeding 1000 ms, FTIC achieves the highest compression performance (-14.5% BD-rate) with an encoding latency of 125 ms and decoding latency of 242 ms, offering substantially lower computational complexity (141 Enc / 349 Dec GMACs vs 317 Enc / 453 Dec GMACs) and faster inference than prior SOTA transformer-CNN hybrid models (Liu et al., 2023).

Knowl 10 — Limitations of Directional Decomposition and Scope in FTIC
limitation
The Frequency-Aware Transformer architecture exhibits two primary limitations:
1. Restricted Directional Decompositions: The Frequency-Decomposition Window Attention (FDWA) module is restricted to two orthogonal directions—horizontal ( $4s \times s$ ) and vertical ( $s \times 4s$ ). It lacks support for diagonal, curvilinear, or arbitrary angular directional frequency subbands, which limits its ability to fully capture complex directional textures compared to general multiscale geometric analysis frameworks like scattering transforms.
2. Scope Restricted to Still Image Compression: The FTIC formulation is designed exclusively for 2D spatial image compression and does not incorporate temporal frequency decomposition or inter-frame motion compensation mechanisms required for video compression.

Coverage note — None was omitted; all key contributions—including the FTIC overall architecture, FDWA, FMFFN, T-CA entropy model, two-stage training scheme, rate-distortion benchmarks, detailed ablations, and stated limitations—are fully covered.

References

1.Nicola Asuni and Andrea Giachetti. Testimages: a large-scale archive for testing visual devices and basic image processing algorithms. In STAG, pp. 63–70, 2014.
2.Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In Proceedings of the International Conference on Learning Representations, 2018.
3.Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029, 2020.
4.Gisle Bjontegaard. Calculation of average psnr differences between rd-curves. In VCEG-M33, 2001.
5.Emmanuel Jean Candès and David Leigh Donoho. Curvelets: A surprisingly effective nonadaptive representation for objects with edges. Vanderbilt Univ. Press, 2000.
6.Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213–229. Springer, 2020.
7.Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7939–7948, 2020.
8.CLIC. Workshop and challenge on learned image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
9.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. pp. 248–255, 2009.
10.Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu, Qingyi Chen, Jue Wang, and Hongkai Xiong. Motion-aware contrastive video representation learning via foregroundbackground merging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9716–9726, 2022a.
11.Shuangrui Ding, Rui Qian, and Hongkai Xiong. Dual contrastive learning for spatio-temporal representation. In Proceedings of the 30th ACM international conference on multimedia, pp. 5649–5658, 2022b.
12.M.N. Do and M. Vetterli. The contourlet transform: an efficient directional multiresolution image representation. IEEE Transactions on Image Processing, 14(12):2091–2106, 2005.
13.David L. Donoho and Xiaoming Huo. Beamlets and multiscale image analysis. In Multiscale and Multiresolution Methods, pp. 149–196, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg.
14.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
15.Haisheng Fu, Feng Liang, Jie Liang, Binglin Li, Guohe Zhang, and Jingning Han. Asymmetric learned image compression with multi-scale residual block, importance scaling, and postquantization filtering. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
16.Ge Gao, Pei You, Rong Pan, Shunyuan Han, Yuanyuan Zhang, Yuchao Dai, and Hojae Lee. Neural image compression via attentional multi-scale back projection and frequency decomposition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14677–14686, 2021.
17.Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
18.Eastman Kodak. Kodak lossless true color image suite (photocd pcd0992). 1993.
19.A Burakhan Koyuncu, Han Gao, and Eckehard Steinbach. Contextformer: A transformer with spatio-channel attention for context modeling in learned image compression. In European Conference on Computer Vision, 2022.
20.Han Li, Bowen Shi, Wenrui Dai, Hongwei Zheng, Botao Wang, Yu Sun, Min Guo, Chenglin Li, Junni Zou, and Hongkai Xiong. Pose-oriented transformer with uncertainty-guided refinement for 2d-to-3d human pose estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1296–1304, 2023.
21.Jiaheng Liu, Guo Lu, Zhihao Hu, and Dong Xu. A unified end-to-end framework for efficient deep image compression. arXiv preprint arXiv:2002.03370, 2020.
22.Jinming Liu, Heming Sun, and Jiro Katto. Learned image compression with mixed transformercnn architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14388–14397, 2023.
23.Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022, 2021.
24.Ming Lu, Peiyao Guo, Huiqing Shi, Chuntong Cao, and Zhan Ma. Transformer-based image compression. In 2022 Data Compression Conference (DCC), pp. 469–469. IEEE, 2022.
25.Haichuan Ma, Dong Liu, Ruiqin Xiong, and Feng Wu. iwave: Cnn-based wavelet-like transform for image compression. IEEE Transactions on Multimedia, 22(7):1667–1679, 2019.
26.Haichuan Ma, Dong Liu, Ning Yan, Houqiang Li, and Feng Wu. End-to-end optimized versatile image compression with wavelet-like transform. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1247–1263, 2020.
27.Stephane G Mallat. A theory for multiresolution signal decomposition: the wavelet representation. IEEE transactions on pattern analysis and machine intelligence, 11(7):674–693, 1989.
28.Fabian Mentzer, George Toderici, David Minnen, Sung-Jin Hwang, Sergi Caelles, Mario Lucic, and Eirikur Agustsson. Vct: A video compression transformer. arXiv preprint arXiv:2206.07307, 2022.
29.David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned image compression. In IEEE International Conference on Image Processing (ICIP), pp. 3339–3343. IEEE, 2020.
30.David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
31.Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention. Advances in Neural Information Processing Systems, 35:14541–14554, 2022.
32.Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=IDwN6xjHnK8.
33.Badri Patro and Vijay Agneeswaran. Scattering vision transformer: Spectral mixing matters. Advances in Neural Information Processing Systems, 36, 2024.
34.Yichen Qian, Xiuyu Sun, Ming Lin, Zhiyu Tan, and Rong Jin. Entroformer: A transformer-based entropy model for learned image compression. In International Conference on Learning Representations, 2022.
35.Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. Advances in neural information processing systems, 34:980–993, 2021.
36.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
37.Yueqi Xie, Ka Leong Cheng, and Qifeng Chen. Enhanced invertible encoding for learned image compression. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 162–170, 2021.
38.Ali Zafari, Atefeh Khoshkhahtinat, Piyush Mehta, Mohammad Saeed Ebrahimi Saadabadi, Mohammad Akyash, and Nasser M Nasrabadi. Frequency disentangled features in neural image compression. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 2815–2819. IEEE, 2023.
39.Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-based transform coding. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=IDwN6xjHnK8.
40.Renjie Zou, Chunfeng Song, and Zhaoxiang Zhang. The devil is in the details: Window-based attention for image compression. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022.

Citation

MLA

Li, H., et al. “Frequency-Aware Transformer for Learned Image Compression”. arXiv, 2023, http://arxiv.org/abs/2310.16387v4.

APA

Li, H., Li, S., Dai, W., Li, C., Zou, J., & Xiong, H. (2023). Frequency-Aware Transformer for Learned Image Compression. arXiv. http://arxiv.org/abs/2310.16387v4

Chicago

Li, H., S. Li, W. Dai, C. Li, J. Zou, and H. Xiong. 2023. “Frequency-Aware Transformer for Learned Image Compression”. arXiv. http://arxiv.org/abs/2310.16387v4.

Harvard

Li, H. et al. (2023) “Frequency-Aware Transformer for Learned Image Compression”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2310.16387v4.

Vancouver

1. Li H, Li S, Dai W, Li C, Zou J, Xiong H (2023) Frequency-Aware Transformer for Learned Image Compression. arXiv

BibTeX

@article{li2023frequency,
  title = {Frequency-Aware Transformer for Learned Image Compression},
  author = {Li, Han and Li, Shaohui and Dai, Wenrui and Li, Chenglin and Zou, Junni and Xiong, Hongkai},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2310.16387v4},
  eprint = {2310.16387}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Frequency-Aware Transformer for Learned Image Compression

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — FTIC Framework and Rate-Distortion Objective

Knowl 2 — Frequency-Decomposed Window Attention

Knowl 3 — Frequency-Modulation Feed-Forward Network

Knowl 4 — Transformer-Based Channel-Wise Autoregressive Entropy Model

Knowl 5 — Two-Stage Training Strategy for Autoregressive Learned Image Compression

Knowl 6 — Rate-Distortion Compression Performance Across Standard Benchmarks

Knowl 7 — Ablation Analysis of FAT Block Components

Knowl 8 — Parameter and Architectural Ablation of T-CA Entropy Model

Knowl 9 — Coding Complexity and Latency Benchmark

Knowl 10 — Limitations of Directional Decomposition and Scope in FTIC

References

Citation

BibTeX

Source Code

Access the Paper