Deep Back-Projection Networks for Super-Resolution

Muhammad HarisGreg ShakhnarovichNorimichi Ukita

article2018CVPR1,501 citationsWinner (1st) of NTIRE2018 Competition (Track: x8 Bicubic Downsampling), Winner of PIRM2018 (1st on Region 2, 3rd on Region 1, and 5th on Region 3)

Proposes Deep Back-Projection Networks, an architecture that uses iterative up- and down-sampling stages with an error-feedback mechanism to capture mutual image dependencies and achieve state-of-the-art super-resolution at large scaling factors.

Listen

Generating high-resolution images from low-resolution inputs—known as image super-resolution—is critical for modern computer vision tasks across industries such as security, medical imaging, and media processing. Traditional models often struggle to reconstruct fine details, particularly at large scaling factors, or require excessive computing resources that make real-time deployment difficult.

The article evaluates Deep Back-Projection Networks (DBPN) across several architectural variants to demonstrate how iterative up- and down-sampling with error feedback enhances reconstruction quality while maintaining computational efficiency.

To establish credibility, the authors conducted empirical benchmarking using standard datasets (such as DIV2K, Set5, Set14, BSDS100, Urban100, and Manga109) across 2×, 4×, and 8× enlargement tasks. They compared DBPN variants against leading methods, including LapSRN and EDSR, by measuring visual quality metrics (peak signal-to-noise ratio and structural similarity index) and runtime performance on dedicated graphics hardware.

The analysis reveals several key findings. First, DBPN consistently matches or outperforms competing models across benchmarks, achieving notable image fidelity gains on complex datasets like Urban100 and Manga109. Second, incorporating an error feedback mechanism provides a direct performance boost, improving reconstruction quality on test datasets by 0.26 to 0.53 dB over networks without feedback. Third, the models demonstrate superior computational efficiency; for 8× enlargement, the flagship D-DBPN model processes images in roughly 0.32 seconds compared to about 1.15 seconds for EDSR, while smaller DBPN variants operate in tens of milliseconds. Finally, processing full color RGB channels directly simplifies implementation without sacrificing image quality compared to traditional single-channel luminance approaches.

These results indicate that DBPN variants offer a practical solution for operational environments requiring high-speed, high-fidelity image enlargement. Organizations can deploy smaller variants (such as DBPN-SS or DBPN-S) for low-latency, edge-computing needs or larger variants (D-DBPN) when maximum image sharpness is required at significantly lower computing costs than existing heavy architectures.

For practical implementation, technical teams should select the model variant aligned with their specific hardware and throughput constraints, prioritizing direct RGB processing to streamline pipelines. Before large-scale deployment, organizations should conduct pilot testing on domain-specific imagery, as the evaluations rely primarily on standardized benchmarks and occasionally exhibit minor visual artifacts in challenging pattern reconstructions.

Toyota Technological Institute at Chicago image degradation low resolution network architecture deep back-projection networks image super-resolution deep convolutional networks

arXiv: 1803.02735 alterzero/DBPN-Pytorch

Paper: Super-resolution from a single image, Daniel Glasner et al. (2009). Introduces the classical iterative back-projection concept for single-image super-resolution that DBPN adapts and incorporates into an end-to-end deep neural architecture.
Paper: Densely Connected Convolutional Networks, Gao Huang et al. (2017). Introduces dense feature reuse across network stages, directly motivating the Dense DBPN variant proposed in the source paper.
Paper: Learning a Deep Convolutional Network for Image Super-Resolution, Chao Dong et al. (2014). Establishes the foundational convolutional feed-forward super-resolution framework whose lack of iterative feedback DBPN directly addresses.
Paper: Enhanced Deep Residual Networks for Single Image Super-Resolution, Bee Lim et al. (2017). Provides a leading benchmark architecture and training strategy for deep residual super-resolution that DBPN compares against and seeks to improve upon.
Paper: Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution, Wei-Sheng Lai et al. (2017). Develops progressive multi-scale upsampling for super-resolution, representing an earlier key alternative to direct single-pass scaling architectures.
Paper: Accurate Image Super-Resolution Using Very Deep Convolutional Networks, Jiwon Kim et al. (2016). Demonstrates the efficacy of deep convolutional residual architectures for super-resolution, establishing core benchmark datasets and training protocols used by DBPN.
Paper: Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, Wenzhe Shi et al. (2016). Introduces efficient sub-pixel convolution upsampling in low-resolution feature space, a core upsampling mechanism in modern deep restoration pipelines.
Paper: Image Super-Resolution via Deep Recursive Residual Network, Ying Tai et al. (2017). Explores deep recursive residual architectures for image super-resolution to constrain parameters across repetitive reconstruction units.

Paper: Residual Dense Network for Image Super-Resolution, Yulun Zhang et al. (2018). Advances deep super-resolution by combining dense connections with residual learning in residual dense blocks to capture hierarchical features.
Paper: Image Super-Resolution Using Very Deep Residual Channel Attention Networks, Yulun Zhang et al. (2018). Scales super-resolution networks to hundreds of layers by integrating channel attention mechanisms with residual-in-residual structures.
Paper: Second-Order Attention Network for Single Image Super-Resolution, Tao Dai et al. (2019). Enhances feature representation beyond standard first-order attention in deep super-resolution networks by exploiting second-order feature statistics.
Paper: Multi-Stage Progressive Image Restoration, Syed Waqas Zamir et al. (2021). Extends multi-stage feedback and progressive reconstruction paradigms across multiple image restoration tasks including deblurring and denoising.
Paper: Deep Learning for Image Super-Resolution: A Survey, Zhihao Wang et al. (2019). Provides a comprehensive taxonomy and survey of deep learning architectures for super-resolution, contextualizing iterative back-projection methods like DBPN.
Paper: ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks, Xintao Wang et al. (2018). Builds on deep feature extraction backbones with relativistic adversarial objectives and perceptual losses to improve high-factor perceptual super-resolution.
Paper: Pre-Trained Image Processing Transformer, Hanting Chen et al. (2020). Explores unified Transformer architectures pre-trained at scale for diverse restoration tasks, succeeding classical CNN-based super-resolution designs.
Paper: Restormer: Efficient Transformer for High-Resolution Image Restoration, Syed Waqas Zamir et al. (2022). Develops an efficient multi-Dconv head-transposed Transformer that scales restoration to large high-resolution images across diverse degradation types.

Cover for Deep Back-Projection Networks for Super-Resolution

Abstract

The feed-forward architectures of recently proposed deep super-resolution networks learn representations of low-resolution inputs, and the non-linear mapping from those to high-resolution output. However, this approach does not fully address the mutual dependencies of low- and high-resolution images. We propose Deep Back-Projection Networks (DBPN), that exploit iterative up- and down-sampling layers, providing an error feedback mechanism for projection errors at each stage. We construct mutually-connected up- and down-sampling stages each of which represents different types of image degradation and high-resolution components. We show that extending this idea to allow concatenation of features across up- and down-sampling stages (Dense DBPN) allows us to reconstruct further improve super-resolution, yielding superior results and in particular establishing new state of the art results for large scaling factors such as 8x across multiple data sets.

Topics

image degradation low resolution network architecture image resolution super-resolution reconstruction

Keywords

deep back-projection networks image super-resolution deep convolutional networks error feedback

1 Introduction
2 Related Work
2.1 Image super-resolution using deep networks
2.2 Feedback networks
2.3 Adversarial training
2.4 Back-projection
3 Deep Back-Projection Networks
3.1 Projection units
3.2 Dense projection units
3.3 Network architecture
4 Experimental Results
4.1 Implementation and training details
4.2 Model analysis
4.3 Comparison with the-state-of-the-arts
5 Conclusion
References

Knowls

Knowl 1 — Network Architectures and Parameter Configurations of DBPN Variants

model/method

The Deep Back-Projection Network (DBPN) family includes six structural variants: DBPN-SS (Super Small), DBPN-S (Small), DBPN-M (Medium), DBPN-L (Large), D-DBPN-L (Dense DBPN-Large), and D-DBPN (Dense DBPN). Each architecture consists of initial feature extraction layers ( $\text{Feat0}$ and $\text{Feat1}$ ), iterative back-projection stages ( $T$ stages of alternating up- and down-projection units), and a final reconstruction layer. In the layer notation $\text{conv}(f, n, st, pd)$ , $f$ denotes the spatial filter size, $n$ the number of feature filters, $st$ the stride, and $pd$ the spatial padding.

Scale / Property	DBPN-SS	DBPN-S	DBPN-M	DBPN-L	D-DBPN-L	D-DBPN
Input/Output	Luminance	Luminance	Luminance	Luminance	Luminance	RGB
Feat0	$\text{conv}(3,64,1,1)$	$\text{conv}(3,128,1,1)$	$\text{conv}(3,128,1,1)$	$\text{conv}(3,128,1,1)$	$\text{conv}(3,128,1,1)$	$\text{conv}(3,256,1,1)$
Feat1	$\text{conv}(1,18,1,0)$	$\text{conv}(1,32,1,0)$	$\text{conv}(1,32,1,0)$	$\text{conv}(1,32,1,0)$	$\text{conv}(1,32,1,0)$	$\text{conv}(1,64,1,0)$
Reconstruction	$\text{conv}(1,1,1,0)$	$\text{conv}(1,1,1,0)$	$\text{conv}(1,1,1,0)$	$\text{conv}(1,1,1,0)$	$\text{conv}(1,1,1,0)$	$\text{conv}(3,3,1,1)$
BP stages ( $2\times$ )	$\text{conv}(6,18,2,2)$	$\text{conv}(6,32,2,2)$	$\text{conv}(6,32,2,2)$	$\text{conv}(6,32,2,2)$	$\text{conv}(6,32,2,2)$	$\text{conv}(6,64,2,2)$
BP stages ( $4\times$ )	$\text{conv}(8,18,4,2)$	$\text{conv}(8,32,4,2)$	$\text{conv}(8,32,4,2)$	$\text{conv}(8,32,4,2)$	$\text{conv}(8,32,4,2)$	$\text{conv}(8,64,4,2)$
BP stages ( $8\times$ )	$\text{conv}(12,18,8,2)$	$\text{conv}(12,32,8,2)$	$\text{conv}(12,32,8,2)$	$\text{conv}(12,32,8,2)$	$\text{conv}(12,32,8,2)$	$\text{conv}(12,64,8,2)$
Parameters ( $2\times$ )	$106\text{k}$	$337\text{k}$	$779\text{k}$	$1221\text{k}$	$1230\text{k}$	$5819\text{k}$
Parameters ( $4\times$ )	$188\text{k}$	$595\text{k}$	$1381\text{k}$	$2168\text{k}$	$2176\text{k}$	$10291\text{k}$
Parameters ( $8\times$ )	$421\text{k}$	$1332\text{k}$	$3101\text{k}$	$4871\text{k}$	$4879\text{k}$	$23071\text{k}$
Depth	12	12	24	36	40	52
No. of stages ( $T$ )	2	2	4	6	6	7
Dense connection	No	No	No	No	Yes	Yes

Dense connection indicates whether feature maps generated by all previous projection units are concatenated and passed into each subsequent unit.

Knowl 2 — Super-Resolution Inference Runtime Comparison

data/table

Inference time (in seconds) was evaluated across super-resolution architectures on an Nvidia TITAN X GPU ( $12\text{ GB}$ VRAM) with an input patch size of $64 \times 64$ pixels upscaled to $128 \times 128$ ( $2\times$ ), $256 \times 256$ ( $4\times$ ), and $512 \times 512$ ( $8\times$ ). Runtimes represent the average of 10 trials.

Algorithm	$2\times$ ( $128\times 128$ )	$4\times$ ( $256\times 256$ )	$8\times$ ( $512\times 512$ )
VDSR	0.02223	0.03225	0.06856
DRRN	0.25413	0.32893	N.A. (Out of Memory)
EDSR	0.85790	1.24580	1.14770
DBPN-SS	–	0.01672	0.02692
DBPN-S	–	0.02073	0.03812
DBPN-M	–	0.04511	0.08106
DBPN-L	–	0.06971	0.12635
D-DBPN	0.15331	0.19396	0.31851

DBPN-SS and DBPN-S obtain the fastest and second-fastest runtimes for $4\times$ and $8\times$ enlargement. D-DBPN runs faster than EDSR across all scaling factors ( $0.15331\text{ s}$ vs. $0.8579\text{ s}$ for $2\times$ , $0.19396\text{ s}$ vs. $1.2458\text{ s}$ for $4\times$ , and $0.31851\text{ s}$ vs. $1.1477\text{ s}$ for $8\times$ ). VDSR and DRRN only process the luminance channel and require bicubic interpolation preprocessing, which is not included in the network forward time.

Knowl 3 — Ablation of Error Feedback in DBPN-S

empirical result

Error feedback (EF) in back-projection units guides early reconstruction by mapping generated high-resolution features back to the low-resolution domain to compute a residual error signal. In the baseline setting without EF, up- and down-projection units are replaced with standard non-iterative deconvolution and convolution layers.

Evaluating DBPN-S on $4\times$ enlargement demonstrates that error feedback provides substantial quantitative gains:

Error feedback improves DBPN-S performance by $+0.53\text{ dB}$ on Set5 and $+0.26\text{ dB}$ on Set14. Even without EF, the mutual low-to-high and high-to-low projection structure outperforms SRCNN by $0.57\text{ dB}$ and FSRCNN by $0.35\text{ dB}$ on Set5.

Knowl 4 — Controlled Capacity and Dataset Comparison Between DBPN-S and LapSRN

empirical result

To control for network capacity and training dataset size, DBPN-S ( $595\text{k}$ parameters) is compared against LapSRN ( $812\text{k}$ parameters) on $4\times$ super-resolution when both models are trained on the 800-image DIV2K dataset, as well as against the original LapSRN trained on BSDS200 + T91.

	DBPN-S (DIV2K)		LapSRN (DIV2K)		LapSRN (BSDS200+T91)
Dataset	PSNR (dB)	SSIM	PSNR (dB)	SSIM	PSNR (dB)	SSIM
Set5	31.57	0.886	31.64	0.886	31.54	0.885
Set14	28.20	0.771	28.25	0.772	28.19	0.772
BSDS100	27.38	0.728	27.36	0.728	27.32	0.728
Urban100	25.49	0.762	25.37	0.759	25.21	0.756
Manga109	29.39	0.891	29.24	0.891	29.09	0.890

Despite having $26.7\%$ fewer parameters, DBPN-S outperforms LapSRN trained on the same DIV2K dataset on BSDS100 ( $+0.02\text{ dB}$ ), Urban100 ( $+0.12\text{ dB}$ , $+0.003$ SSIM), and Manga109 ( $+0.15\text{ dB}$ ).

Knowl 5 — Filter Size Selection for Back-Projection Stages

empirical result

The choice of convolutional filter spatial dimensions inside the up- and down-projection stages of D-DBPN affects $4\times$ reconstruction quality. Evaluating filter sizes of $6 \times 6$ , $8 \times 8$ , and $10 \times 10$ with striding of 4 on Set5 and Set14 benchmarks yields:

A filter size of $8 \times 8$ provides the best performance, outperforming $6 \times 6$ by $0.08\text{ dB}$ on Set5 ( $0.04\text{ dB}$ on Set14) and outperforming $10 \times 10$ by $0.09\text{ dB}$ on Set5 ( $0.03\text{ dB}$ on Set14).

Knowl 6 — Comparison of RGB and Luminance Color Processing in DBPN-L

empirical result

DBPN-L evaluated on $4\times$ image super-resolution shows negligible difference when processing all three RGB color channels directly compared to processing only the luminance (Y) channel:

Direct RGB processing matches or slightly exceeds luminance PSNR ( $+0.02\text{ dB}$ on Set5, identical on Set14) while simplifying the super-resolution pipeline by eliminating the need for separate bicubic interpolation on chrominance channels.

Knowl 7 — Convergence Characteristics of DBPN Architectures
empirical result
Across both $4\times$ and $8\times$ enlargement scales on the Set5 benchmark, DBPN variants demonstrate fast convergence. In particular, D-DBPN reaches a PSNR within 50,000 training iterations that outperforms competing baseline models (DRCN, LapSRN, DRRN, and VDSR). D-DBPN reaches asymptotic performance around $500\text{k}$ to $600\text{k}$ iterations, achieving over $32.4\text{ dB}$ on $4\times$ and over $27.1\text{ dB}$ on $8\times$ .

Coverage note — Qualitative visual comparison figures (Figures 4 through 16) displaying specific image crops for 8x super-resolution across Manga109 and Urban100 were omitted because qualitative visual images cannot be represented as standalone quantitative knowls, and their empirical conclusions are captured in the quantitative evaluation tables.

References

1.P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Con- tour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898–916, 2011. 1
2.C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2016. 1, 2
3.C. Dong, C. C. Loy, and X. Tang. Accelerating the super- resolution convolutional neural network. In European Con- ference on Computer Vision, pages 391–407. Springer, 2016. 1, 2
4.J. Kim, J. Kwon Lee, and K. Mu Lee. Accurate image super- resolution using very deep convolutional networks. In Pro- ceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 1646–1654, June 2016. 2, 4
5.W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang. Deep laplacian pyramid networks for fast and accurate super- resolution. In IEEE Conferene on Computer Vision and Pat- tern Recognition, 2017. 1, 2, 4
6.B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced deep residual networks for single image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR) Workshops, July 2017. 2, 4
7.Y. Tai, J. Yang, and X. Liu. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, 2017. 2, 4
8.R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, L. Zhang, B. Lim, S. Son, H. Kim, S. Nah, K. M. Lee, et al. Ntire 2017 challenge on single image super-resolution: Meth- ods and results. In Computer Vision and Pattern Recogni- tion Workshops (CVPRW), 2017 IEEE Conference on, pages 1110–1121. IEEE, 2017. 1
9.J. Yang, J. Wright, T. S. Huang, and Y. Ma. Image super- resolution via sparse representation. Image Processing, IEEE Transactions on, 19(11):2861–2873, 2010. 1

Citation

MLA

Haris, M., et al. “Deep Back-Projection Networks For Super-Resolution”. arXiv, 2018, http://arxiv.org/abs/1803.02735v1.

APA

Haris, M., Shakhnarovich, G., & Ukita, N. (2018). Deep Back-Projection Networks For Super-Resolution. arXiv. http://arxiv.org/abs/1803.02735v1

Chicago

Haris, M., G. Shakhnarovich, and N. Ukita. 2018. “Deep Back-Projection Networks For Super-Resolution”. arXiv. http://arxiv.org/abs/1803.02735v1.

Harvard

Haris, M., Shakhnarovich, G. and Ukita, N. (2018) “Deep Back-Projection Networks For Super-Resolution”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1803.02735v1.

Vancouver

1. Haris M, Shakhnarovich G, Ukita N (2018) Deep Back-Projection Networks For Super-Resolution. arXiv

BibTeX

@article{haris2018deep,
  title = {Deep Back-Projection Networks For Super-Resolution},
  author = {Haris, Muhammad and Shakhnarovich, Greg and Ukita, Norimichi},
  year = {2018},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1803.02735v1},
  eprint = {1803.02735}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

Model	Set5 (PSNR dB)	Set14 (PSNR dB)
SRCNN	30.49	27.61
FSRCNN	30.71	27.70
DBPN-S (Without EF)	31.06	27.95
DBPN-S (With EF)	31.59	28.21

Filter Size	Striding	Padding	Set5 PSNR (dB)	Set14 PSNR (dB)
$6 \times 6$	4	1	32.39	28.78
$8 \times 8$	4	2	32.47	28.82
$10 \times 10$	4	3	32.38	28.79

Channel Configuration	Set5 PSNR (dB)	Set14 PSNR (dB)
RGB	31.88	28.47
Luminance	31.86	28.47

Deep Back-Projection Networks for Super-Resolution

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Network Architectures and Parameter Configurations of DBPN Variants

Knowl 2 — Super-Resolution Inference Runtime Comparison

Knowl 3 — Ablation of Error Feedback in DBPN-S

Knowl 4 — Controlled Capacity and Dataset Comparison Between DBPN-S and LapSRN

Knowl 5 — Filter Size Selection for Back-Projection Stages

Knowl 6 — Comparison of RGB and Luminance Color Processing in DBPN-L

Knowl 7 — Convergence Characteristics of DBPN Architectures

References

Citation

BibTeX

Source Code

Access the Paper