Back

Multi-Stage Progressive Image Restoration

Syed Waqas ZamirAditya AroraSalman KhanMunawar HayatFahad Shahbaz KhanMing-Hsuan YangLing Shao

article2021CVPR2,286 citations

Proposes MPRNet, a multi-stage progressive architecture that balances contextual representations with high-resolution spatial details using supervised attention and cross-stage feature exchange, setting strong performance benchmarks across image deblurring, deraining, and denoising.

Listen

Digital image quality often degrades due to adverse ambient conditions or hardware limitations, resulting in noise, blur, and rain streaks. Restoring clean images is essential for modern visual applications, yet algorithms face a fundamental design conflict: they must extract broad context without losing fine, high-resolution spatial details. Existing single-stage deep learning models typically excel at either contextual analysis or detail preservation, but struggle to balance both simultaneously.

The article evaluates a novel multi-stage deep learning architecture called MPRNet (Multi-Stage Progressive Image Restoration Network). The objective is to demonstrate that progressively breaking down the image restoration process across specialized, interlinked stages achieves superior reconstruction accuracy and efficiency across multiple degradation types.

The researchers developed a three-stage framework where earlier stages use multi-scale encoder-decoder networks to capture contextual information, while the final stage operates directly on the original resolution to retain fine details. Rather than simply chaining these subnetworks, the model incorporates two core mechanisms: Supervised Attention Modules that recalibrate intermediate features under ground-truth supervision, and Cross-Stage Feature Fusion to propagate multi-scale representations laterally. The approach was evaluated across ten synthetic and real-world benchmark datasets covering image deraining, motion deblurring, and photographic denoising.

Across all benchmarks, the proposed network established new state-of-the-art performance benchmarks. For image deraining, it achieved an average 20% relative error reduction over the previous leading method, while requiring 3.7 times fewer parameters and running 2.4 times faster. In motion deblurring, it outperformed previous leading approaches on synthetic datasets (reducing error by roughly 9% to 21%) and generalized effectively when applied directly to real-world blurred photographs. For image denoising on smartphone camera datasets, it consistently surpassed specialized competitors while preserving natural structural textures rather than introducing smoothing artifacts. Furthermore, ablation experiments confirmed that combining multi-scale and full-resolution stages with attention-guided feature fusion was essential for these gains.

These findings indicate that image restoration pipelines can achieve significantly higher visual fidelity and detail recovery without relying on excessively heavy or computationally expensive architectures. The multi-stage design provides practical flexibility: early stages generate rapid, lightweight predictions suitable for constrained hardware, while full execution delivers peak accuracy for demanding downstream systems. This balances computational performance with output quality in consumer hardware and automated vision tasks.

Organizations developing computer vision systems should consider adopting multi-stage progressive restoration pipelines for preprocessing degraded imagery. The release of open-source code and pre-trained models allows engineering teams to pilot this architecture directly on proprietary datasets. Future efforts should explore deployment across broader camera sensor types, edge-device optimizations, and real-time video workflows to confirm performance under dynamic, unconstrained field conditions.

GoogleInception Institute of AIMohamed bin Zayed University of Artificial IntelligenceMonash UniversityUniversity of California, MercedYonsei Universityarchitecture designsencoder-decoderimage restorationImageImage RestorationMulti-Stage
arXiv: 2102.02808swz30/MPRNet
Cover for Multi-Stage Progressive Image Restoration

Abstract

Image restoration tasks demand a complex balance between spatial details and high-level contextualized information while recovering images. In this paper, we propose a novel synergistic design that can optimally balance these competing goals. Our main proposal is a multi-stage architecture, that progressively learns restoration functions for the degraded inputs, thereby breaking down the overall recovery process into more manageable steps. Specifically, our model first learns the contextualized features using encoder-decoder architectures and later combines them with a high-resolution branch that retains local information. At each stage, we introduce a novel per-pixel adaptive design that leverages in-situ supervised attention to reweight the local features. A key ingredient in such a multi-stage architecture is the information exchange between different stages. To this end, we propose a two-faceted approach where the information is not only exchanged sequentially from early to late stages, but lateral connections between feature processing blocks also exist to avoid any loss of information. The resulting tightly interlinked multi-stage architecture, named as MPRNet, delivers strong performance gains on ten datasets across a range of tasks including image deraining, deblurring, and denoising. The source code and pre-trained models are available at this https URL.

Topics

architecture designsencoder-decoderimage restorationinformation exchangeslocal featuresource codes

Keywords

ImageImage RestorationMulti-StageStageCVPRDeblurringPSNRProgressiveMPRNetProgressive Image

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Multi-Stage Progressive Restoration
  • 3.1 Complementary Feature Processing
  • 3.2 Cross-stage Feature Fusion
  • 3.3 Supervised Attention Module
  • 4 Experiments and Analysis
  • 4.1 Datasets and Evaluation Protocol
  • 4.2 Implementation Details
  • 4.3 Image Deraining Results
  • 4.4 Image Deblurring Results
  • 4.5 Image Denoising Results
  • 4.6 Ablation Studies
  • 5 Resource Efficient Image Restoration
  • 6 Conclusion
  • References

Knowls

  1. Knowl 1 — Multi-Stage Progressive Restoration Architecture (MPRNet)

    model/method

    The Multi-Stage Progressive Restoration Network (MPRNet) is a three-stage image restoration framework designed to balance contextual representation with high-resolution spatial details.

    • Stage 1 and Stage 2 (Contextual Stages): Use modified U-Net encoder-decoder subnetworks equipped with Channel Attention Blocks (CABs) to capture broad contextual information through downsampled representations and large receptive fields. Stage 1 processes the degraded image divided into 4 non-overlapping patches, while Stage 2 processes 2 non-overlapping patches.
    • Stage 3 (High-Resolution Stage): Employs an Original Resolution Subnetwork (ORSNet) that operates directly on the full-resolution un-downsampled image to preserve fine spatial details and textural structures without downsampling artifacts.
    • Residual Learning Formulation: At each stage S∈{1,2,3}S \in \{1, 2, 3\}, rather than directly predicting the clean image, the subnetwork predicts a residual map RS∈RH×W×3R_S \in \mathbb{R}^{H \times W \times 3}, and the stage output is formed by adding the original input degraded image I∈RH×W×3I \in \mathbb{R}^{H \times W \times 3}:

    XS=I+RSX_S = I + R_S

    Information is passed between stages via two mechanisms: sequentially through Supervised Attention Modules (SAM) and laterally across subnetwork layers via Cross-Stage Feature Fusion (CSFF).

  2. Knowl 2 — Supervised Attention Module (SAM)

    model/method

    The Supervised Attention Module (SAM) is placed at the transition between successive restoration stages to dynamically refine features using intermediate image reconstructions supervised by ground truth.

    Given the output feature map Fin∈RH×W×CF_{in} \in \mathbb{R}^{H \times W \times C} from the current stage (where H×WH \times W is the spatial dimension and CC is the channel count):

    1. Residual and Image Prediction: A 1×11 \times 1 convolution generates a 3-channel residual image RS=Conv1×1(Fin)∈RH×W×3R_S = \text{Conv}_{1 \times 1}(F_{in}) \in \mathbb{R}^{H \times W \times 3}. The restored image for stage SS is obtained as XS=I+RSX_S = I + R_S, where I∈RH×W×3I \in \mathbb{R}^{H \times W \times 3} is the input degraded image. XSX_S receives direct ground-truth supervision via the multi-stage loss.
    2. Attention Map Generation: An attention map M∈RH×W×CM \in \mathbb{R}^{H \times W \times C} is computed from the restored image XSX_S using a 1×11 \times 1 convolution followed by a sigmoid activation function σ\sigma:

    M=σ(Conv1×1(XS))M = \sigma(\text{Conv}_{1 \times 1}(X_S))

    1. Feature Recalibration and Output: The transformed incoming features are scaled element-wise by the attention map and added to an identity pathway:

    Fout=(Conv1×1(Fin)⊙M)+Conv1×1(Fin)F_{out} = (\text{Conv}_{1 \times 1}(F_{in}) \odot M) + \text{Conv}_{1 \times 1}(F_{in})

    where ⊙\odot represents element-wise multiplication. The refined feature representation Fout∈RH×W×CF_{out} \in \mathbb{R}^{H \times W \times C} is forwarded to the subsequent stage.

  3. Knowl 3 — Cross-Stage Feature Fusion (CSFF)

    model/method

    Cross-Stage Feature Fusion (CSFF) provides lateral connections between intermediate feature representations of different stages in MPRNet:

    • Encoder-Decoder to Encoder-Decoder (Stage 1 to Stage 2): Intermediate multi-scale features from each level of the encoder and decoder in Stage 1 are passed through 1×11 \times 1 convolution layers and aggregated with the corresponding encoder and decoder feature maps in Stage 2.
    • Encoder-Decoder to ORSNet (Stage 2 to Stage 3): Multi-scale feature representations from Stage 2's encoder and decoder are processed with 1×11 \times 1 convolutions, upsampled or mapped to match the target scale, and merged into the Original Resolution Blocks (ORBs) of Stage 3.

    CSFF prevents the loss of multi-scale contextual information across downsampling/upsampling operations and eases gradient flow across stages, stabilizing multi-stage optimization.

  4. Knowl 4 — Multi-Stage Progressive Restoration Loss Function

    equation

    MPRNet is optimized end-to-end using a joint multi-stage loss function that penalizes reconstruction and edge discrepancies across all three stages:

    L=∑S=13[Lchar(XS,Y)+λLedge(XS,Y)]\mathcal{L} = \sum_{S=1}^{3} \left[ \mathcal{L}_{char}(X_S, Y) + \lambda \mathcal{L}_{edge}(X_S, Y) \right]

    where S∈{1,2,3}S \in \{1, 2, 3\} denotes the stage index, XS∈RH×W×3X_S \in \mathbb{R}^{H \times W \times 3} is the restored image at stage SS, Y∈RH×W×3Y \in \mathbb{R}^{H \times W \times 3} is the ground-truth clean image, and λ=0.05\lambda = 0.05 balances the loss components.

    The Charbonnier loss Lchar\mathcal{L}_{char} is defined as:

    Lchar(XS,Y)=∥XS−Y∥2+ε2\mathcal{L}_{char}(X_S, Y) = \sqrt{\|X_S - Y\|^2 + \varepsilon^2}

    and the edge loss Ledge\mathcal{L}_{edge} measures structural boundary errors using the discrete Laplacian operator Δ\Delta:

    Ledge(XS,Y)=∥Δ(XS)−Δ(Y)∥2+ε2\mathcal{L}_{edge}(X_S, Y) = \sqrt{\|\Delta(X_S) - \Delta(Y)\|^2 + \varepsilon^2}

    where ε=10−3\varepsilon = 10^{-3} is a constant penalty parameter.

  5. Knowl 5 — Subnetwork Building Blocks: CAB and ORSNet

    model/method

    The internal subnetwork components of MPRNet consist of:

    • Channel Attention Block (CAB): A feature enhancement unit containing two 3×33 \times 3 convolutional layers separated by a PReLU activation. Channel attention weights are generated via Global Average Pooling (GAP) followed by two 1×11 \times 1 convolutions with PReLU and Sigmoid activations. The input feature is scaled by the channel weights and combined with the input via an identity skip connection.
    • Encoder-Decoder Structure: Based on U-Net with 2 CABs per scale. To eliminate checkerboard artifacts typically caused by transposed convolutions, decoder upsampling is performed using bilinear upsampling followed by a standard convolution layer.
    • Original Resolution Subnetwork (ORSNet): Used in Stage 3 to retain fine spatial detail without downsampling. It comprises 3 Original Resolution Blocks (ORBs), each containing 8 consecutive CABs.

  6. Knowl 6 — Image Deraining Performance Comparison

    data/table

    Image deraining evaluation on five benchmark datasets using PSNR (dB) and SSIM computed on the Y channel (YCbCr color space). MPRNet is trained on 13,712 rain-clean image pairs from Rain14000, Rain1800, Rain800, and Rain12.

    Method Test100 Rain100H Rain100L Test2800 Test1200 Average
    PSNR / SSIM PSNR / SSIM PSNR / SSIM PSNR / SSIM PSNR / SSIM PSNR / SSIM
    DerainNet 22.77 / 0.810 14.92 / 0.592 27.03 / 0.884 24.31 / 0.861 23.38 / 0.835 22.48 / 0.796
    SEMI 22.35 / 0.788 16.56 / 0.486 25.03 / 0.842 24.43 / 0.782 26.05 / 0.822 22.88 / 0.744
    DIDMDN 22.56 / 0.818 17.35 / 0.524 25.23 / 0.741 28.13 / 0.867 29.65 / 0.901 24.58 / 0.770
    UMRL 24.41 / 0.829 26.01 / 0.832 29.18 / 0.923 29.97 / 0.905 30.55 / 0.910 28.02 / 0.880
    RESCAN 25.00 / 0.835 26.36 / 0.786 29.80 / 0.881 31.29 / 0.904 30.51 / 0.882 28.59 / 0.857
    PreNet 24.81 / 0.851 26.77 / 0.858 32.44 / 0.950 31.75 / 0.916 31.36 / 0.911 29.42 / 0.897
    MSPFN 27.50 / 0.876 28.66 / 0.860 32.40 / 0.933 32.82 / 0.930 32.39 / 0.916 30.75 / 0.903
    MPRNet (Ours) 30.27 / 0.897 30.41 / 0.890 36.40 / 0.965 33.64 / 0.938 32.91 / 0.916 32.73 / 0.921

    MPRNet achieves an average PSNR improvement of 1.98 dB over the prior state-of-the-art MSPFN (a ~20% relative error reduction), with gains up to 4.0 dB on Rain100L, while requiring 3.7×\times fewer parameters and running 2.4×\times faster.

  7. Knowl 7 — Image Deblurring Performance on Synthetic and Real-World Datasets

    data/table

    Deblurring performance measured in PSNR (dB) and SSIM on synthetic motion blur datasets (GoPro, HIDE) and real-world blur datasets (RealBlur-R and RealBlur-J under cross-dataset generalization from GoPro and direct training on RealBlur ‡\ddagger).

    Method GoPro HIDE RealBlur-R RealBlur-J
    PSNR / SSIM PSNR / SSIM Zero-shot Trained (‡\ddagger) Zero-shot Trained (‡\ddagger)
    Nah et al. 29.08 / 0.914 25.73 / 0.874 32.51 / 0.841 - 27.87 / 0.827 -
    DeblurGAN-v2 29.55 / 0.934 26.61 / 0.875 35.26 / 0.944 36.44 / 0.935 28.70 / 0.866 29.69 / 0.870
    SRN 30.26 / 0.934 28.36 / 0.915 35.66 / 0.947 38.65 / 0.965 28.56 / 0.867 31.38 / 0.909
    DBGAN 31.10 / 0.942 28.94 / 0.915 - - - -
    MT-RNN 31.15 / 0.945 29.15 / 0.918 - - - -
    DMPHN 31.20 / 0.940 29.09 / 0.924 35.70 / 0.948 - 28.42 / 0.860 -
    Suin et al. 31.85 / 0.948 29.98 / 0.930 - - - -
    MPRNet (Ours) 32.66 / 0.959 30.96 / 0.939 35.99 / 0.952 39.31 / 0.972 28.70 / 0.873 31.76 / 0.922

    When trained solely on GoPro, MPRNet gains +0.81 dB on GoPro and +0.98 dB on HIDE over Suin et al., demonstrating strong cross-dataset generalization. On real blur, MPRNet achieves the highest fidelity in both zero-shot and native domain settings.

  8. Knowl 8 — Real Image Denoising Evaluation on SIDD and DND Benchmarks

    data/table

    Real-image denoising results on the Smartphone Image Denoising Dataset (SIDD) validation set (1,280 patches) and the Darmstadt Noise Dataset (DND) benchmark (1,000 blind test patches). MPRNet is trained exclusively on 320 high-resolution images from SIDD and evaluated directly on DND without additional fine-tuning.

    Method Extra Training Data SIDD DND
    PSNR (dB) SSIM PSNR (dB) SSIM
    DnCNN No 23.66 0.583 32.43 0.790
    BM3D No 25.65 0.685 34.51 0.851
    CBDNet Yes 30.78 0.801 38.06 0.942
    RIDNet Yes 38.71 0.951 39.26 0.953
    AINDNet Yes 38.95 0.952 39.37 0.951
    VDN No 39.28 0.956 39.38 0.952
    SADNet Yes 39.46 0.957 39.59 0.952
    DANet+ Yes 39.47 0.957 39.58 0.955
    CycleISP Yes 39.52 0.957 39.56 0.956
    MPRNet (Ours) No 39.71 0.958 39.80 0.954

    MPRNet surpasses CycleISP by 0.19 dB on SIDD and SADNet by 0.21 dB on DND without requiring synthetic data pre-training.

  9. Knowl 9 — Ablation Analysis of MPRNet Architectural Components

    data/table

    Ablation study on the GoPro deblurring dataset evaluating the impact of stage numbers, subnetwork combinations, the Supervised Attention Module (SAM), and Cross-Stage Feature Fusion (CSFF). Models are trained on 128×128128 \times 128 patches for 10510^5 iterations.

    #Stages Stage Combination SAM CSFF PSNR (dB)
    1 U-Net (baseline) – – 28.94
    1 ORSNet (baseline) – – 28.91
    2 U-Net + U-Net 29.40
    2 ORSNet + ORSNet 29.53
    2 U-Net + ORSNet 29.70
    3 2 U-Nets + ORSNet 29.86
    3 2 U-Nets + ORSNet ✓ 30.07
    3 2 U-Nets + ORSNet ✓ 30.31
    3 2 U-Nets + ORSNet ✓ ✓ 30.49

    Key takeaways:

    1. Pairing multi-scale encoder-decoders with a final full-resolution ORSNet outperforms homogeneous stage designs (29.70 dB vs. 29.40 dB for U-Net+U-Net and 29.53 dB for ORSNet+ORSNet).
    2. Adding SAM increases PSNR by +0.45 dB (from 29.86 to 30.31 dB without CSFF, and from 30.07 to 30.49 dB with CSFF).
    3. Adding CSFF increases PSNR by +0.21 dB without SAM and +0.18 dB with SAM, with both modules combined yielding the peak 30.49 dB.

  10. Knowl 10 — Stage-Wise Deblurring Performance and Computational Efficiency Trade-Off

    data/table

    Inference time (seconds on Nvidia Titan Xp GPU), parameter count (Millions), and restoration quality (PSNR in dB on GoPro) across individual stages of MPRNet compared to competing methods.

    Metric DeblurGAN-v2 SRN DMPHN Suin et al. MPRNet (Ours)
    Stage-1 Stage-2 Stage-3 (Full)
    PSNR (dB) 29.55 30.10 31.20 31.85 30.43 31.81 32.66
    #Params (M) 60.9 6.8 21.7 23.0 5.6 11.3 20.1
    Time (s) 0.21 0.57 1.07 0.34 0.04 0.08 0.18

    MPRNet enables flexible progressive inference:

    • Stage 1 alone (5.6M parameters, 0.04 s) outperforms SRN (+0.33 dB) and DeblurGAN-v2 (+0.88 dB) while being 14×14\times and 5×5\times faster, respectively.
    • Stage 2 alone (11.3M parameters, 0.08 s) beats DMPHN (+0.61 dB) with ∼2×\sim 2\times fewer parameters and 13×13\times faster runtime.
    • Full 3-stage model (20.1M parameters, 0.18 s) delivers the highest accuracy (32.66 dB), outperforming Suin et al. by 0.81 dB.

Coverage note — None was omitted; all main architectural designs, mathematical formulations, experimental benchmarks across three tasks, ablation studies, and resource-efficiency analyses are covered.

References

  1. 1.Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In CVPR, 2018. 4, 5, 6, 7, 8
  2. 2.Abdelrahman Abdelhamed, Radu Timofte, and Michael S Brown. NTIRE 2019 challenge on real image denoising: Methods and results. In CVPRW, 2019. 2
  3. 3.Michal Aharon, Michael Elad, and Alfred Bruckstein. K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. Trans. Sig. Proc., 2006. 2
  4. 4.Saeed Anwar and Nick Barnes. Real image denoising with feature attention. ICCV, 2019. 1, 2, 7, 8
  5. 5.Saeed Anwar and Nick Barnes. Densely residual laplacian super-resolution. TPAMI, 2020. 2
  6. 6.Saeed Anwar, Salman Khan, and Nick Barnes. A deep journey into super-resolution: A survey. ACM Computing Surveys, 2019. 3
  7. 7.Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. Unprocessing images for learned raw denoising. In CVPR, 2019. 1, 3, 4
  8. 8.Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In CVPR, 2005. 2
  9. 9.Harold C Burger, Christian J Schuler, and Stefan Harmeling. Image denoising: Can plain neural networks compete with BM3D? In CVPR, 2012. 7
  10. 10.Tony F Chan and Chiu-Kwong Wong. Total variation blind deconvolution. TIP, 1998. 2
  11. 11.Meng Chang, Qi Li, Huajun Feng, and Zhihai Xu. Spatial-adaptive network for single image denoising. In ECCV, 2020. 6, 7, 8
  12. 12.Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud. Two deterministic half-quadratic regularization algorithms for computed imaging. In ICIP, 1994. 3
  13. 13.Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In CVPR, 2018. 1, 3
  14. 14.Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In CVPR, 2018. 1
  15. 15.Bowen Cheng, Liang-Chieh Chen, Yunchao Wei, Yukun Zhu, Zilong Huang, Jinjun Xiong, Thomas S Huang, Wen-Mei Hwu, and Honghui Shi. SPGNet: Semantic prediction guidance for scene parsing. In ICCV, 2019. 1
  16. 16.Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-D transform-domain collaborative filtering. TIP, 2007. 2, 7, 8
  17. 17.Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In CVPR, 2019. 1, 2
  18. 18.Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 2015. 3
  19. 19.Weisheng Dong, Lei Zhang, Guangming Shi, and Xiaolin Wu. Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. TIP, 2011. 1
  20. 20.Yazan Abu Farha and Jurgen Gall. MS-TCN: Multi-stage temporal convolutional network for action segmentation. In CVPR, 2019. 1
  21. 21.Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019. 2
  22. 22.Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paisley. Clearing the skies: A deep network architecture for single-image rain removal. TIP, 2017. 5, 6
  23. 23.Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via a deep detail network. In CVPR, 2017. 4, 5
  24. 24.Xueyang Fu, Borong Liang, Yue Huang, Xinghao Ding, and John Paisley. Lightweight pyramid networks for image deraining. TNNLS, 2019. 2
  25. 25.Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In CVPR, 2019. 6, 7
  26. 26.Pallabi Ghosh, Yi Yao, Larry Davis, and Ajay Divakaran. Stacked spatio-temporal graph convolutional networks for action segmentation. In WACV, 2020. 1
  27. 27.Dong Gong, Jie Yang, Lingqiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton Van Den Hengel, and Qinfeng Shi. From motion blur to motion flow: a deep learning solution for removing heterogeneous motion blur. In CVPR, 2017. 6
  28. 28.Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In CVPR, 2019. 2, 7, 8
  29. 29.Kaiming He, Jian Sun, and Xiaoou Tang. Single image haze removal using dark channel prior. TPAMI, 2010. 1
  30. 30.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 2
  31. 31.Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. In NeurIPS, 2018. 2
  32. 32.Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excitation networks. IEEE TPAMI, 2019. 2
  33. 33.Zhe Hu, Sunghyun Cho, Jue Wang, and Ming-Hsuan Yang. Deblurring low-light images with light streaks. In CVPR, 2014. 6
  34. 34.Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017. 2
  35. 35.Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. CCNet: Criss-cross attention for semantic segmentation. In ICCV, 2019. 2
  36. 36.Tae Hyun Kim, Byeongjoo Ahn, and Kyoung Mu Lee. Dynamic scene deblurring. In ICCV, 2013. 6
  37. 37.Kui Jiang, Zhongyuan Wang, Peng Yi, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. Multi-scale progressive fusion network for single image deraining. In CVPR, 2020. 2, 3, 4, 5, 6
  38. 38.Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. arXiv:2101.01169, 2021. 2
  39. 39.Kwang In Kim and Younghee Kwon. Single-image super-resolution using sparse regression and natural image prior. TPAMI, 2010. 1
  40. 40.Yoonsik Kim, Jae Woong Soh, Gu Yong Park, and Nam Ik Cho. Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. In CVPR, 2020. 7, 8
  41. 41.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014. 5
  42. 42.Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiřı Matas. DeblurGAN: Blind motion deblurring using conditional adversarial networks. In CVPR, 2018. 2, 6
  43. 43.Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. DeblurGAN-v2: Deblurring (orders-of-magnitude) faster and better. In ICCV, 2019. 1, 2, 3, 4, 6, 7, 8
  44. 44.Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, 2017. 1
  45. 45.Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming-Ming Cheng, and Juergen Gall. MS-TCN++: Multi-stage temporal convolutional network for action segmentation. TPAMI, 2020. 1
  46. 46.Wenbo Li, Zhicheng Wang, Binyi Yin, Qixiang Peng, Yuming Du, Tianzi Xiao, Gang Yu, Hongtao Lu, Yichen Wei, and Jian Sun. Rethinking on multi-stage networks for human pose estimation. arXiv:1901.00148, 2019. 1
  47. 47.Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In ECCV, 2018. 1, 2, 5, 6
  48. 48.Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown. Rain streak removal using layer priors. In CVPR, 2016. 4, 5
  49. 49.Wei Liu, Andrew Rabinovich, and Alexander C Berg. ParseNet: Looking wider to see better. arXiv:1506.04579, 2015. 4
  50. 50.Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017. 5
  51. 51.Yu Luo, Yong Xu, and Hui Ji. Removing rain from a single image via discriminative sparse coding. In ICCV, 2015. 2
  52. 52.Julien Mairal, Michael Elad, and Guillermo Sapiro. Sparse representation for color image restoration. TIP, 2007. 2
  53. 53.Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017. 1, 2, 4, 5, 6, 7, 8
  54. 54.Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV, 2016. 1
  55. 55.Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts. Distill, 2016. 3
  56. 56.Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. Blind image deblurring using dark channel prior. In CVPR, 2016. 6
  57. 57.Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In ECCV, 2020. 1, 2
  58. 58.Dongwon Park, Dong Un Kang, Jisoo Kim, and Se Young Chun. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In ECCV, 2020. 6, 7
  59. 59.Pietro Perona and Jitendra Malik. Scale-space and edge detection using anisotropic diffusion. TPAMI, 1990. 1
  60. 60.Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In CVPR, 2017. 4, 5, 6, 7, 8
  61. 61.Kuldeep Purohit and AN Rajagopalan. Region-adaptive dense network for efficient motion deblurring. In AAAI, 2020. 2
  62. 62.Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In CVPR, 2018. 1
  63. 63.Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In CVPR, 2019. 1, 2, 5, 6
  64. 64.Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV, 2020. 4, 5, 6
  65. 65.Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: convolutional networks for biomedical image segmentation. In MICCAI, 2015. 1, 2, 3
  66. 66.Stefan Roth and Michael J Black. Fields of experts: A framework for learning image priors. In CVPR, 2005. 1
  67. 67.Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 1992. 1, 2
  68. 68.Qi Shan, Jiaya Jia, and Aseem Agarwala. High-quality motion deblurring from a single image. ToG, 2008. 2
  69. 69.Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human-aware motion deblurring. In ICCV, 2019. 4, 5, 6
  70. 70.Maitreya Suin, Kuldeep Purohit, and A. N. Rajagopalan. Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In CVPR, 2020. 1, 2, 3, 4, 5, 6, 7, 8
  71. 71.Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In CVPR, 2018. 1, 2, 4, 5, 6, 7, 8
  72. 72.Chunwei Tian, Lunke Fei, Wenxian Zheng, Yong Xu, Wangmeng Zuo, and Chia-Wen Lin. Deep learning on image denoising: An overview. Neural Networks, 2020. 2
  73. 73.Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. Image super-resolution using dense skip connections. In ICCV, 2017. 1
  74. 74.Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018. 2
  75. 75.Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. ESRGAN: enhanced super-resolution generative adversarial networks. In ECCVW, 2018. 1
  76. 76.Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004. 4
  77. 77.Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, and Ying Wu. Semi-supervised transfer learning for image rain removal. In CVPR, 2019. 5, 6
  78. 78.Oliver Whyte, Josef Sivic, Andrew Zisserman, and Jean Ponce. Non-uniform deblurring for shaken images. IJCV, 2012. 6
  79. 79.Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV, 2018. 2
  80. 80.Li Xu, Shicheng Zheng, and Jiaya Jia. Unnatural l0 sparse representation for natural image deblurring. In CVPR, 2013. 2, 6
  81. 81.Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from a single image. In CVPR, 2017. 1, 4, 5
  82. 82.Rajeev Yasarla and Vishal M Patel. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In CVPR, 2019. 5, 6
  83. 83.Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016. 2
  84. 84.Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang. Variational denoising network: Toward blind noise modeling and removal. In NeurIPS, 2019. 7, 8
  85. 85.Zongsheng Yue, Qian Zhao, Lei Zhang, and Deyu Meng. Dual adversarial network: Toward real-world noise removal and noise generation. In ECCV, 2020. 7, 8
  86. 86.Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. CycleISP: Real image restoration via improved data synthesis. In CVPR, 2020. 1, 2, 6, 7, 8
  87. 87.Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and enhancement. In ECCV, 2020. 1
  88. 88.Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR, 2019. 1, 2, 3, 4, 5, 6, 7, 8
  89. 89.He Zhang and Vishal M Patel. Density-aware single image de-raining using a multi-stream dense network. In CVPR, 2018. 2, 4, 5, 6
  90. 90.He Zhang, Vishwanath Sindagi, and Vishal M Patel. Image de-raining using a conditional generative adversarial network. TCSVT, 2019. 1, 4, 5
  91. 91.Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson WH Lau, and Ming-Hsuan Yang. Dynamic scene deblurring using spatially variant recurrent neural networks. In CVPR, 2018. 1, 6
  92. 92.Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In CVPR, 2020. 1, 6, 7
  93. 93.Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. TIP, 2017. 1, 2, 3, 7
  94. 94.Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep cnn denoiser prior for image restoration. In CVPR, 2017. 1
  95. 95.Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, 2018. 1, 2, 3
  96. 96.Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention networks for image restoration. In ICLR, 2019. 1
  97. 97.Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image restoration. TPAMI, 2020. 1, 2, 3
  98. 98.Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. Psanet: Point-wise spatial attention network for scene parsing. In ECCV, 2018. 2
  99. 99.Yupei Zheng, Xin Yu, Miaomiao Liu, and Shunli Zhang. Residual multiscale based single image deraining. In BMVC, 2019. 2
  100. 100.Song Chun Zhu and David Mumford. Prior learning and gibbs reaction-diffusion. TPAMI, 1997. 1

Citation

MLA
Zamir, S. W., et al. “Multi-Stage Progressive Image Restoration”. arXiv, 2021, http://arxiv.org/abs/2102.02808v2.
APA
Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M.-H., & Shao, L. (2021). Multi-Stage Progressive Image Restoration. arXiv. http://arxiv.org/abs/2102.02808v2
Chicago
Zamir, S. W., A. Arora, S. Khan, et al. 2021. “Multi-Stage Progressive Image Restoration”. arXiv. http://arxiv.org/abs/2102.02808v2.
Harvard
Zamir, S.W. et al. (2021) “Multi-Stage Progressive Image Restoration”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2102.02808v2.
Vancouver
1. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H, Shao L (2021) Multi-Stage Progressive Image Restoration. arXiv

BibTeX

@article{zamir2021multi,
  title = {Multi-Stage Progressive Image Restoration},
  author = {Zamir, Syed Waqas and Arora, Aditya and Khan, Salman and Hayat, Munawar and Khan, Fahad Shahbaz and Yang, Ming-Hsuan and Shao, Ling},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2102.02808v2},
  eprint = {2102.02808}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE