This repo is built for survey paper: Arbitrary-Scale Super-Resolution via Deep Learning: A Comprehensive Survey
Paper: https://www.sciencedirect.com/science/article/pii/S1566253523003317
Super-resolution (SR) is an essential class of low-level vision tasks, which aims to improve the resolution of images or videos in computer vision. In recent years, significant progress has been made in image and video super-resolution techniques based on deep learning. Nevertheless, most of the methods only consider SR with a few integer scale factors, which limits the application of the SR techniques to real-world problems. Recently, the methods to achieve arbitrary-scale super-resolution via a single model have attracted much attention. However, there is no work to thoroughly analyze the arbitrary-scale methods based on deep learning. In this work, we present a comprehensive and systematic review of 45 existing deep learning-based methods for arbitrary-scale image and video SR. We first classify the existing SR methods according to the resolved scale factors. Furthermore, we propose an in-depth taxonomy for state-of-the-art methods based on the core problem of how to achieve arbitrary-scale super-resolution, i.e., how to perform arbitrary-scale upsampling. Moreover, the performance of existing arbitrary-scale SR methods is compared, and their advantages and limitations are analyzed. We also provide some guidance for the selection of these methods in different real-world applications. Finally, we briefly discuss the future directions of arbitrary-scale super-resolution, which shows some inspirations for the progress of subsequent works on arbitrary-scale image and video super-resolution tasks.
🌏 Citations
If our survey helps your research or work, please cite it.
The following is a BibTeX reference.
@article{liu2023arbitrary,
title={Arbitrary-scale super-resolution via deep learning: A comprehensive survey},
author={Liu, Hongying and Li, Zekun and Shang, Fanhua and Liu, Yuanyuan and Wan, Liang and Feng, Wei and Timofte, Radu},
journal={Information Fusion},
pages={102015},
year={2023},
publisher={Elsevier}
}- 2024.07.26: add 2 new methods LMF (CVPR'2024) and COZ (CVPR'2024) in taxonomy INRASU.
- 2024.07.25: add 2 new methods SAVSR (AAAI'2024) and DCGU (AAAI'2024) in in taxonomy LAASU
- 2023.12.03: add 5 new methods MoEISR (arXiv'2023), Thera (arXiv'2023), Diff-SR (arXiv'2023) and FFEINR (ChinaVis'2023) in taxonomy INRASU, SG-SR (NN'2024) in taxonomy LAASU.
- 2023.11.22: add 1 new method DuDoINet (ACM MM'2023) in taxonomy INRASU.
- 2023.10.21: add 2 new methods U-LIIF (ICIP'2023) and Dual-ArbNet (MICCAI'2023) in taxonomy INRASU.
- 2023.10.15: add 1 new method learnable interpolation (IJCAI'2023) in taxonomy LAASU.
- 2023.09.28: add 3 new methods McASSR (ICCV'2023), CuNeRF (ICCV'2023) and MoTIF (ICCV'2023) in taxonomy INRASU.
- 2023.09.08: add 2 new methods DIIF (arXiv'2023) and SVAESR (ICIP'2023) in taxonomy INRASU.
- 2023.09.14: Our paper "Arbitrary-Scale Super-Resolution via Deep Learning: A Comprehensive Survey" is accepted by Information Fusion.
Single-scale v.s. Mutil-scale v.s. Arbitrary-scale model
- The proposed scale-based taxonomy for arbitrary-scale super-resolution. Note that this taxonomy shows representative methods by scales, and some methods can achieve super-resolution scales that are not limited to their taxonomic scale. For instance, ArbSR can also achieve symmetric scales, and LTEW can achieve both asymmetric and symmetric scales.
- The proposed upsampling-based taxonomy for recent arbitrary-scale super-resolution methods.
- Timeline of the development of deep learning-based arbitrary-scale super-resolution methods.
- Implementation based on arbitrary-scale interpolation. The "
$r$ " represents an arbitrary upscaling scale. The “FEM” stands for feature extraction module.
| Paper | Model | Code | Published |
|---|---|---|---|
| Accurate Image Super-Resolution Using Very Deep Convolutional Networks | VDSR | MATLAB, PyTorch | CVPR'2016, arXiv'2015 |
| Single Image Super-Resolution: From Discrete to Continuous Scale Without Retraining | MSSR | - | ACCESS'2020 |
| A Unified Network for Arbitrary Scale Super-Resolution of Video Satellite Images | ASSR | - | TGRS'2020 |
| OverNet: Lightweight Multi-Scale Super-Resolution with Overscaling Network | OverNet | PyTorch | WACV'2021 |
- Framework of the Meta-SR.
| Paper | Model | Code | Published |
|---|---|---|---|
| Meta-SR: A Magnification-Arbitrary Network for Super-Resolution | Meta-SR | PyTorch | CVPR'2019 |
| Arbitrary Scale Super-Resolution for Brain MRI Images | Meta-SRGAN | - | AIAI'2020 |
| MIASSR: An Approach for Medical Image Arbitrary Scale Super-Resolution | MIASSR | PyTorch | arXiv'2021 |
| Second-Order Attention Network for Magnification-Arbitrary Single Image Super-Resolution | Meta-SAN | - | ICDH'2020 |
| Meta-USR: A Unified Super-Resolution Network for Multiple Degradation Parameters | Meta-USR | - | TNNLS'2020 |
| Residual scale attention network for arbitrary scale image super-resolution | RSAN | - | NEUCOM'2021 |
- An overview of ArbSR.
| Paper | Model | Code | Published |
|---|---|---|---|
| Learning A Single Network for Scale-Arbitrary Super-Resolution | ArbSR | PyTorch | ICCV'2021 |
| Bilateral Upsampling Network for Single Image Super-Resolution With Arbitrary Scaling Factors | BiSR | PyTorch | TIP'2021 |
| Learning for Unconstrained Space-Time Video Super-Resolution | USTVSRNet | - | TBC'2021 |
| Scale-arbitrary Invertible Image Downscaling | AIDN | - | arXiv'2022 |
| FaceFormer: Scale-aware Blind Face Restoration with Transformers | FaceFormer | - | arXiv'2022 |
| Deep Arbitrary-Scale Image Super-Resolution via Scale-Equivariance Pursuit | EQSR | PyTorch | CVPR'2023 |
| Update (Note: the following methods published after our survey, they are not introduced in the survey) | |||
| A Novel Learnable Interpolation Approach for Scale-Arbitrary Image Super-Resolution | Learnable Interpolation | - | IJCAI'2023 |
| An efficient multi-scale learning method for image super-resolution networks | SG-SR | - | NN'2024 |
| SAVSR: Arbitrary-Scale Video Super-Resolution via a Learned Scale-Adaptive Network (Arbitrary-Scale VSR) | SAVSR | PyTorch | AAAI'2024 |
| Arbitrary-Scale Video Super-resolution Guided by Dynamic Context (Arbitrary-Scale VSR) | DCGU | - | AAAI'2024 |
- The overall network structure of LIIF.
| Paper | Model | Code | Published |
|---|---|---|---|
| Learning Continuous Image Representation with Local Implicit Image Function | LIIF | PyTorch | CVPR'2021 |
| Paper | Model | Code | Published |
|---|---|---|---|
| Spectral bias | |||
| UltraSR: Spatial Encoding is a Missing Key for Implicit Image Function-based Arbitrary-Scale Super-Resolution | UltraSR | PyTorch (only repo, no code) | arXiv'2021 |
| Enhancing Multi-Scale Implicit Learning in Image Super-Resolution with Integrated Positional Encoding | IPE-LIIF | - | arXiv'2021 |
| Cross Transformer Network for Scale-Arbitrary Image Super-Resolution | CrossSR | - | KSEM'2022 |
| Local Texture Estimator for Implicit Representation Function | LTE | PyTorch | CVPR'2022 |
| Adaptive Local Implicit Image Function for Arbitrary-Scale Super-Resolution | A-LIIF | PyTorch | ICIP'2022 |
| Single Image Super-Resolution via a Dual Interactive Implicit Neural Network | DIINN | PyTorch | WACV'2023 |
| Recovering Realistic Details for Magnification-Arbitrary Image Super-Resolution | IPF | - | TIP'2022 |
| Photo-Realistic Continuous Image Super-Resolution with Implicit Neural Networks and Generative Adversarial Networks | CiSR-GAN | - | NLDL'2022 |
| Scale-Aware Dynamic Network for Continuous-Scale Super-Resolution (arXiv) Learning Dynamic Scale Awareness and Global Implicit Functions for Continuous-Scale Super-Resolution of Remote Sensing Images (TGRS) |
SADN | PyTorch | arXiv'2021, TGRS'2023 |
| Local Implicit Normalizing Flow for Arbitrary-Scale Image Super-Resolution | LINF | PyTorch | CVPR'2023 |
| Implicit Diffusion Models for Continuous Super-Resolution | IDM | PyTorch | CVPR'2023 |
| Super-Resolution Neural Operator | SRNO | PyTorch | CVPR'2023 |
| Flipping consistency decline | |||
| OPE-SR: Orthogonal Position Encoding for Designing a Parameter-Free Upsampling Module in Arbitrary-Scale Image Super-Resolution | OPE-SR | PyTorch | CVPR'2023 |
| Local ensemble | |||
| Cascaded Local Implicit Transformer for Arbitrary-Scale Super-Resolution | CLIT | PyTorch | CVPR'2023 |
| CiaoSR: Continuous Implicit Attention-in-Attention Network for Arbitrary-Scale Image Super-Resolution | CiaoSR | PyTorch | CVPR'2023 |
| Others | |||
| Implicit Transformer Network for Screen Content Image Continuous Super-Resolution | ITSRN | PyTorch | NeurIPS'2021 |
| ITSRN++: Stronger and Better Implicit Transformer Network for Continuous Screen Content Image Super-Resolution | ITSRN++ | - | arXiv'2022 |
| B-Spline Texture Coefficients Estimator for Screen Content Image Super-Resolution | BTC | PyTorch | CVPR'2023 |
| Learning Local Implicit Fourier Representation for Image Warping | LTEW | PyTorch | ECCV'2022 |
| Towards Bidirectional Arbitrary Image Rescaling: Joint Optimization and Cycle Idempotence | BAIRNet | - | CVPR'2022 |
| VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution (Arbitrary-Scale VSR) | VideoINR | PyTorch | CVPR'2022 |
| Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolution | EGVSR | PyTorch | CVPR'2023 |
| An Arbitrary Scale Super-Resolution Approach for 3-Dimensional Magnetic Resonance Image using Implicit Neural Representation | ArSSR | PyTorch | JBHI'2022 |
| Learning Continuous Representation of Audio for Arbitrary Scale Super Resolution | LISA | - | ICASSP'2022 |
| Update (Note: the following methods published after our survey, they are not introduced in the survey) | |||
| Dynamic Implicit Image Function for Efficient Arbitrary-Scale Image Representation | DIIF | Code (only repo, no code) | arXiv'2023 |
| Rethinking Multi-Contrast MRI Super-Resolution: Rectangle-Window Cross-Attention Transformer and Arbitrary-Scale Upsampling | McASSR | Code (only repo, no code) | ICCV'2023 |
| CuNeRF: Cube-Based Neural Radiance Field for Zero-Shot Medical Image Arbitrary-Scale Super Resolution | CuNeRF | Code (only repo, no code) | ICCV'2023 |
| MoTIF: Learning Motion Trajectories with Local Implicit Neural Functions for Continuous Space-Time Video Super-Resolution (Arbitrary-Scale VSR) | MoTIF | PyTorch | ICCV'2023 |
| Soft-IntroVAE for Continuous Latent space Image Super-Resolution | SVAESR | - | ICIP'2023 |
| Uncertainty Aware Implicit Image Function for Arbitrary-Scale Super-Resolution | U-LIIF | - | ICIP'2023 |
| Dual Arbitrary Scale Super-Resolution for Multi-Contrast MRI | Dual-ArbNet | PyTorch | MICCAI'2023 |
| DuDoINet: Dual-Domain Implicit Network for Multi-Modality MR Image Arbitrary-scale Super-Resolution | DuDoINet | - | ACM MM'2023 |
| Dissecting Arbitrary-scale Super-resolution Capability from Pre-trained Diffusion Generative Models | Diff-SR | - | arXiv'2023 |
| Efficient Model Agnostic Approach for Implicit Neural Representation Based Arbitrary-Scale Image Super-Resolution | MoEISR | - | arXiv'2023 |
| Neural Fields with Thermal Activations for Arbitrary-Scale Super-Resolution | Thera | - | arXiv'2023 |
| FFEINR: Flow Feature-Enhanced Implicit Neural Representation for Spatio-temporal Super-Resolution | FFEINR | - | ChinaVis'2023, arXiv'2023 |
| Latent Modulated Function for Computational Optimal Continuous Image Representation | LMF | PyTorch | CVPR‘2024 |
| Continuous Optical Zooming: A Benchmark for Arbitrary-Scale Image Super-Resolution in Real World | COZ | PyTorch | CVPR'2024 |
| Paper | Model | Code | Published |
|---|---|---|---|
| ASDN: A Deep Convolutional Network for Arbitrary Scale Image Super-Resolution | ASDN | PyTorch | MNA'2021 |
| SRWarp: Generalized Image Super-Resolution under Arbitrary Transformation | SRWarp | PyTorch | CVPR'2021 |
| Single Image Super-Resolution with Arbitrary Magnification Based on High-Frequency Attention Network | H2A2-SR | - | MATH'2022 |
| Progressive Image Super-Resolution via Neural Differential Equation | NODE-SR | - | ICASSP'2022 |
- The PSNR results in the cases of
$\times 4$ and$\times 2.5$ scales and the number of parameters for arbitrary-scale super-resolution methods on the B100 dataset. The name in the brackets denotes the backbone of the implementation. The horizontal axis and the vertical axis denote the PSNR results in the case of non-integer scale$\times 2.5$ and integer scale$\times 4$ , respectively, and the circle size represents the number of parameters.
- Visual comparison for symmetric integer scale SR on benchmark datasets. Moreover, we also report the PSNR and SSIM results for each method.
- Visual comparison for symmetric non-integer scale SR on the B100 dataset. Moreover, we also report the PSNR and SSIM results for each method.
