Adaptive Transform Domain Image Super-Resolution via Orthogonally Regularized Deep Networks

Tiantong Guo, Hojjat Seyed Mousavi, Vishal Monga

Research output: Contribution to journalArticle

Abstract

Deep learning methods, in particular, trained convolutional neural networks (CNNs) have recently been shown to produce compelling results for single image super-resolution (SR). Invariably, a CNN is learned to map the low resolution (LR) image to its corresponding high resolution (HR) version in the spatial domain. We propose a novel network structure for learning the SR mapping function in an image transform domain, specifically the discrete cosine transform (DCT). As the first contribution, we show that DCT can be integrated into the network structure as a convolutional DCT (CDCT) layer. With the CDCT layer, we construct the DCT deep SR (DCT-DSR) network. We further extend the DCT-DSR to allow the CDCT layer to become trainable (i.e., optimizable). Because this layer represents an image transform, we enforce pairwise orthogonality constraints and newly formulated complexity order constraints on the individual basis functions/filters. This orthogonally regularized deep SR network (ORDSR) simplifies the SR task by taking advantage of image transform domain while adapting the design of transform basis to the training image set. The experimental results show ORDSR achieves state-of-the-art SR image quality with fewer parameters than most of the deep CNN methods. A particular success of ORDSR is in overcoming the artifacts introduced by bicubic interpolation. A key burden of deep SR has been identified as the requirement of generous training LR and HR image pairs; ORSDR exhibits a much more graceful degradation as training size is reduced with significant benefits in the regime of limited training. Analysis of memory and computation requirements confirms that ORDSR can allow for a more efficient network with faster inference.

Original languageEnglish (US)
Article number8704993
Pages (from-to)4685-4700
Number of pages16
JournalIEEE Transactions on Image Processing
Volume28
Issue number9
DOIs
StatePublished - Sep 2019

Fingerprint

Discrete cosine transforms
Mathematical transformations
Optical resolving power
Image resolution
Neural networks
Image quality
Interpolation
Data storage equipment
Degradation

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design

Cite this

@article{6476f79d5fe64d629252e97ff79796bb,
title = "Adaptive Transform Domain Image Super-Resolution via Orthogonally Regularized Deep Networks",
abstract = "Deep learning methods, in particular, trained convolutional neural networks (CNNs) have recently been shown to produce compelling results for single image super-resolution (SR). Invariably, a CNN is learned to map the low resolution (LR) image to its corresponding high resolution (HR) version in the spatial domain. We propose a novel network structure for learning the SR mapping function in an image transform domain, specifically the discrete cosine transform (DCT). As the first contribution, we show that DCT can be integrated into the network structure as a convolutional DCT (CDCT) layer. With the CDCT layer, we construct the DCT deep SR (DCT-DSR) network. We further extend the DCT-DSR to allow the CDCT layer to become trainable (i.e., optimizable). Because this layer represents an image transform, we enforce pairwise orthogonality constraints and newly formulated complexity order constraints on the individual basis functions/filters. This orthogonally regularized deep SR network (ORDSR) simplifies the SR task by taking advantage of image transform domain while adapting the design of transform basis to the training image set. The experimental results show ORDSR achieves state-of-the-art SR image quality with fewer parameters than most of the deep CNN methods. A particular success of ORDSR is in overcoming the artifacts introduced by bicubic interpolation. A key burden of deep SR has been identified as the requirement of generous training LR and HR image pairs; ORSDR exhibits a much more graceful degradation as training size is reduced with significant benefits in the regime of limited training. Analysis of memory and computation requirements confirms that ORDSR can allow for a more efficient network with faster inference.",
author = "Tiantong Guo and Mousavi, {Hojjat Seyed} and Vishal Monga",
year = "2019",
month = "9",
doi = "10.1109/TIP.2019.2913500",
language = "English (US)",
volume = "28",
pages = "4685--4700",
journal = "IEEE Transactions on Image Processing",
issn = "1057-7149",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "9",

}

Adaptive Transform Domain Image Super-Resolution via Orthogonally Regularized Deep Networks. / Guo, Tiantong; Mousavi, Hojjat Seyed; Monga, Vishal.

In: IEEE Transactions on Image Processing, Vol. 28, No. 9, 8704993, 09.2019, p. 4685-4700.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Adaptive Transform Domain Image Super-Resolution via Orthogonally Regularized Deep Networks

AU - Guo, Tiantong

AU - Mousavi, Hojjat Seyed

AU - Monga, Vishal

PY - 2019/9

Y1 - 2019/9

N2 - Deep learning methods, in particular, trained convolutional neural networks (CNNs) have recently been shown to produce compelling results for single image super-resolution (SR). Invariably, a CNN is learned to map the low resolution (LR) image to its corresponding high resolution (HR) version in the spatial domain. We propose a novel network structure for learning the SR mapping function in an image transform domain, specifically the discrete cosine transform (DCT). As the first contribution, we show that DCT can be integrated into the network structure as a convolutional DCT (CDCT) layer. With the CDCT layer, we construct the DCT deep SR (DCT-DSR) network. We further extend the DCT-DSR to allow the CDCT layer to become trainable (i.e., optimizable). Because this layer represents an image transform, we enforce pairwise orthogonality constraints and newly formulated complexity order constraints on the individual basis functions/filters. This orthogonally regularized deep SR network (ORDSR) simplifies the SR task by taking advantage of image transform domain while adapting the design of transform basis to the training image set. The experimental results show ORDSR achieves state-of-the-art SR image quality with fewer parameters than most of the deep CNN methods. A particular success of ORDSR is in overcoming the artifacts introduced by bicubic interpolation. A key burden of deep SR has been identified as the requirement of generous training LR and HR image pairs; ORSDR exhibits a much more graceful degradation as training size is reduced with significant benefits in the regime of limited training. Analysis of memory and computation requirements confirms that ORDSR can allow for a more efficient network with faster inference.

AB - Deep learning methods, in particular, trained convolutional neural networks (CNNs) have recently been shown to produce compelling results for single image super-resolution (SR). Invariably, a CNN is learned to map the low resolution (LR) image to its corresponding high resolution (HR) version in the spatial domain. We propose a novel network structure for learning the SR mapping function in an image transform domain, specifically the discrete cosine transform (DCT). As the first contribution, we show that DCT can be integrated into the network structure as a convolutional DCT (CDCT) layer. With the CDCT layer, we construct the DCT deep SR (DCT-DSR) network. We further extend the DCT-DSR to allow the CDCT layer to become trainable (i.e., optimizable). Because this layer represents an image transform, we enforce pairwise orthogonality constraints and newly formulated complexity order constraints on the individual basis functions/filters. This orthogonally regularized deep SR network (ORDSR) simplifies the SR task by taking advantage of image transform domain while adapting the design of transform basis to the training image set. The experimental results show ORDSR achieves state-of-the-art SR image quality with fewer parameters than most of the deep CNN methods. A particular success of ORDSR is in overcoming the artifacts introduced by bicubic interpolation. A key burden of deep SR has been identified as the requirement of generous training LR and HR image pairs; ORSDR exhibits a much more graceful degradation as training size is reduced with significant benefits in the regime of limited training. Analysis of memory and computation requirements confirms that ORDSR can allow for a more efficient network with faster inference.

UR - http://www.scopus.com/inward/record.url?scp=85069775108&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85069775108&partnerID=8YFLogxK

U2 - 10.1109/TIP.2019.2913500

DO - 10.1109/TIP.2019.2913500

M3 - Article

AN - SCOPUS:85069775108

VL - 28

SP - 4685

EP - 4700

JO - IEEE Transactions on Image Processing

JF - IEEE Transactions on Image Processing

SN - 1057-7149

IS - 9

M1 - 8704993

ER -