ISMORE: An iterative self super-resolution algorithm

Can Zhao, Seoyoung Son, Yongsoo Kim, Jerry L. Prince

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In 3D medical imaging, images with isotropic high resolution (HR) are almost always preferred. In practice, however, many acquired images, including magnetic resonance imaging (MRI) and fluorescence microscopy, have HR in the in-plane directions and low resolution (LR) in the through-plane direction. The blurriness and aliasing artifacts that result cannot be solved by simple interpolation. Instead, many researchers have proposed super-resolution algorithms including state-of-art convolutional neural network (CNN)-based methods that require matched training data that have paired LR/HR examples. Since these data are often unavailable in practice, self super-resolution algorithms that do not need external training data have also been proposed. These self super-resolution methods assume that the in-plane slices are HR, and can therefore be used as HR training data. By degrading them into LR images, 2D CNNs can be trained and then used to restore the images in the through-plane. However, there are two issues with these approaches. The first one is that the assumption of HR in-plane slices is actually not solid since these thick in-plane slices are averaged true HR thin slices. Training on thick slices is equivalent to training on averaged true HR images, which is suboptimal. The second one relates to the 2D CNNs used on 3D volume, which cannot guarantee slice consistency. Regarding both issues as well as the generalizability of algorithm, we made four contributions. We show in this paper that one of the existing 2D CNN-based self super-resolution methods, SMORE, can be further improved by iteratively applying it using 2D or 3D networks, yielding 2D and 3D iSMORE. This iterative framework improves training data from thick slices to thinner slices after each iteration, thus improves super-resolution accuracy after each iteration, and solves the first issue. The second contribution is that it uses a 3D network to preserve slice consistency. The third contribution is the use of an edge-based loss function and noise reduction to enhance the performance. Finally, we perform iSMORE on both MRI and two-photon fluorescence microscopy, which demonstrates its generalizability.

Original languageEnglish (US)
Title of host publicationSimulation and Synthesis in Medical Imaging - 4th International Workshop, SASHIMI 2019, Held in Conjunction with MICCAI 2019, Proceedings
EditorsNinon Burgos, Ali Gooya, David Svoboda
PublisherSpringer
Pages130-139
Number of pages10
ISBN (Print)9783030327774
DOIs
StatePublished - Jan 1 2019
Event4th International Workshop on Simulation and Synthesis in Medical Imaging, SASHIMI 2019, held in conjunction with the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2019 - Shenzhen, China
Duration: Oct 13 2019Oct 13 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11827 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference4th International Workshop on Simulation and Synthesis in Medical Imaging, SASHIMI 2019, held in conjunction with the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2019
CountryChina
CityShenzhen
Period10/13/1910/13/19

Fingerprint

Super-resolution
Slice
Fluorescence microscopy
Magnetic resonance
High Resolution
Image resolution
Neural networks
Imaging techniques
Medical imaging
Noise abatement
Fluorescence Microscopy
Interpolation
Photons
Magnetic Resonance Imaging
Neural Networks
Iteration
3D Imaging
Aliasing
Medical Imaging
Noise Reduction

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Zhao, C., Son, S., Kim, Y., & Prince, J. L. (2019). ISMORE: An iterative self super-resolution algorithm. In N. Burgos, A. Gooya, & D. Svoboda (Eds.), Simulation and Synthesis in Medical Imaging - 4th International Workshop, SASHIMI 2019, Held in Conjunction with MICCAI 2019, Proceedings (pp. 130-139). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11827 LNCS). Springer. https://doi.org/10.1007/978-3-030-32778-1_14
Zhao, Can ; Son, Seoyoung ; Kim, Yongsoo ; Prince, Jerry L. / ISMORE : An iterative self super-resolution algorithm. Simulation and Synthesis in Medical Imaging - 4th International Workshop, SASHIMI 2019, Held in Conjunction with MICCAI 2019, Proceedings. editor / Ninon Burgos ; Ali Gooya ; David Svoboda. Springer, 2019. pp. 130-139 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{2a627c9b6bb447d2a58505c3278ea9bd,
title = "ISMORE: An iterative self super-resolution algorithm",
abstract = "In 3D medical imaging, images with isotropic high resolution (HR) are almost always preferred. In practice, however, many acquired images, including magnetic resonance imaging (MRI) and fluorescence microscopy, have HR in the in-plane directions and low resolution (LR) in the through-plane direction. The blurriness and aliasing artifacts that result cannot be solved by simple interpolation. Instead, many researchers have proposed super-resolution algorithms including state-of-art convolutional neural network (CNN)-based methods that require matched training data that have paired LR/HR examples. Since these data are often unavailable in practice, self super-resolution algorithms that do not need external training data have also been proposed. These self super-resolution methods assume that the in-plane slices are HR, and can therefore be used as HR training data. By degrading them into LR images, 2D CNNs can be trained and then used to restore the images in the through-plane. However, there are two issues with these approaches. The first one is that the assumption of HR in-plane slices is actually not solid since these thick in-plane slices are averaged true HR thin slices. Training on thick slices is equivalent to training on averaged true HR images, which is suboptimal. The second one relates to the 2D CNNs used on 3D volume, which cannot guarantee slice consistency. Regarding both issues as well as the generalizability of algorithm, we made four contributions. We show in this paper that one of the existing 2D CNN-based self super-resolution methods, SMORE, can be further improved by iteratively applying it using 2D or 3D networks, yielding 2D and 3D iSMORE. This iterative framework improves training data from thick slices to thinner slices after each iteration, thus improves super-resolution accuracy after each iteration, and solves the first issue. The second contribution is that it uses a 3D network to preserve slice consistency. The third contribution is the use of an edge-based loss function and noise reduction to enhance the performance. Finally, we perform iSMORE on both MRI and two-photon fluorescence microscopy, which demonstrates its generalizability.",
author = "Can Zhao and Seoyoung Son and Yongsoo Kim and Prince, {Jerry L.}",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/978-3-030-32778-1_14",
language = "English (US)",
isbn = "9783030327774",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer",
pages = "130--139",
editor = "Ninon Burgos and Ali Gooya and David Svoboda",
booktitle = "Simulation and Synthesis in Medical Imaging - 4th International Workshop, SASHIMI 2019, Held in Conjunction with MICCAI 2019, Proceedings",

}

Zhao, C, Son, S, Kim, Y & Prince, JL 2019, ISMORE: An iterative self super-resolution algorithm. in N Burgos, A Gooya & D Svoboda (eds), Simulation and Synthesis in Medical Imaging - 4th International Workshop, SASHIMI 2019, Held in Conjunction with MICCAI 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11827 LNCS, Springer, pp. 130-139, 4th International Workshop on Simulation and Synthesis in Medical Imaging, SASHIMI 2019, held in conjunction with the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2019, Shenzhen, China, 10/13/19. https://doi.org/10.1007/978-3-030-32778-1_14

ISMORE : An iterative self super-resolution algorithm. / Zhao, Can; Son, Seoyoung; Kim, Yongsoo; Prince, Jerry L.

Simulation and Synthesis in Medical Imaging - 4th International Workshop, SASHIMI 2019, Held in Conjunction with MICCAI 2019, Proceedings. ed. / Ninon Burgos; Ali Gooya; David Svoboda. Springer, 2019. p. 130-139 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11827 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - ISMORE

T2 - An iterative self super-resolution algorithm

AU - Zhao, Can

AU - Son, Seoyoung

AU - Kim, Yongsoo

AU - Prince, Jerry L.

PY - 2019/1/1

Y1 - 2019/1/1

N2 - In 3D medical imaging, images with isotropic high resolution (HR) are almost always preferred. In practice, however, many acquired images, including magnetic resonance imaging (MRI) and fluorescence microscopy, have HR in the in-plane directions and low resolution (LR) in the through-plane direction. The blurriness and aliasing artifacts that result cannot be solved by simple interpolation. Instead, many researchers have proposed super-resolution algorithms including state-of-art convolutional neural network (CNN)-based methods that require matched training data that have paired LR/HR examples. Since these data are often unavailable in practice, self super-resolution algorithms that do not need external training data have also been proposed. These self super-resolution methods assume that the in-plane slices are HR, and can therefore be used as HR training data. By degrading them into LR images, 2D CNNs can be trained and then used to restore the images in the through-plane. However, there are two issues with these approaches. The first one is that the assumption of HR in-plane slices is actually not solid since these thick in-plane slices are averaged true HR thin slices. Training on thick slices is equivalent to training on averaged true HR images, which is suboptimal. The second one relates to the 2D CNNs used on 3D volume, which cannot guarantee slice consistency. Regarding both issues as well as the generalizability of algorithm, we made four contributions. We show in this paper that one of the existing 2D CNN-based self super-resolution methods, SMORE, can be further improved by iteratively applying it using 2D or 3D networks, yielding 2D and 3D iSMORE. This iterative framework improves training data from thick slices to thinner slices after each iteration, thus improves super-resolution accuracy after each iteration, and solves the first issue. The second contribution is that it uses a 3D network to preserve slice consistency. The third contribution is the use of an edge-based loss function and noise reduction to enhance the performance. Finally, we perform iSMORE on both MRI and two-photon fluorescence microscopy, which demonstrates its generalizability.

AB - In 3D medical imaging, images with isotropic high resolution (HR) are almost always preferred. In practice, however, many acquired images, including magnetic resonance imaging (MRI) and fluorescence microscopy, have HR in the in-plane directions and low resolution (LR) in the through-plane direction. The blurriness and aliasing artifacts that result cannot be solved by simple interpolation. Instead, many researchers have proposed super-resolution algorithms including state-of-art convolutional neural network (CNN)-based methods that require matched training data that have paired LR/HR examples. Since these data are often unavailable in practice, self super-resolution algorithms that do not need external training data have also been proposed. These self super-resolution methods assume that the in-plane slices are HR, and can therefore be used as HR training data. By degrading them into LR images, 2D CNNs can be trained and then used to restore the images in the through-plane. However, there are two issues with these approaches. The first one is that the assumption of HR in-plane slices is actually not solid since these thick in-plane slices are averaged true HR thin slices. Training on thick slices is equivalent to training on averaged true HR images, which is suboptimal. The second one relates to the 2D CNNs used on 3D volume, which cannot guarantee slice consistency. Regarding both issues as well as the generalizability of algorithm, we made four contributions. We show in this paper that one of the existing 2D CNN-based self super-resolution methods, SMORE, can be further improved by iteratively applying it using 2D or 3D networks, yielding 2D and 3D iSMORE. This iterative framework improves training data from thick slices to thinner slices after each iteration, thus improves super-resolution accuracy after each iteration, and solves the first issue. The second contribution is that it uses a 3D network to preserve slice consistency. The third contribution is the use of an edge-based loss function and noise reduction to enhance the performance. Finally, we perform iSMORE on both MRI and two-photon fluorescence microscopy, which demonstrates its generalizability.

UR - http://www.scopus.com/inward/record.url?scp=85075647573&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85075647573&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-32778-1_14

DO - 10.1007/978-3-030-32778-1_14

M3 - Conference contribution

AN - SCOPUS:85075647573

SN - 9783030327774

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 130

EP - 139

BT - Simulation and Synthesis in Medical Imaging - 4th International Workshop, SASHIMI 2019, Held in Conjunction with MICCAI 2019, Proceedings

A2 - Burgos, Ninon

A2 - Gooya, Ali

A2 - Svoboda, David

PB - Springer

ER -

Zhao C, Son S, Kim Y, Prince JL. ISMORE: An iterative self super-resolution algorithm. In Burgos N, Gooya A, Svoboda D, editors, Simulation and Synthesis in Medical Imaging - 4th International Workshop, SASHIMI 2019, Held in Conjunction with MICCAI 2019, Proceedings. Springer. 2019. p. 130-139. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-32778-1_14