Deep multi-patch aggregation network for image style, aesthetics, and quality estimation

Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech, James Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

130 Scopus citations

Abstract

This paper investigates problems of image style, aesthetics, and quality estimation, which require fine-grained details from high-resolution images, utilizing deep neural network training approach. Existing deep convolutional neural networks mostly extracted one patch such as a down-sized crop from each image as a training example. However, one patch may not always well represent the entire image, which may cause ambiguity during training. We propose a deep multi-patch aggregation network training approach, which allows us to train models using multiple patches generated from one image. We achieve this by constructing multiple, shared columns in the neural network and feeding multiple patches to each of the columns. More importantly, we propose two novel network layers (statistics and sorting) to support aggregation of those patches. The proposed deep multi-patch aggregation network integrates shared feature learning and aggregation function learning into a unified framework. We demonstrate the effectiveness of the deep multi-patch aggregation network on the three problems, i.e., image style recognition, aesthetic quality categorization, and image quality estimation. Our models trained using the proposed networks significantly outperformed the state of the art in all three applications.

Original languageEnglish (US)
Title of host publication2015 International Conference on Computer Vision, ICCV 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages990-998
Number of pages9
ISBN (Electronic)9781467383912
DOIs
StatePublished - Feb 17 2015
Event15th IEEE International Conference on Computer Vision, ICCV 2015 - Santiago, Chile
Duration: Dec 11 2015Dec 18 2015

Other

Other15th IEEE International Conference on Computer Vision, ICCV 2015
CountryChile
CitySantiago
Period12/11/1512/18/15

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition

Fingerprint Dive into the research topics of 'Deep multi-patch aggregation network for image style, aesthetics, and quality estimation'. Together they form a unique fingerprint.

  • Cite this

    Lu, X., Lin, Z., Shen, X., Mech, R., & Wang, J. (2015). Deep multi-patch aggregation network for image style, aesthetics, and quality estimation. In 2015 International Conference on Computer Vision, ICCV 2015 (pp. 990-998). [7410476] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCV.2015.119