Rating Image Aesthetics Using Deep Learning

Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, James Wang

Research output: Contribution to journalArticle

82 Citations (Scopus)

Abstract

This paper investigates unified feature learning and classifier training approaches for image aesthetics assessment. Existing methods built upon handcrafted or generic image features and developed machine learning and statistical modeling techniques utilizing training examples. We adopt a novel deep neural network approach to allow unified feature learning and classifier training to estimate image aesthetics. In particular, we develop a double-column deep convolutional neural network to support heterogeneous inputs, i.e., global and local views, in order to capture both global and local characteristics of images. In addition, we employ the style and semantic attributes of images to further boost the aesthetics categorization performance. Experimental results show that our approach produces significantly better results than the earlier reported results on the AVA dataset for both the generic image aesthetics and content -based image aesthetics. Moreover, we introduce a 1.5-million image dataset (IAD) for image aesthetics assessment and we further boost the performance on the AVA test set by training the proposed deep neural networks on the IAD dataset.

Original languageEnglish (US)
Article number7243357
Pages (from-to)2021-2034
Number of pages14
JournalIEEE Transactions on Multimedia
Volume17
Issue number11
DOIs
StatePublished - Nov 1 2015

Fingerprint

Classifiers
Learning systems
Semantics
Neural networks
Deep learning
Deep neural networks

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this

Lu, Xin ; Lin, Zhe ; Jin, Hailin ; Yang, Jianchao ; Wang, James. / Rating Image Aesthetics Using Deep Learning. In: IEEE Transactions on Multimedia. 2015 ; Vol. 17, No. 11. pp. 2021-2034.
@article{4260aef6b392484596d6522f395b9288,
title = "Rating Image Aesthetics Using Deep Learning",
abstract = "This paper investigates unified feature learning and classifier training approaches for image aesthetics assessment. Existing methods built upon handcrafted or generic image features and developed machine learning and statistical modeling techniques utilizing training examples. We adopt a novel deep neural network approach to allow unified feature learning and classifier training to estimate image aesthetics. In particular, we develop a double-column deep convolutional neural network to support heterogeneous inputs, i.e., global and local views, in order to capture both global and local characteristics of images. In addition, we employ the style and semantic attributes of images to further boost the aesthetics categorization performance. Experimental results show that our approach produces significantly better results than the earlier reported results on the AVA dataset for both the generic image aesthetics and content -based image aesthetics. Moreover, we introduce a 1.5-million image dataset (IAD) for image aesthetics assessment and we further boost the performance on the AVA test set by training the proposed deep neural networks on the IAD dataset.",
author = "Xin Lu and Zhe Lin and Hailin Jin and Jianchao Yang and James Wang",
year = "2015",
month = "11",
day = "1",
doi = "10.1109/TMM.2015.2477040",
language = "English (US)",
volume = "17",
pages = "2021--2034",
journal = "IEEE Transactions on Multimedia",
issn = "1520-9210",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "11",

}

Lu, X, Lin, Z, Jin, H, Yang, J & Wang, J 2015, 'Rating Image Aesthetics Using Deep Learning', IEEE Transactions on Multimedia, vol. 17, no. 11, 7243357, pp. 2021-2034. https://doi.org/10.1109/TMM.2015.2477040

Rating Image Aesthetics Using Deep Learning. / Lu, Xin; Lin, Zhe; Jin, Hailin; Yang, Jianchao; Wang, James.

In: IEEE Transactions on Multimedia, Vol. 17, No. 11, 7243357, 01.11.2015, p. 2021-2034.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Rating Image Aesthetics Using Deep Learning

AU - Lu, Xin

AU - Lin, Zhe

AU - Jin, Hailin

AU - Yang, Jianchao

AU - Wang, James

PY - 2015/11/1

Y1 - 2015/11/1

N2 - This paper investigates unified feature learning and classifier training approaches for image aesthetics assessment. Existing methods built upon handcrafted or generic image features and developed machine learning and statistical modeling techniques utilizing training examples. We adopt a novel deep neural network approach to allow unified feature learning and classifier training to estimate image aesthetics. In particular, we develop a double-column deep convolutional neural network to support heterogeneous inputs, i.e., global and local views, in order to capture both global and local characteristics of images. In addition, we employ the style and semantic attributes of images to further boost the aesthetics categorization performance. Experimental results show that our approach produces significantly better results than the earlier reported results on the AVA dataset for both the generic image aesthetics and content -based image aesthetics. Moreover, we introduce a 1.5-million image dataset (IAD) for image aesthetics assessment and we further boost the performance on the AVA test set by training the proposed deep neural networks on the IAD dataset.

AB - This paper investigates unified feature learning and classifier training approaches for image aesthetics assessment. Existing methods built upon handcrafted or generic image features and developed machine learning and statistical modeling techniques utilizing training examples. We adopt a novel deep neural network approach to allow unified feature learning and classifier training to estimate image aesthetics. In particular, we develop a double-column deep convolutional neural network to support heterogeneous inputs, i.e., global and local views, in order to capture both global and local characteristics of images. In addition, we employ the style and semantic attributes of images to further boost the aesthetics categorization performance. Experimental results show that our approach produces significantly better results than the earlier reported results on the AVA dataset for both the generic image aesthetics and content -based image aesthetics. Moreover, we introduce a 1.5-million image dataset (IAD) for image aesthetics assessment and we further boost the performance on the AVA test set by training the proposed deep neural networks on the IAD dataset.

UR - http://www.scopus.com/inward/record.url?scp=84946732451&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84946732451&partnerID=8YFLogxK

U2 - 10.1109/TMM.2015.2477040

DO - 10.1109/TMM.2015.2477040

M3 - Article

AN - SCOPUS:84946732451

VL - 17

SP - 2021

EP - 2034

JO - IEEE Transactions on Multimedia

JF - IEEE Transactions on Multimedia

SN - 1520-9210

IS - 11

M1 - 7243357

ER -