The limitations of deep learning in adversarial settings

Nicolas Papernot, Patrick Mcdaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, Ananthram Swami

Research output: Chapter in Book/Report/Conference proceedingConference contribution

947 Scopus citations

Abstract

Deep learning takes advantage of large datasets and computationally efficient training algorithms to outperform other approaches at various machine learning tasks. However, imperfections in the training phase of deep neural networks make them vulnerable to adversarial samples: inputs crafted by adversaries with the intent of causing deep neural networks to misclassify. In this work, we formalize the space of adversaries against deep neural networks (DNNs) and introduce a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs. In an application to computer vision, we show that our algorithms can reliably produce samples correctly classified by human subjects but misclassified in specific targets by a DNN with a 97% adversarial success rate while only modifying on average 4.02% of the input features per sample. We then evaluate the vulnerability of different sample classes to adversarial perturbations by defining a hardness measure. Finally, we describe preliminary work outlining defenses against adversarial samples by defining a predictive measure of distance between a benign input and a target classification.

Original languageEnglish (US)
Title of host publicationProceedings - 2016 IEEE European Symposium on Security and Privacy, EURO S and P 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages372-387
Number of pages16
ISBN (Electronic)9781509017515
DOIs
StatePublished - May 9 2016
Event1st IEEE European Symposium on Security and Privacy, EURO S and P 2016 - Saarbruecken, Germany
Duration: Mar 21 2016Mar 24 2016

Publication series

NameProceedings - 2016 IEEE European Symposium on Security and Privacy, EURO S and P 2016

Other

Other1st IEEE European Symposium on Security and Privacy, EURO S and P 2016
CountryGermany
CitySaarbruecken
Period3/21/163/24/16

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'The limitations of deep learning in adversarial settings'. Together they form a unique fingerprint.

Cite this