Sequential anomaly detection in a batch with growing number of tests: Application to network intrusion detection

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Scopus citations

Abstract

For high (N)-dimensional feature spaces, we consider detection of an unknown, anomalous class of samples amongst a batch of collected samples (of size T), under the null hypothesis that all samples follow the same probability law. Since the features which will best identify the anomalies are a priori unknown, several common detection strategies are: 1) evaluating atypicality of a sample (its p-value) based on the null distribution defined on the full N-dimensional feature space; 2) considering a (combinatoric) set of low order distributions, e.g. all singletons and all feature pairs, with detections made based on the smallest p-value yielded over all such low order tests. The first approach relies on accurate estimation of the joint distribution, while the second may suffer from increased false alarm rates as N and T grow. Alternatively, inspired by greedy feature selection commonly used in supervised learning, we propose a novel sequential anomaly detection procedure with a growing number of tests. Here, new tests are (greedily) included only when they are needed, i.e., when their use (on currently undetected samples) will yield greater aggregate statistical significance of (multiple testing corrected) detections than obtainable using the existing test cadre. Our approach thus aims to maximize aggregate statistical significance of all detections made up until a finite horizon. Our method is evaluated, along with supervised methods, for a network intrusion domain, detecting Zeus bot (intrusion) packet flows embedded amongst (normal)Web flows. It is shown that judicious feature representation is essential for discriminating Zeus from Web.

Original languageEnglish (US)
Title of host publication2012 IEEE International Workshop on Machine Learning for Signal Processing - Proceedings of MLSP 2012
DOIs
StatePublished - Dec 12 2012
Event2012 22nd IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2012 - Santander, Spain
Duration: Sep 23 2012Sep 26 2012

Publication series

NameIEEE International Workshop on Machine Learning for Signal Processing, MLSP
ISSN (Print)2161-0363
ISSN (Electronic)2161-0371

Other

Other2012 22nd IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2012
CountrySpain
CitySantander
Period9/23/129/26/12

All Science Journal Classification (ASJC) codes

  • Human-Computer Interaction
  • Signal Processing

Fingerprint Dive into the research topics of 'Sequential anomaly detection in a batch with growing number of tests: Application to network intrusion detection'. Together they form a unique fingerprint.

Cite this