Exchangeability martingales for selecting features in anomaly detection

Giovanni Cherubin, Adrian Baldwin, Jonathan Griffin
Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications, PMLR 91:157-170, 2018.

Abstract

We consider the problem of feature selection for unsupervised anomaly detection (AD) in time-series, where only normal examples are available for training. We develop a method based on exchangeability martingales that only keeps features that exhibit the same pattern (i.e., are i.i.d.) under normal conditions of the observed phenomenon. We apply this to the problem of monitoring a Windows service and detecting anomalies it exhibits if compromised; results show that our method: i) strongly improves the AD system’s performance, and ii) it reduces its computational complexity. Furthermore, it gives results that are easy to interpret for analysts, and it potentially increases robustness against AD evasion attacks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v91-cherubin18a, title = {Exchangeability martingales for selecting features in anomaly detection}, author = {Cherubin, Giovanni and Baldwin, Adrian and Griffin, Jonathan}, booktitle = {Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications}, pages = {157--170}, year = {2018}, editor = {Gammerman, Alex and Vovk, Vladimir and Luo, Zhiyuan and Smirnov, Evgueni and Peeters, Ralf}, volume = {91}, series = {Proceedings of Machine Learning Research}, month = {11--13 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v91/cherubin18a/cherubin18a.pdf}, url = {https://proceedings.mlr.press/v91/cherubin18a.html}, abstract = {We consider the problem of feature selection for unsupervised anomaly detection (AD) in time-series, where only normal examples are available for training. We develop a method based on exchangeability martingales that only keeps features that exhibit the same pattern (i.e., are i.i.d.) under normal conditions of the observed phenomenon. We apply this to the problem of monitoring a Windows service and detecting anomalies it exhibits if compromised; results show that our method: i) strongly improves the AD system’s performance, and ii) it reduces its computational complexity. Furthermore, it gives results that are easy to interpret for analysts, and it potentially increases robustness against AD evasion attacks.} }
Endnote
%0 Conference Paper %T Exchangeability martingales for selecting features in anomaly detection %A Giovanni Cherubin %A Adrian Baldwin %A Jonathan Griffin %B Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications %C Proceedings of Machine Learning Research %D 2018 %E Alex Gammerman %E Vladimir Vovk %E Zhiyuan Luo %E Evgueni Smirnov %E Ralf Peeters %F pmlr-v91-cherubin18a %I PMLR %P 157--170 %U https://proceedings.mlr.press/v91/cherubin18a.html %V 91 %X We consider the problem of feature selection for unsupervised anomaly detection (AD) in time-series, where only normal examples are available for training. We develop a method based on exchangeability martingales that only keeps features that exhibit the same pattern (i.e., are i.i.d.) under normal conditions of the observed phenomenon. We apply this to the problem of monitoring a Windows service and detecting anomalies it exhibits if compromised; results show that our method: i) strongly improves the AD system’s performance, and ii) it reduces its computational complexity. Furthermore, it gives results that are easy to interpret for analysts, and it potentially increases robustness against AD evasion attacks.
APA
Cherubin, G., Baldwin, A. & Griffin, J.. (2018). Exchangeability martingales for selecting features in anomaly detection. Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications, in Proceedings of Machine Learning Research 91:157-170 Available from https://proceedings.mlr.press/v91/cherubin18a.html.

Related Material