Uncovering Unknown Unknowns in Financial Services Big Data by Unsupervised Methodologies: Present and Future trends

Gil Shabat, David Segev, Amir Averbuch
Proceedings of the KDD 2017: Workshop on Anomaly Detection in Finance, PMLR 71:8-19, 2018.

Abstract

Currently, unknown unknowns in high dimensional big data environments can go unnoticed for a long period of time. The failure to detect anomalies in critical infrastructure data can result in extensive financial, operational, reputational and life threatening consequences. In this paper, we describe algorithms for an automatic and unsupervised anomaly detection that do not necessitate domain expertise, signatures, rules, patterns or semantics understanding of the features. We propose several new methodologies for anomaly detection to protect critical infrastructures, with emphasis on finance, spanning from theory to actionable technology. Although anomalies can originate from several sources, we also show that cyber threat, financial and operational malfunction are converging into a single detection paradigm. Performance comparison between different algorithms (ours and others) is presented as well as examples from real use cases.

Cite this Paper

BibTeX
@InProceedings{pmlr-v71-shabat18a, title = {Uncovering Unknown Unknowns in Financial Services Big Data by Unsupervised Methodologies: Present and Future trends}, author = {Shabat, Gil and Segev, David and Averbuch, Amir}, booktitle = {Proceedings of the KDD 2017: Workshop on Anomaly Detection in Finance}, pages = {8--19}, year = {2018}, editor = {Anandakrishnan, Archana and Kumar, Senthil and Statnikov, Alexander and Faruquie, Tanveer and Xu, Di}, volume = {71}, series = {Proceedings of Machine Learning Research}, month = {14 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v71/shabat18a/shabat18a.pdf}, url = {https://proceedings.mlr.press/v71/shabat18a.html}, abstract = {Currently, unknown unknowns in high dimensional big data environments can go unnoticed for a long period of time. The failure to detect anomalies in critical infrastructure data can result in extensive financial, operational, reputational and life threatening consequences. In this paper, we describe algorithms for an automatic and unsupervised anomaly detection that do not necessitate domain expertise, signatures, rules, patterns or semantics understanding of the features. We propose several new methodologies for anomaly detection to protect critical infrastructures, with emphasis on finance, spanning from theory to actionable technology. Although anomalies can originate from several sources, we also show that cyber threat, financial and operational malfunction are converging into a single detection paradigm. Performance comparison between different algorithms (ours and others) is presented as well as examples from real use cases.} }
Endnote
%0 Conference Paper %T Uncovering Unknown Unknowns in Financial Services Big Data by Unsupervised Methodologies: Present and Future trends %A Gil Shabat %A David Segev %A Amir Averbuch %B Proceedings of the KDD 2017: Workshop on Anomaly Detection in Finance %C Proceedings of Machine Learning Research %D 2018 %E Archana Anandakrishnan %E Senthil Kumar %E Alexander Statnikov %E Tanveer Faruquie %E Di Xu %F pmlr-v71-shabat18a %I PMLR %P 8--19 %U https://proceedings.mlr.press/v71/shabat18a.html %V 71 %X Currently, unknown unknowns in high dimensional big data environments can go unnoticed for a long period of time. The failure to detect anomalies in critical infrastructure data can result in extensive financial, operational, reputational and life threatening consequences. In this paper, we describe algorithms for an automatic and unsupervised anomaly detection that do not necessitate domain expertise, signatures, rules, patterns or semantics understanding of the features. We propose several new methodologies for anomaly detection to protect critical infrastructures, with emphasis on finance, spanning from theory to actionable technology. Although anomalies can originate from several sources, we also show that cyber threat, financial and operational malfunction are converging into a single detection paradigm. Performance comparison between different algorithms (ours and others) is presented as well as examples from real use cases.
APA
Shabat, G., Segev, D. & Averbuch, A.. (2018). Uncovering Unknown Unknowns in Financial Services Big Data by Unsupervised Methodologies: Present and Future trends. Proceedings of the KDD 2017: Workshop on Anomaly Detection in Finance, in Proceedings of Machine Learning Research 71:8-19 Available from https://proceedings.mlr.press/v71/shabat18a.html.

Related Material