Comparison of Machine Learning and Statistical Approaches of Detecting Anomalies Using a Simulation Study

Abstract

Aim: An anomaly is an observation or a group of observations that is unusual for a given dataset. Anomaly detection has many applications, not only as a step of data preparation but also, for example, as a way of identifying credit card fraud detection, network intrusions and much more. There are diverse methods of anomaly detection. In particular two groups of methods have been developed independently – statistical methods and machine learning algorithms. Those methods are rarely compared. While statistical methods focus on formulating a measure of the abnormality of the observations, supervised machine learning makes it possible to use data about typical observations and previously identified anomalies. The aim of this paper was to compare the two approaches by conducting a simulation study.

Methodology: A simulation study was conducted, during which the data was generated using copula functions. For the purpose of generating different types of anomalies, marginal distributions of the variables were manipulated. The effectiveness of each method was evaluated based on measures of classification model performance.

Results: While the accuracy of the statistical methods was dependent on the precise prediction of the percentage of the anomalies that would occur in the data, the machine learning algorithms’ recall was significantly lower when the change in the marginal distribution of the value parameters was smaller.

Implications and recommendations: For the statistical methods included in the study, knowledge about the distribution of the variables was crucial while the supervised machine learning algorithms required acquiring a training dataset. Unlike machine learning algorithms, the statistical methods performed with similar accuracy even when the change in the marginal distribution parameters’ value was smaller.

Originality/value: The two approaches to anomaly detection presented in the paper are not often compared, usually used by two separate groups of researchers – statisticians and machine learning or data science specialists.