Android malware detection: Investigating the impact of imbalanced data-sets on the performance of machine learning models

Abstract

Artificial intelligence has revolutionized many areas of research, including research on malicious application detection and classification. Nowadays, there are many approaches that learn from existing data and predict the classes of new data. Machine learning principles recommend a balance of classes in the training dataset, but the reality in the field is quite different. The majority of datasets used for malicious application detection are imbalanced. Class imbalance degrades classifier performance, so it is a common problem in classification tasks. This observation is much more significant in the area of Android malware detection and classification. There are few works to our knowledge on the effects of imbalanced datasets in the field of Android malware detection. Our contribution focuses on the impact of imbalanced datasets on the performance of different algorithms and the suitability of using evaluation metrics in Android malware detection. We show that for malicious application detection, some classification algorithms are not suitable for unbalanced datasets. We also proved that some of the most used in literature performance evaluation metrics (Accuracy, Precision, Recall) are not very well adapted to imbalanced datasets. On the other hand, the metrics (Balanced_accuracy, Geometric mean) are more adapted. These results were obtained by evaluating the performances of eleven classification algorithms and also the adequacy of the different evaluation metrics (Accuracy, Recall, Precision, F1_score, Balanced accuracy, Matthews corrcoef, Geometric mean, Fowlkes_mallows).