Enhancing Malware Detection Performance Using Hybrid Machine Learning Based Feature Selection Methods

Abstract

The rapid proliferation of Android malware poses a persistent threat to mobile security, yetmachine learning-based detectors typically operate on high-dimensional feature spaces that introduce significant computational overhead, reduce model interpretability, and limit practical deployment in resource-constrained environments. This thesis addresses these challenges by proposing a hybrid two-stage feature selection framework that combines an ensemble of three filter methods Information Gain, Fisher Score, and Chi-Squared with a Binary Whale Optimisation Algorithm (BWOA) wrapper search guided by Random Forest and XGBoost classifiers, using task-specific fitness functions: F2-score for binary detection to prioritise malware recall, and F1-macro for multiclass categorisation to balance performance across malware families. Evaluated on the CICMalDroid 2020 and TUANDROMD benchmarks, the proposed BWOA-RF configuration achieves 97.93% binary detection accuracy and the highest malware recall (96.05%) among all configurations, using only 87 features an 81.5% reduction from the original 470 while the BWOA-Ensemble configuration achieves 95.00% accuracy and a macro F1 of 0.9405 with 116 features (75.3% reduction), outperforming both fullfeature baselines. Cross-dataset validation onthestructurally distinct TUANDROMD benchmark confirms cross-representation generalisation with 99.78% accuracy and a 56.8% feature reduction. Feature stability analysis identifies a consistent set of securitymeaningful behavioural indicators including getDeviceId,mprotect, andwritevselected reliably across independent optimisation runs, providing actionable insight for realworld Android malware detection