ENSEMBLE-BASED MODEL FOR MITIGATING FEATURE DISCREPANCIES FOR ENHANCED THREAT DETECTION USING DOMAIN ADAPTATION

Abstract

In today’s highly interconnected digital world, there are varieties of threat actors and threat types which necessitate a deep and robust threat detection system. Algorithms for detecting threats rely on various features of security data to identify potential threats. However, some threats are feature-dependent making it nontrivial to detect all types of threats using the same set of features in the dataset. Again, there is the problem of variations in the number and type of features in different datasets which are used in threat detection, which lead to feature discrepancy. Discrepancy in security telemetry datasets can be a potential cause of threat misclassification and consequently low threat detection system performance. In this paper, we propose an ensemble technique (Ensemble-DAFE) that integrates two techniques for mitigating feature discrepancy in security data viz domain adaptation (DA) and feature engineering (FE) techniques leveraging the strengths of the two to improve threat detection accuracy. We conducted experiments to determine the impact of feature discrepancies on threat detection performance. We obtained a threat detection performance accuracy of 99.96%. when the combined DA and FE was implemented compared to performance accuracy 96.38% without DA. Our result for Ensemble-DAFE with DA combined with FE outperforms state-of-the-art methods without DA compared to ours in terms of detection accuracy. We evaluate the effectiveness of our Ensemble-DAFE threat detection model using a synthetic dataset of network traffic with real-world security features. Based on the result, we noticed a 3.58% improvement in detection performance due to the integration of DA in the threat detection process and demonstrate its ability to reduce false negatives and false positives compared to individual feature-based detection methods.