Zero-Day APT Detection Using OpenSet Recognition with Adaptive Feature Selection

Abstract

Zero-day Advanced Persistent Threats (APTs) ex-ploit previously unknown vulnerabilities to evade signature-based defences and persist across the enterprise kill chain. This paper presents an end-to-end framework for zero-day APT detection that couples an adaptive feature-selection pipeline with open-set recognition. First, we apply a hybrid Mutual Information → Symmetric Uncertainty → mRMR procedure with adaptive thresholds to capture non-linear relevance while suppressing redundancy and high-cardinality bias, yielding a compact, dis-criminative feature set suitable for real-time inference. On top of this representation, we integrate OpenMax (EVT-based logit calibration) with kernel density estimation (KDE) and Monte Carlo Dropout (MCD) to jointly assess distributional fit and epistemic uncertainty, routing ambiguous samples to an explicit Unknown class. Using the multi-stage DAPT2020 enterprise dataset, we evaluate with 10-fold cross-validation and a leave-one-class-out protocol that simulates unseen (zero-day) attacks. In closed-set classification, ensemble models achieve near-perfect performance (e.g., Random Forest: Accuracy/Precision/Recall/F1 = 0.9997). Under open-set conditions, the proposed OpenMax+KDE+MCD approach attains class-wise accuracies between 0.933 and 0.995, with high-volume behaviours (e.g., brute force, network scans) exceeding 0.99 and stealthier behaviours (e.g., backdoors, web vulnerability scans) detected at 0.93-0.94 while being safely rejected as Unknown when uncertain. The results demonstrate robust zero-day recognition with reduced false negatives and operationally actionable uncertainty signals, offering a practical path toward resilient, next-generation intrusion detection