Abstract

Over the last few years, a plethora of papers presenting machine learning-based approaches for intrusion detection has been published. However, the majority of those papers does not compare their results with a proper baseline of a signature-based intrusion detection system. Thus violating good machine learning practices. In order to evaluate the pros and cons of the machine learning-based approach, we replicated a research study which use a deep neural network model for intrusion detection. The results of our replicated research study expose several systematic problems with the used datasets and evaluation methods. In our experiments, a signature-based intrusion detection system with a minimal setup was able to outperform the tested model even under small traffic changes. While testing the replicated neural network on a new dataset recorded in the same environment with the same attacks using the same tools showed that the accuracy of the neural network dropped to 54%. Furthermore, the often claimed advantage of being able to detect zero-day attacks could not be seen in our experiments.