How glcrealm/ids-anomaly-detection-with-nsl-kdd works
Project type and purpose
• This is an open-source machine-learning tool for network intrusion detection.
• It trains and evaluates an anomaly detection model (Random Forest) on the NSL-KDD dataset and provides a simple prediction interface via a test script.
File structure
• Top-level assets:
• IDS_train.ipynb – interactive notebook for data preparation, training, evaluation
• test_model.py – standalone inference script
• ebest_model.pkl – serialized trained model
• dataset/ – raw data files (KDDTrain+.txt, KDDTest+.txt, kddcup.names, attack.types)
• requirements.txt – Python dependencies
• No separate frontend or backend; it is a data-pipeline + model serving tool.
README for additional insights
• Confirms the use of NSL-KDD dataset, preprocessing steps, feature engineering, hyperparameter tuning, metrics.
• No external services or UI components; it is a CLI/notebook-driven workflow.
a. Main components
1. Data Storage
– Raw dataset folder with training and test data
2. Data Preprocessing & Feature Engineering
– Encoding categorical features, computing protocol/attack probabilities
3. Model Training Pipeline
– Randomized search, cross-validation, Random Forest fitting
4. Model Persistence
– Export/serialize best model to ebest_model.pkl via joblib
5. Inference Engine
– test_model.py loads ebest_model.pkl, applies same preprocessing, outputs normal/anomaly
6. Evaluation Module (within notebook)
– Accuracy, precision, recall, F1, confusion matrix
b. Relationships and interactions
c. Architectural patterns and principles
– Pipeline pattern: clear separation of data ingestion, transformation, training, serving
– Modular design: reusable preprocessing functions imported both by training notebook and test script
– Versioning of artifacts: model serialized with a deterministic name for reproducibility
d. Relevant technologies/libraries
– pandas, numpy for data handling
– scikit-learn for modeling (RandomForestClassifier, model_selection)
– joblib for model I/O
– matplotlib (notebook) for metric visualization
– Python scripts/notebook as orchestration layer