How katiie104/ids-using-machinelearning works
Project type and purpose
• This is a real-time Network Intrusion Detection System (IDS) using machine learning.
• It has two distinct modes:
a. Offline model training (batch pipeline)
b. Real-time packet capture and inference
File structure
Top-level:
• main.py – entry point that wires together training and monitoring
• dataset/ – raw NSL-KDD training data
• logs/ – application logs
• requirements.txt – dependencies
• src/ – core modules
src/ modules:
• config.py – central configuration (paths, hyperparameters)
• preprocess.py – data cleaning, feature encoding, scaling
• train_model.py – XGBoost training with GridSearchCV, model persistence
• explain_model.py – SHAP explanations for feature importance
• stream_monitor.py – real-time capture via pyshark, feature extraction, inference
• zeek_feature_extractor.py – optional feature extraction from Zeek logs
Extract key architectural insights from README
• Offline pipeline: read KDDTrain → preprocess → train XGBoost → save preprocessor & model
• Online pipeline: capture packets from network interface → extract same features → apply model → output Normal/Attack
• Dependencies: pandas, scikit-learn, xgboost, pyshark, joblib
Main components
A. Data Sources
1. Historical dataset (NSL-KDD files)
2. Live network interface (via pyshark)
B. Offline Pipeline
1. Preprocessing module
2. Model training module
3. Model storage (joblib files in models/)
4. Model explanation module (SHAP)
C. Online Pipeline
1. Packet capture (pyshark.LiveCapture)
2. Feature extractor (zeek_feature_extractor / inline extractor)
3. Preprocessor (load preprocessor.pkl)
4. Inference engine (load xgb_model.pkl)
5. Alerting / terminal output
D. Supporting services
1. Configuration manager (config.py)
2. Logging service (logs/app.log)
Relationships and data flows
• Configuration fed into both pipelines
• Logs written by all modules
Architectural patterns and principles
• Separation of concerns: offline vs online, preprocessing vs inference
• Modular design: each major step in its own module
• Reusability: same preprocessing logic in training and live monitoring
• Extensibility: plugging new feature extractors (Zeek) or models
Technologies, frameworks, and libraries
• Python modules: pandas, scikit-learn, xgboost, joblib, pyshark
• Logging: Python logging module writing to logs/app.log
• Model explainability: SHAP via explain_model.py
Final recommendation
• Capturing both the offline ML pipeline and the online inference pipeline as two parallel streams that share preprocessing logic and configuration.