How zaffar0561/ids works
1. This repository is a small, pipeline-style intrusion detection system for a home network. It is split into two major paths: an offline model-building path and an online monitoring path. The offline side prepares data and trains the classifier; the online side captures live packets, scores them, enriches suspicious activity, and surfaces results in a dashboard.
2. The offline ML subsystem lives in src/dataset_prep.py and src/train_model.py. dataset_prep.py prepares the NSL-KDD dataset, which is the training source mentioned in the README. train_model.py then trains the detection model and saves artifacts into models/. This is the only part of the repo that depends on historical labeled data rather than live traffic.
3. The inference artifacts are stored separately from code in models/. The presence of rf_model.joblib and proto_encoder.joblib indicates a serialized Random Forest classifier plus a preprocessing/encoding artifact used at runtime. That separation is important architecturally: the live detector depends on these saved artifacts rather than retraining in place.
4. Live packet collection is handled by src/sniffer.py. Its role is to observe network traffic on the wire and persist packet observations for downstream processing. In the architecture diagram, this is the ingress point for real-time data and sits before any ML scoring or threat lookup.
5. Real-time detection is implemented in src/realtime_detect.py. This module is the main online processing stage: it consumes the captured traffic, extracts features, loads the saved model artifacts, and classifies packets or flows as normal versus suspicious. It is the bridge between raw packet capture and security actions.
6. src/threat_intel.py is the enrichment and risk-assessment boundary. The README says it integrates with AbuseIPDB, so suspicious IPs can be checked against an external reputation service using an API key from the environment. This module likely adds context to alerts and helps decide whether an IP should be treated as high risk.
7. The system’s response layer is split between logging, alerting, and optional blocking. The README describes auto-blocking on Windows using firewall rules, so the runtime likely branches from “suspicious” detections into either passive alert generation or active remediation. This is the main control boundary in the design because it turns inference results into host-level actions.
8. dashboard/app.py is the user-facing observability layer. It is a Streamlit application that presents live traffic, alerts, and threat intelligence. Architecturally, it does not appear to be the detector itself; it reads the outputs generated by the capture/detection pipeline and visualizes them.
9. Persistence is file-based rather than service-based. The README references captured packets, alerts, and logs stored on disk, and the tree shows prebuilt model artifacts in models/. That means the main data flow is local filesystem-backed: capture writes data, detection reads it and writes alerts, and the dashboard reads those outputs.
10. The repo has a clear runtime boundary between offline preparation and online monitoring. Offline steps build the model and encoder once; online steps require network capture permissions, an optional AbuseIPDB API key, and dashboard access on port 8501. The architecture is therefore a single-machine, locally deployed IDS with one external service dependency and optional OS-specific firewall integration.
Main entry points: realtime_detect.py Live Detect · threat_intel.py Threat Intel · models Model Store · app.py Dashboard
Components 15
Offline ML
Historical dataset preparation and model training.
- NSL-KDD · Historical labeled dataset used to build the detector.
- Prep Data · Cleans and prepares the training dataset for model building.
src/dataset_prep.py - Train Model · Trains the classifier and exports runtime artifacts.
src/train_model.py
Online IDS
Live packet capture, scoring, enrichment, and response.
- Sniffer · Captures live network traffic from the wire.
src/sniffer.py - Live Detect · Extracts features, loads artifacts, and scores traffic in real time.
src/realtime_detect.py - Threat Intel · Checks suspicious IPs against a reputation service and adds context.
src/threat_intel.py - Response · Branches detections into alerts, logging, and optional blocking.
- Firewall Rule · Windows firewall enforcement for active blocking.
Local Storage
Filesystem-backed artifacts and runtime outputs.
- Model Store · Serialized inference artifacts consumed by the live detector.
models - RF Model · Saved Random Forest classifier used for scoring.
models/rf_model.joblib - Proto Encoder · Saved encoding/preprocessing artifact used at runtime.
models/proto_encoder.joblib - Packet Log · Persisted packet observations written for downstream processing.
- Alert Log · Persisted suspicious-event and alert records.
Dashboard
Human-facing monitoring interface.
- Dashboard · Visualizes live traffic, alerts, and threat intel.
dashboard/app.py
External Service
Third-party reputation lookup used during enrichment.
- AbuseIPDB · External IP reputation source used during enrichment.
Connections 19
- NSL-KDD → Prep Data: input · Training data is prepared for feature extraction and modeling.
- Prep Data → Train Model: prepared data · Prepared dataset feeds model training.
- Train Model → Model Store: exports · Training saves artifacts for live inference.
- Model Store → RF Model: contains · Serialized classifier artifact stored on disk.
- Model Store → Proto Encoder: contains · Serialized encoder/preprocessor stored on disk.
- Sniffer → Packet Log: writes · Captured traffic is persisted locally.
- Packet Log → Live Detect: feeds · Recorded observations are consumed by the detector.
- Sniffer → Live Detect: live stream · Raw packets flow into real-time scoring.
- RF Model → Live Detect: scores with · Classifier artifact is loaded during inference.
- Proto Encoder → Live Detect: preprocesses · Encoding artifact is loaded during inference.
- Live Detect → Threat Intel: suspicious ips · Suspicious detections are enriched with reputation data.
- Threat Intel → AbuseIPDB: lookup · Reputation checks are sent to the external service.
- Live Detect → Response: detections · Inference results flow into the response boundary.
- Threat Intel → Response: risk context · Enrichment informs alerting and blocking decisions.
- Response → Alert Log: alerts · Detected incidents are written for later review.
- Response → Firewall Rule: block · Optional active remediation applies host firewall rules.
- Alert Log → Dashboard: visualizes · Dashboard reads alerts and incident history.
- Packet Log → Dashboard: visualizes · Dashboard displays captured traffic summaries.
- Threat Intel → Dashboard: enriches view · Threat intelligence is surfaced in the UI.