# katiie104/ids-using-machinelearning architecture

> Architecture diagram of the GitHub repository katiie104/ids-using-machinelearning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/katiie104/ids-using-machinelearning
- Repository: https://github.com/katiie104/ids-using-machinelearning

## Overview

##### Project type and purpose
- This is a real-time Network Intrusion Detection System (IDS) using machine learning.
- It has two distinct modes:
a. Offline model training (batch pipeline)
b. Real-time packet capture and inference

##### File structure
Top-level:
- main.py – entry point that wires together training and monitoring
- dataset/ – raw NSL-KDD training data
- logs/ – application logs
- requirements.txt – dependencies
- src/ – core modules
src/ modules:
- config.py – central configuration (paths, hyperparameters)
- preprocess.py – data cleaning, feature encoding, scaling
- train_model.py – XGBoost training with GridSearchCV, model persistence
- explain_model.py – SHAP explanations for feature importance
- stream_monitor.py – real-time capture via pyshark, feature extraction, inference
- zeek_feature_extractor.py – optional feature extraction from Zeek logs

##### Extract key architectural insights from README
- Offline pipeline: read KDDTrain → preprocess → train XGBoost → save preprocessor & model
- Online pipeline: capture packets from network interface → extract same features → apply model → output Normal/Attack
- Dependencies: pandas, scikit-learn, xgboost, pyshark, joblib

##### Main components
A. Data Sources
1. Historical dataset (NSL-KDD files)
2. Live network interface (via pyshark)
B. Offline Pipeline
1. Preprocessing module
2. Model training module
3. Model storage (joblib files in models/)
4. Model explanation module (SHAP)
C. Online Pipeline
1. Packet capture (pyshark.LiveCapture)
2. Feature extractor (zeek_feature_extractor / inline extractor)
3. Preprocessor (load preprocessor.pkl)
4. Inference engine (load xgb_model.pkl)
5. Alerting / terminal output
D. Supporting services
1. Configuration manager (config.py)
2. Logging service (logs/app.log)

##### Relationships and data flows
- Configuration fed into both pipelines
- Logs written by all modules

##### Architectural patterns and principles
- Separation of concerns: offline vs online, preprocessing vs inference
- Modular design: each major step in its own module
- Reusability: same preprocessing logic in training and live monitoring
- Extensibility: plugging new feature extractors (Zeek) or models

##### Technologies, frameworks, and libraries
- Python modules: pandas, scikit-learn, xgboost, joblib, pyshark
- Logging: Python logging module writing to logs/app.log
- Model explainability: SHAP via explain_model.py

##### Final recommendation
- Capturing both the offline ML pipeline and the online inference pipeline as two parallel streams that share preprocessing logic and configuration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration and Entry
    Config["Configuration Manager\n(src/config.py)"]:::config
    Entry["Entry Point\n(main.py)"]:::entry
    Requirements["Dependencies\n(requirements.txt)"]:::entry
    LogsStore["Logging Service\n(logs/app.log)"]:::logs

    %% Training Environment
    subgraph "Training Environment"
        direction TB
        Dataset["Historical Dataset\n(NSL-KDD files)"]:::storage
        PreprocessorTrain["Preprocessing Module\n(src/preprocess.py)"]:::offline
        Trainer["Model Training Module\n(src/train_model.py)"]:::offline
        ModelStore["Model Storage\n(models/*.pkl)"]:::storage
        Explain["Model Explanation Module\n(src/explain_model.py)"]:::offline
    end

    %% Production/Monitoring Environment
    subgraph "Production/Monitoring Environment"
        direction TB
        LiveCapture["Live Network Interface\n(src/stream_monitor.py)"]:::online
        ZeekExtractor["Zeek Feature Extractor\n(src/zeek_feature_extractor.py)"]:::online
        PreprocessorOnline["Preprocessor (load preprocessor.pkl)\n(src/preprocess.py)"]:::online
        Inference["Inference Engine\n(load xgb_model.pkl)"]:::online
        Alert["Alert Sink\n(Terminal / UI)"]:::online
    end

    %% Data Stores
    Dataset -->|"raw data"| PreprocessorTrain
    PreprocessorTrain -->|"cleaned features"| Trainer
    Trainer -->|"trained model"| ModelStore
    Trainer -->|"preprocessor.pkl"| ModelStore
    ModelStore <-->|"retrieve model/preprocessor"| PreprocessorOnline
    ModelStore <-->|"retrieve model/preprocessor"| Inference
    PreprocessorOnline -->|"features"| Inference
    LiveCapture -->|"packets"| ZeekExtractor
    ZeekExtractor -->|"extracted features"| PreprocessorOnline
    Inference -->|"Normal/Attack"| Alert

    %% Explanation Flow
    Trainer -.->|"feature importance"| Explain

    %% Configuration Flows
    Config -.-> Entry
    Config -.-> PreprocessorTrain
    Config -.-> Trainer
    Config -.-> PreprocessorOnline
    Config -.-> Inference
    Config -.-> ZeekExtractor

    %% Logging
    Entry --> LogsStore
    Dataset --> LogsStore
    PreprocessorTrain --> LogsStore
    Trainer --> LogsStore
    Explain --> LogsStore
    LiveCapture --> LogsStore
    ZeekExtractor --> LogsStore
    PreprocessorOnline --> LogsStore
    Inference --> LogsStore
    Alert --> LogsStore

    %% Entry wiring
    Entry --> Dataset
    Entry --> PreprocessorTrain
    Entry --> Trainer
    Entry --> Explain
    Entry --> LiveCapture

    %% Dependencies
    Requirements -.-> Entry
    Requirements -.-> PreprocessorTrain
    Requirements -.-> Trainer
    Requirements -.-> Explain
    Requirements -.-> LiveCapture
    Requirements -.-> ZeekExtractor
    Requirements -.-> PreprocessorOnline
    Requirements -.-> Inference

    %% Click Events
    click Dataset "https://github.com/katiie104/ids-using-machinelearning/tree/main/dataset/NSL-KDD-Dataset"
    click LiveCapture "https://github.com/katiie104/ids-using-machinelearning/blob/main/src/stream_monitor.py"
    click PreprocessorTrain "https://github.com/katiie104/ids-using-machinelearning/blob/main/src/preprocess.py"
    click PreprocessorOnline "https://github.com/katiie104/ids-using-machinelearning/blob/main/src/preprocess.py"
    click Trainer "https://github.com/katiie104/ids-using-machinelearning/blob/main/src/train_model.py"
    click Explain "https://github.com/katiie104/ids-using-machinelearning/blob/main/src/explain_model.py"
    click ZeekExtractor "https://github.com/katiie104/ids-using-machinelearning/blob/main/src/zeek_feature_extractor.py"
    click Config "https://github.com/katiie104/ids-using-machinelearning/blob/main/src/config.py"
    click LogsStore "https://github.com/katiie104/ids-using-machinelearning/blob/main/logs/app.log"
    click Entry "https://github.com/katiie104/ids-using-machinelearning/blob/main/main.py"
    click Requirements "https://github.com/katiie104/ids-using-machinelearning/blob/main/requirements.txt"

    %% Styles
    classDef offline fill:#cfe2f3,stroke:#000,stroke-width:1px;
    classDef online fill:#d9ead3,stroke:#000,stroke-width:1px;
    classDef storage fill:#f9cb9c,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5;
    classDef config fill:#d9d2e9,stroke:#000,stroke-width:1px;
    classDef logs fill:#ececec,stroke:#000,stroke-width:1px;
    classDef entry fill:#fff2cc,stroke:#000,stroke-width:1px;
```
