# danish0511/network_security architecture

> Architecture diagram of the GitHub repository danish0511/network_security, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/danish0511/network_security
- Repository: https://github.com/danish0511/network_security

## Overview

##### Project Type and Purpose
- This is a machine‐learning orchestration project (a full “ML pipeline” application) for network security data.
- It provides end-to-end functionality: data ingestion, validation, transformation, model training, evaluation, and deployment—coordinated via Apache Airflow and tracked with MLflow.
- It can perform scheduled batch predictions and push “production” models to storage (local or S3), and offers a Jupyter notebook for EDA.

##### Key Architectural Components
a. Data Sources
- Raw CSV files under Network_Data/
- External data (potentially fetched via get_data.py)
b. Orchestration Layer
- Airflow (airflow/dags/training_pipeline.py, batch_prediction.py)
- Docker Compose + start.sh to spin up Airflow, MLflow server, etc.
c. Pipeline Modules (networksecurity/components)
- DataIngestion → DataValidation → DataTransformation → ModelTrainer → ModelEvaluation → ModelPusher
d. Metadata & Config
- data_schema/schema.yaml
- networksecurity/constant/… and networksecurity/entity/… for config artifacts
e. Experiment Tracking & Model Registry
- MLflow (mlruns/)
- networksecurity/cloud/s3_syncer.py pushes artifacts/models to S3
f. Persisted Models
- saved_models/…/model.pkl
g. User Interaction
- Jupyter Notebook (Notebook/EDA.ipynb)
- Potential HTML report (templates/table.html)
- CLI entrypoints: main.py, start_training.py, test.py

1. Data Layer (leftmost)
2. Orchestration Layer (Airflow)
3. Compute/Processing Layer (pipeline)
4. Storage & Registry Layer (MLflow, S3, local FS)
5. User & Reporting Layer (Notebook, HTML templates)
- Rectangles for compute modules & DAGs
Raw Data → DataIngestion → DataValidation → DataTransformation → ModelTrainer → ModelEvaluation → ModelPusher → (SavedModels & MLflow & S3)
Airflow DAG (training) orchestrates all pipeline steps.
Airflow DAG (batch_prediction) reads deployed model & raw input → Prediction → Output storage or report.
User (Notebook) reads from raw & model registry to produce EDA reports.
- Blue = Data sources
- Orange = Orchestration (Airflow)
- Green = Pipeline compute
- Purple = Storage & tracking
- Gray = User interfaces

##### Relationships & Interactions
- Airflow (orange) invokes Python modules in networksecurity/pipeline and networksecurity/components
- The components read/write local file system (Network_Data/ & saved_models/) and MLflow tracking server (mlruns/)
- ModelPusher uses s3_syncer to upload models/artifacts to S3
- Notebook (user) pulls raw data and model artifacts from local FS or S3 for analysis
- Docker Compose wraps Airflow, MLflow server, and mounts volumes for mlruns/ & saved_models/

##### Architectural Patterns & Principles
- Modular “component‐based” ML pipeline: each step separately testable and reusable
- Orchestration via DAGs in Airflow to manage dependencies and scheduling
- Centralized experiment tracking with MLflow
- Separation of concerns: ingestion, validation, transformation, training, evaluation, deployment
- Infrastructure as code: DockerFile + docker-compose.yaml + start.sh for reproducible environments

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RawCSV["Raw CSV Files"]:::data
        getData["External Data Fetcher"]:::data
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        DockerCompose["Docker Compose"]:::orchestration
        StartScript["start.sh"]:::orchestration
        Dockerfile["DockerFile"]:::orchestration
        AirflowTrain["Training DAG"]:::orchestration
        AirflowBatch["Batch Prediction DAG"]:::orchestration
    end

    %% Compute/Processing Layer
    subgraph "Compute Layer"
        direction TB
        TrainWrapper["Training Pipeline Wrapper"]:::compute
        BatchWrapper["Batch Prediction Wrapper"]:::compute
        DataIngestion["DataIngestion Module"]:::compute
        DataValidation["DataValidation Module"]:::compute
        DataTransformation["DataTransformation Module"]:::compute
        ModelTrainer["ModelTrainer Module"]:::compute
        ModelEvaluation["ModelEvaluation Module"]:::compute
        ModelPusher["ModelPusher Module"]:::compute
        S3Syncer["S3 Sync Utility"]:::compute
    end

    %% Storage & Registry Layer
    subgraph "Storage & Registry Layer"
        direction TB
        MLflowDB["MLflow Tracking Server"]:::storage
        MLRuns["mlruns Directory"]:::storage
        SavedModels["Saved Models FS"]:::storage
        S3Bucket["S3 Bucket"]:::storage
    end

    %% User & Reporting Layer
    subgraph "User & Reporting Layer"
        direction TB
        Notebook["EDA Notebook"]:::user
        HTMLReport["HTML Report Template"]:::user
        CLIGroup["CLI Entrypoints"]:::user
        MainScript["main.py"]:::user
        StartTrainCLI["start_training.py"]:::user
        TestCLI["test.py"]:::user
    end

    %% Data Flow
    RawCSV -->|provides data| DataIngestion
    getData -->|fetches data| DataIngestion

    %% Orchestration invoking wrappers
    DockerCompose --> StartScript
    StartScript --> Dockerfile
    DockerCompose --> AirflowTrain
    DockerCompose --> AirflowBatch
    AirflowTrain -->|triggers| TrainWrapper
    AirflowBatch -->|triggers| BatchWrapper

    %% Pipeline flow
    TrainWrapper --> DataIngestion
    DataIngestion --> DataValidation
    DataValidation --> DataTransformation
    DataTransformation --> ModelTrainer
    ModelTrainer --> ModelEvaluation
    ModelEvaluation --> ModelPusher

    %% Model deployment
    ModelPusher --> SavedModels
    ModelPusher --> MLflowDB
    ModelPusher --> S3Syncer
    S3Syncer --> S3Bucket

    %% Batch prediction
    BatchWrapper --> SavedModels
    BatchWrapper --> DataIngestion
    BatchWrapper -->|"writes predictions"| SavedModels

    %% Experiment tracking
    MLflowDB --> MLRuns

    %% User interactions
    Notebook --> RawCSV
    Notebook --> MLRuns
    Notebook --> SavedModels
    Notebook --> S3Bucket
    CLIGroup --> MainScript
    CLIGroup --> StartTrainCLI
    CLIGroup --> TestCLI
    MainScript --> TrainWrapper
    StartTrainCLI --> TrainWrapper
    TestCLI --> TrainWrapper

    %% Click Events
    click RawCSV "https://github.com/danish0511/network_security/blob/master/Network_Data/NetworkData.csv"
    click getData "https://github.com/danish0511/network_security/blob/master/get_data.py"
    click AirflowTrain "https://github.com/danish0511/network_security/blob/master/airflow/dags/training_pipeline.py"
    click AirflowBatch "https://github.com/danish0511/network_security/blob/master/airflow/dags/batch_prediction.py"
    click DockerCompose "https://github.com/danish0511/network_security/blob/master/docker-compose.yaml"
    click StartScript "https://github.com/danish0511/network_security/blob/master/start.sh"
    click Dockerfile "https://github.com/danish0511/network_security/tree/master/DockerFile"
    click DataIngestion "https://github.com/danish0511/network_security/blob/master/networksecurity/components/data_ingestion.py"
    click DataValidation "https://github.com/danish0511/network_security/blob/master/networksecurity/components/data_validation.py"
    click DataTransformation "https://github.com/danish0511/network_security/blob/master/networksecurity/components/data_transformation.py"
    click ModelTrainer "https://github.com/danish0511/network_security/blob/master/networksecurity/components/model_trainer.py"
    click ModelEvaluation "https://github.com/danish0511/network_security/blob/master/networksecurity/components/model_evaluation.py"
    click ModelPusher "https://github.com/danish0511/network_security/blob/master/networksecurity/components/model_pusher.py"
    click TrainWrapper "https://github.com/danish0511/network_security/blob/master/networksecurity/pipeline/training_pipeline.py"
    click BatchWrapper "https://github.com/danish0511/network_security/blob/master/networksecurity/pipeline/batch_prediction.py"
    click MLRuns "https://github.com/danish0511/network_security/tree/master/mlruns/"
    click S3Syncer "https://github.com/danish0511/network_security/blob/master/networksecurity/cloud/s3_syncer.py"
    click SavedModels "https://github.com/danish0511/network_security/tree/master/saved_models/"
    click Notebook "https://github.com/danish0511/network_security/blob/master/Notebook/EDA.ipynb"
    click HTMLReport "https://github.com/danish0511/network_security/blob/master/templates/table.html"
    click MainScript "https://github.com/danish0511/network_security/blob/master/main.py"
    click StartTrainCLI "https://github.com/danish0511/network_security/blob/master/start_training.py"
    click TestCLI "https://github.com/danish0511/network_security/blob/master/test.py"

    %% Styles
    classDef data fill:#E0F7FA,stroke:#006064,color:#006064
    classDef orchestration fill:#FFE0B2,stroke:#E65100,color:#E65100
    classDef compute fill:#C8E6C9,stroke:#2E7D32,color:#2E7D32
    classDef storage fill:#E1BEE7,stroke:#6A1B9A,color:#6A1B9A
    classDef user fill:#ECEFF1,stroke:#455A64,color:#455A64
```
