# lokesh-kummari/network-security architecture

> Architecture diagram of the GitHub repository lokesh-kummari/network-security, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lokesh-kummari/network-security
- Repository: https://github.com/lokesh-kummari/network-security

## Overview

##### Project type and purpose
- This is a Python-based machine learning training and batch-prediction pipeline for phishing detection (network security use case).
- The code implements end-to-end stages: raw data ingestion, validation, transformation, model training, model artifact storage, batch inference, plus optional web UI and cloud synchronization.

##### File structure
Top-level scripts and folders:
- app.py / templates/table.html – lightweight web interface to display prediction results.
- main.py – orchestration entry point for training pipeline.
- push_data.py – utility to sync local artifacts to S3 (networksecurity/cloud/s3_syncer.py).
- networksecurity/components – four core components: data_ingestion.py, data_validation.py, data_transformation.py, model_trainer.py.
- networksecurity/pipelines – orchestration classes for training_pipeline.py and batch_prediction.py.
- networksecurity/utils/main_utils/ml_utils – helpers for metrics, estimators, general utilities.
- data_schema/schema.yaml – schema definition for raw CSV.
- final_model – holds final model.pkl and preprocessor.pkl.
- Network_data, valid_data, prediction_output – example input/output folders.
- logs – pipeline execution logs.
- setup.py, requirements.txt, Dockerfile – build and deployment.

##### Architecture patterns and dependencies
- Layered pipeline: each component (ingestion → validation → transformation → training) is independent, communicates via artifacts on disk or S3.
- CI/CD via GitHub Actions (/.github/workflows/main.yml).
- Cloud integration for artifact backup (S3 syncer).
- Separation of concerns: components, pipelines, utils, entities, exceptions, logging.

##### Main components and data flows
a. Data Sources
– Local CSV (“Network_data/phisingData.csv”) or S3 bucket.
b. Data Ingestion
– Reads raw CSV, writes feature_store and ingested/train.csv, ingested/test.csv.
c. Data Validation
– Validates ingested data against schema.yaml, generates drift_report (YAML) and validated/train.csv, validated/test.csv.
d. Data Transformation
– Loads validated CSVs, applies preprocessing pipeline (preprocessing.pkl), outputs transformed NumPy arrays.
e. Model Trainer
– Trains a classifier on transformed data, outputs model.pkl.
f. Artifact Management
– final_model and Artifacts/YYYY… directories hold timestamps of each run.
– push_data.py / S3 syncer pushes artifacts to cloud.
g. Batch Prediction Pipeline
– Invoked via networksecurity/pipelines/batch_prediction.py or app.py, reads new CSV, loads preprocessor+model, writes prediction_output/output.csv.
h. Web Interface (optional)
– app.py + templates/table.html serve output.csv as an HTML table.

##### Architectural principles
- Modular pipeline stages with clear input/output contracts.
- Artifact-driven orchestration—enables reproducibility and auditing.
- Separation of concerns:
– Components (pure processing logic) vs. pipelines (orchestration).
– Entities for config/artifact definitions.
– Utils for cross-cutting concerns (logging, metrics).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Training Orchestration
    main["main.py"]:::control
    TP["Training Pipeline Orchestrator"]:::component
    click main "https://github.com/lokesh-kummari/network-security/blob/main/main.py"
    click TP "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/pipelines/training_pipeline.py"
    main --> TP

    subgraph "Training Pipeline"
        direction TB
        A_source["Raw Data CSV\n(Network_data/phisingData.csv)"]:::storage
        B_ingest["Data Ingestion"]:::component
        C_feature_store["Feature Store\n(ingested CSVs)"]:::storage
        D_validation["Data Validation"]:::component
        drift["Drift Report\n(.yaml)"]:::storage
        E_validated["Validated Data\n(validated CSVs)"]:::storage
        F_trans["Data Transformation"]:::component
        transformed["Transformed Data\n(.npy)"]:::storage
        G_train["Model Trainer"]:::component
        H_model["Model Artifacts\n(model.pkl, preprocessor.pkl)"]:::storage

        click B_ingest "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/components/data_ingestion.py"
        click C_feature_store "https://github.com/lokesh-kummari/network-security/blob/main/Network_data/phisingData.csv"
        click D_validation "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/components/data_validation.py"
        click drift "https://github.com/lokesh-kummari/network-security/blob/main/data_schema/schema.yaml"
        click E_validated "https://github.com/lokesh-kummari/network-security/blob/main/Network_data/phisingData.csv"
        click F_trans "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/components/data_transformation.py"
        click transformed "https://github.com/lokesh-kummari/network-security/tree/main/Artifacts/"
        click G_train "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/components/model_trainer.py"
        click H_model "https://github.com/lokesh-kummari/network-security/blob/main/final_model/model.pkl"
        click H_model "https://github.com/lokesh-kummari/network-security/blob/main/final_model/preprocessor.pkl"

        A_source -->|"CSV"| B_ingest
        B_ingest -->|"train.csv/test.csv"| C_feature_store
        C_feature_store -->|"ingested CSVs"| D_validation
        D_validation -->|"drift_report.yaml"| drift
        D_validation -->|"train.csv/test.csv"| E_validated
        E_validated -->|"validated CSVs"| F_trans
        F_trans -->|".npy arrays"| transformed
        transformed -->|"features"| G_train
        G_train -->|".pkl artifacts"| H_model
    end

    %% S3 Sync
    H_model --> I_s3ync["S3 Sync Utility"]:::external
    click I_s3ync "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/cloud/s3_syncer.py"
    click I_s3ync "https://github.com/lokesh-kummari/network-security/blob/main/push_data.py"
    I_s3ync --> S3["S3 Bucket"]:::cloud

    %% Batch Prediction Path
    A_pred_source["New Data CSV"]:::storage
    click A_pred_source "https://github.com/lokesh-kummari/network-security/blob/main/Network_data/phisingData.csv"
    J_batch["Batch Prediction Pipeline"]:::component
    click J_batch "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/pipelines/batch_prediction.py"
    pred_output["Prediction Output\n(output.csv)"]:::storage
    click pred_output "https://github.com/lokesh-kummari/network-security/blob/main/prediction_output/output.csv"

    A_pred_source -->|"CSV"| J_batch
    J_batch -->|"predictions.csv"| pred_output

    %% Web Interface
    K_web["Web Interface (Flask)"]:::ui
    L_html["HTML Template\n(table.html)"]:::ui
    click K_web "https://github.com/lokesh-kummari/network-security/blob/main/app.py"
    click L_html "https://github.com/lokesh-kummari/network-security/blob/main/templates/table.html"
    pred_output -->|"reads CSV"| K_web
    K_web -->|"renders"| L_html

    %% Supporting Modules
    subgraph "Supporting Modules"
        direction TB
        Cfg["Config Entities"]:::util
        ArtEnt["Artifact Entities"]:::util
        Excp["Exception Handler"]:::util
        Const["Pipeline Constants"]:::util
        UtilsG["General Utils"]:::util
        ML["ML Utils"]:::util
        Logger["Logging"]:::util
        CI["CI/CD Workflow"]:::external
        Docker["Dockerfile"]:::external
        Setup["Setup & Requirements"]:::external

        click Cfg "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/entity/config_entity.py"
        click ArtEnt "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/entity/artifact_entity.py"
        click Excp "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/exception/exception.py"
        click Const "https://github.com/lokesh-kummari/network-security/tree/main/networksecurity/constants/training_pipeline/"
        click UtilsG "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/utils/main_utils/utils.py"
        click ML "https://github.com/lokesh-kummari/network-security/tree/main/networksecurity/utils/main_utils/ml_utils/"
        click Logger "https://github.com/lokesh-kummari/network-security/blob/main/networksecurity/logging/logger.py"
        click Logger "https://github.com/lokesh-kummari/network-security/tree/main/logs/"
        click CI "https://github.com/lokesh-kummari/network-security/blob/main/.github/workflows/main.yml"
        click Docker "https://github.com/lokesh-kummari/network-security/tree/main/Dockerfile"
        click Setup "https://github.com/lokesh-kummari/network-security/blob/main/setup.py"
        click Setup "https://github.com/lokesh-kummari/network-security/blob/main/requirements.txt"
    end

    classDef component fill:#cce5ff,stroke:#1f78b4,color:#1f78b4
    classDef storage fill:#d5f5e3,stroke:#27ae60,color:#27ae60
    classDef external fill:#fdebd0,stroke:#e67e22,color:#e67e22
    classDef cloud fill:#fadbd8,stroke:#c0392b,color:#c0392b
    classDef ui fill:#f0e5ff,stroke:#9b59b6,color:#9b59b6
    classDef util fill:#ebedef,stroke:#7f8c8d,color:#7f8c8d
    classDef control fill:#d6eaf8,stroke:#2980b9,color:#2980b9
```
