# ds-kushagra/network-security architecture

> Architecture diagram of the GitHub repository ds-kushagra/network-security, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ds-kushagra/network-security
- Repository: https://github.com/ds-kushagra/network-security

## Overview

##### Project Type and Purpose
- This is a machine-learning-driven, full-stack pipeline + micro-web-app for phishing detection.
- Core use cases:
- Offline training pipeline (main.py)
- Batch prediction pipeline (batch_prediction.py)
- Web interface for results (app.py + templates)
- Cloud sync of artifacts (s3_syncer.py)

##### File Structure
- Top-level scripts:
- main.py – orchestrates training pipeline
- app.py – Flask (or similar) web service for UI
- push_data.py – trigger for pushing new data or artifacts
- ML pipelines under networksecurity/pipeline:
- training_pipeline.py – end-to-end training
- batch_prediction.py – runs inference on new data
- Core components in networksecurity/components:
- data_ingestion.py
- data_validation.py
- data_transformation.py
- model_trainer.py
- Utilities & constants under networksecurity/utils and constants
- Schema (data_schema/schema.yaml) used by validation component
- Final_model/*.pkl – saved artifacts
- prediction_output/output.csv – inference results
- cloud sync logic in networksecurity/cloud/s3_syncer.py

##### README for Technical Stack
- Python, scikit-learn, GridSearchCV, YAML configs
- Logging to logs/ via networksecurity/logging/logger.py
- Exception handling via networksecurity/exception/exception.py
- Optional Dockerfile for containerization
- GitHub Actions workflow (CI) under .github/workflows

##### Main Components & Their Relationships
a. Data Sources & Storage
- Raw Data (Network_Data/phisingData.csv)
- Schema Definition (data_schema/schema.yaml)
- Validated Data (valid_data/)
- Artifacts (Final_model/, artifacts directory)
- Prediction Output (prediction_output/)
- Cloud Storage (AWS S3 – via s3_syncer)
b. Pipeline Orchestrator
- main.py invokes TrainingPipeline
- batch_prediction.py invokes BatchPredictionPipeline
c. Components
- DataIngestion → DataValidation → DataTransformation → ModelTraining
- Utils & Constants feed into all components
d. Web Service
- app.py serves UI, reads output.csv, renders templates/table.html
e. External & DevOps
- Docker container wraps app & pipeline
- CI (GitHub Actions) triggers tests/builds
- s3_syncer pushes/pulls model artifacts to/from AWS S3

##### Architectural Patterns & Principles
- Modular pipeline: each step in its own component with clear inputs/outputs
- Configuration-driven via YAML (schema.yaml) and constants modules
- Separation of concerns: ingestion vs. validation vs. transformation vs. training
- Layered structure: core logic under networksecurity/, scripts at root, data & models separated
- Fault tolerance: centralized logging + exception handling

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Cross-cutting concerns
    subgraph "Cross-Cutting Concerns" 
        direction TB
        Logger["Logger Module"]:::util
        Exception["Exception Module"]:::util
    end

    %% Data Stores and Sources
    RawData["Raw Data\n(phisingData.csv)"]:::data
    Schema["Schema Definition\n(schema.yaml)"]:::data
    ValidatedData["Validated Data\n(valid_data/)"]:::data
    Artifacts["Model Artifacts\n(model.pkl, preprocessor.pkl)"]:::data
    PredictionOutput["Prediction Output\n(output.csv)"]:::data
    S3["AWS S3"]:::external

    %% Offline Training Pipeline
    subgraph "Offline Training Pipeline" 
        direction TB
        MainScript["main.py"]:::external
        TrainingPipeline["TrainingPipeline"]:::pipeline
        DataIngestion["Data Ingestion"]:::pipeline
        DataValidation["Data Validation"]:::pipeline
        DataTransformation["Data Transformation"]:::pipeline
        ModelTraining["Model Training"]:::pipeline
    end

    %% Batch Prediction Flow
    subgraph "Batch Prediction Pipeline"
        direction TB
        BatchScript["batch_prediction.py"]:::external
        BatchPipeline["BatchPredictionPipeline"]:::pipeline
    end

    %% Web Service
    subgraph "Web Service"
        direction TB
        App["app.py"]:::external
        Template["UI Template\n(table.html)"]:::external
    end

    %% Utilities & Configs
    Constants["Pipeline Constants"]:::util
    UtilsGeneral["General Utils"]:::util
    UtilsML["ML Utils"]:::util
    Docker["Dockerfile"]:::external
    CI["GitHub Actions\n(main.yaml)"]:::external
    S3Sync["S3 Sync Service"]:::external

    %% Offline Training Flow
    RawData -->|ingest| DataIngestion
    Schema -->|validate| DataValidation
    DataIngestion --> DataValidation
    DataValidation --> DataTransformation
    DataTransformation --> ModelTraining
    ModelTraining --> Artifacts
    Artifacts -->|sync| S3Sync
    MainScript -->|triggers| TrainingPipeline
    TrainingPipeline --> DataIngestion
    TrainingPipeline --> DataValidation
    TrainingPipeline --> DataTransformation
    TrainingPipeline --> ModelTraining
    TrainingPipeline --> Logger
    TrainingPipeline --> Exception
    TrainingPipeline --> Constants
    TrainingPipeline --> UtilsGeneral
    TrainingPipeline --> UtilsML

    %% Batch Prediction Flow
    BatchScript -->|triggers| BatchPipeline
    Schema -->|validate| BatchPipeline
    BatchPipeline --> PredictionOutput
    BatchPipeline --> Logger
    BatchPipeline --> Exception
    BatchPipeline --> Constants
    BatchPipeline --> UtilsGeneral
    BatchPipeline --> UtilsML
    PredictionOutput --> App

    %% Web UI Flow
    App --> Template
    App --> Logger
    App --> Exception
    App --> PredictionOutput

    %% Cloud Sync
    S3Sync --> S3
    Artifacts --> S3Sync

    %% DevOps
    Docker --> MainScript
    Docker --> App
    CI --> Docker
    CI --> MainScript
    CI --> App

    %% Styles
    classDef pipeline fill:#ADD8E6,stroke:#000;
    classDef data fill:#90EE90,stroke:#000;
    classDef external fill:#FFA500,stroke:#000;
    classDef util fill:#D3D3D3,stroke:#000;

    %% Click Events
    click MainScript "https://github.com/ds-kushagra/network-security/blob/main/main.py"
    click BatchScript "https://github.com/ds-kushagra/network-security/blob/main/networksecurity/pipeline/batch_prediction.py"
    click App "https://github.com/ds-kushagra/network-security/blob/main/app.py"
    click Template "https://github.com/ds-kushagra/network-security/blob/main/templates/table.html"
    click S3Sync "https://github.com/ds-kushagra/network-security/blob/main/networksecurity/cloud/s3_syncer.py"
    click DataIngestion "https://github.com/ds-kushagra/network-security/blob/main/networksecurity/components/data_ingestion.py"
    click DataValidation "https://github.com/ds-kushagra/network-security/blob/main/networksecurity/components/data_validation.py"
    click DataTransformation "https://github.com/ds-kushagra/network-security/blob/main/networksecurity/components/data_transformation.py"
    click ModelTraining "https://github.com/ds-kushagra/network-security/blob/main/networksecurity/components/model_tranier.py"
    click TrainingPipeline "https://github.com/ds-kushagra/network-security/blob/main/networksecurity/pipeline/training_pipeline.py"
    click Schema "https://github.com/ds-kushagra/network-security/blob/main/data_schema/schema.yaml"
    click RawData "https://github.com/ds-kushagra/network-security/blob/main/Network_Data/phisingData.csv"
    click ValidatedData "https://github.com/ds-kushagra/network-security/tree/main/valid_data/"
    click Artifacts "https://github.com/ds-kushagra/network-security/blob/main/Final_model/model.pkl"
    click PredictionOutput "https://github.com/ds-kushagra/network-security/blob/main/prediction_output/output.csv"
    click Logger "https://github.com/ds-kushagra/network-security/blob/main/networksecurity/logging/logger.py"
    click Exception "https://github.com/ds-kushagra/network-security/blob/main/networksecurity/exception/exception.py"
    click Constants "https://github.com/ds-kushagra/network-security/tree/main/networksecurity/constants/training_pipeline/"
    click UtilsGeneral "https://github.com/ds-kushagra/network-security/blob/main/networksecurity/utils/main_utils/utils.py"
    click UtilsML "https://github.com/ds-kushagra/network-security/tree/main/networksecurity/utils/ml_utils/"
    click Docker "https://github.com/ds-kushagra/network-security/tree/main/Dockerfile"
    click CI "https://github.com/ds-kushagra/network-security/blob/main/.github/workflows/main.yaml"
```
