# wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system architecture

> Architecture diagram of the GitHub repository wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system
- Repository: https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system

## Overview

##### Project Type and Purpose
- This is an end-to-end MLOps pipeline for network security (phishing detection).
- It comprises an ETL (data ingestion → validation → transformation), model training & tracking, and deployment on AWS (EC2/ECR/S3), plus CI/CD via GitHub Actions.

##### File Structure
- Top-level scripts:
- main.py – orchestrates training pipeline
- app.py – runs inference API/UI
- push_data.py/test_mongodb.py – data loading & DB tests
- Core package “networksecurity/”:
- components/: data_ingestion.py, data_validation.py, data_transformation.py, model_trainer.py
- pipeline/: training_pipeline.py, batch_prediction.py
- cloud/: s3_syncer.py
- utils/, constant/, entity/, exception/, logging/
- Artifacts & final_model folders hold intermediate and final outputs.
- Dockerfile + .github/workflows/main.yml define containerization & CI/CD.

##### README Insights
- Data source: MongoDB Atlas + local phishingData.csv
- MLFlow + Dagshub for experiment tracking
- AWS S3/ECR for artifacts, EC2 for deployment
- Etl pipeline: ingestion → validation → transformation → training → deploy

##### Main Components & Relationships
A. Data Sources
- External: MongoDB Atlas
- Repository: Network_Data/phisingData.csv
B. ETL Pipeline Components (networksecurity/components)
1. Data Ingestion → Reads CSV or MongoDB → writes raw to feature_store & ingested/
2. Data Validation → Loads ingested → drift_report & validated/
3. Data Transformation → Loads validated → produces transformed .npy + preprocessing.pkl
C. Model Training (networksecurity/components/model_trainer + pipeline/training_pipeline)
- Consumes transformed data → trains model → hyperparameter tuning
- Logs metrics & params to local MLFlow + remote Dagshub
- Outputs final_model/model.pkl & preprocessor.pkl
D. Artifact Storage & Cloud Sync
- Local Artifacts/…
- s3_syncer pushes key folders (feature_store, models, logs) to AWS S3/ECR
E. Deployment
- app.py (Flask/FastAPI) loads model.pkl + preprocessor.pkl → exposes REST API or web UI (templates/table.html)
- batch_prediction.py for bulk inference
- pack via setup.py, containerize with Dockerfile, push to AWS ECR
- Deploy container on AWS EC2
F. CI/CD
- .github/workflows/main.yml builds, tests, pushes Docker image, triggers s3 sync & EC2 deployment

##### Architectural Patterns & Design Principles
- Layered ETL → clear separation of ingestion/validation/transformation
- Modular components (each step in its own class/module)
- Pipeline orchestration via main.py/training_pipeline.py
- Experiment tracking abstraction (MLFlow + remote)
- Infrastructure as code via Docker + GitHub Actions
- Cloud-native deployment (S3, ECR, EC2)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        MongoDB[(MongoDB Atlas)]:::external
        CSV["Network_Data/phisingData.csv"]:::external
    end

    %% ETL Layer
    subgraph "ETL Layer"
        Ingestion["Data Ingestion Service"]:::etl
        Validation["Data Validation Service"]:::etl
        Transformation["Data Transformation Service"]:::etl
        MainEntry["Main Pipeline Entrypoint"]:::etl
    end

    %% Training Layer
    subgraph "Training Layer"
        Pipeline["Training Pipeline Orchestrator"]:::training
        Trainer["Model Trainer Component"]:::training
        MLFlow[(MLFlow / Dagshub)]:::external
        Model["final_model/model.pkl"]:::artifact
        Preprocessor["final_model/preprocessor.pkl"]:::artifact
    end

    %% Serving Layer
    subgraph "Serving Layer"
        API["Inference API / Web App"]:::serving
        Template["API UI Template"]:::serving
        Batch["Batch Prediction Pipeline"]:::serving
    end

    %% Cloud Infrastructure
    subgraph "Cloud Infrastructure"
        S3[(AWS S3)]:::cloud
        ECR[(AWS ECR)]:::cloud
        EC2[(AWS EC2)]:::cloud
        Sync["Cloud Sync Utility"]:::cloud
    end

    %% CI/CD Pipeline
    subgraph "CI/CD Pipeline"
        Workflow["CI/CD Workflow Definition"]:::cicd
        DockerfileNode["Docker Container Definition"]:::cicd
        Setup["Package Configuration"]:::cicd
    end

    %% Data Flow
    MongoDB -->|"read data"| Ingestion
    CSV -->|"read CSV"| Ingestion
    Ingestion -->|"write raw data"| Validation
    Validation -->|"write validated data"| Transformation
    Transformation -->|"produce transformed data"| Pipeline
    MainEntry -->|"orchestrate pipeline"| Pipeline
    Pipeline -->|"call ModelTrainer"| Trainer
    Trainer -->|"log metrics"| MLFlow
    Trainer -->|"output model"| Model
    Trainer -->|"output preprocessor"| Preprocessor

    %% Deployment Flow
    Model -->|"sync to S3/ECR"| Sync
    Preprocessor -->|"sync to S3/ECR"| Sync
    Sync -->|"store artifacts"| S3
    DockerfileNode -->|"build image"| ECR
    ECR -->|"pull image"| EC2
    EC2 -->|"run service"| API
    API -->|"load model & preprocessor"| Model
    API -->|"serve UI"| Template
    Batch -->|"bulk inference"| Model

    %% CI/CD Flow
    Workflow -->|"run tests & build"| DockerfileNode
    Workflow -->|"deploy artifacts"| Sync
    Workflow -->|"deploy container"| EC2
    Setup -->|"install package"| DockerfileNode

    %% Click Events
    click Ingestion "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/networksecurity/components/data_ingestion.py"
    click Validation "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/networksecurity/components/data_validation.py"
    click Transformation "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/networksecurity/components/data_transformation.py"
    click Trainer "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/networksecurity/components/model_trainer.py"
    click Pipeline "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/networksecurity/pipeline/training_pipeline.py"
    click Batch "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/networksecurity/pipeline/batch_prediction.py"
    click Sync "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/networksecurity/cloud/s3_syncer.py"
    click API "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/app.py"
    click Template "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/templates/table.html"
    click MainEntry "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/main.py"
    click Workflow "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/.github/workflows/main.yml"
    click DockerfileNode "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/tree/main/Dockerfile"
    click CSV "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/Network_Data/phisingData.csv"
    click Setup "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/setup.py"
    click Model "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/final_model/model.pkl"
    click Preprocessor "https://github.com/wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system/blob/main/final_model/preprocessor.pkl"

    %% Styles
    classDef etl fill:#a3c4dc,stroke:#333,stroke-width:1px
    classDef training fill:#b5e48c,stroke:#333,stroke-width:1px
    classDef serving fill:#ffb703,stroke:#333,stroke-width:1px
    classDef cloud fill:#cdb4db,stroke:#333,stroke-width:1px
    classDef cicd fill:#dee2e6,stroke:#333,stroke-width:1px
    classDef external fill:#f0efeb,stroke:#333,stroke-width:1px
    classDef artifact fill:#ef233c,stroke:#333,stroke-width:1px
```
