# ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation architecture

> Architecture diagram of the GitHub repository ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation
- Repository: https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation

## Overview

##### Project Type and Purpose
- This is an open-source, production-ready MLOps pipeline (inspired by TFX) focused on secure network-security data processing and model deployment.
- It ingests phishing (or other network) data from MongoDB, validates/transforms it, optionally trains and evaluates a model, then securely pushes artifacts (data, models, encrypted messages) to cloud storage (AWS S3, Azure Blob, GCS).

##### File Structure Analysis
Top-level:
- Dockerfile, .github/workflows/main.yml → CI/CD & containerization
- app.py, main.py, push_data.py → CLI/web entry points to trigger pipelines or push data
- requirements.txt, setup.py → Dependency/configuration
- Network_Data/, valid_data/, prediction_output/ → Sample/raw/processed data snapshots
- final_model/ → Pickled model & preprocessor artifacts
- templates/table.html → HTML report template

Core Python package: networksecurity/
- components/ → DataIngestion, DataValidation, DataTransformation, ModelTrainer
- pipeline/ → Orchestration scripts (training_pipeline, batch_prediction)
- cloud/ → s3_syncer for cloud push logic
- constant/, entity/, exception/, logging/, utils/ → Support modules for configs, entities, error handling, logging, generic and ML utilities

##### README Insights
- Describes six pipeline stages: ingestion → validation → transformation → training (optional) → evaluation → deployment
- Emphasizes DevSecOps best practices: artifact versioning, encryption, secure cloud upload
- Mentions configuration in YAML (data_schema/schema.yaml), traceable artifacts under an artifacts/ folder

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Source Layer
    subgraph "Data Sources" 
        direction TB
        MongoDB[(MongoDB)]:::cylinder
        LocalCSV[(Local CSV Files)]:::cylinder
    end

    %% Entry Points
    subgraph "Entrypoints (CLI/Web)" 
        direction TB
        App[/"app.py\nmain.py"/]:::compute
    end

    %% Orchestration Layer
    subgraph "Orchestration" 
        direction TB
        TrainPipe["training_pipeline.py"]:::compute
        BatchPipe["batch_prediction.py"]:::compute
    end

    %% Pipeline Components
    subgraph "Pipeline Stages" 
        direction TB
        Ingest["DataIngestion Service"]:::compute
        Validate["DataValidation Service"]:::compute
        Transform["DataTransformation Service\n(includes encryption)"]:::compute
        Decision{Model Approved?}:::decision
        Train["ModelTrainer Service"]:::compute
        Eval["ModelEvaluation Service"]:::compute
    end

    %% Artifact Store
    subgraph "Artifact Repository (Local FS)" 
        direction TB
        Artifacts[(artifacts/,\nfinal_model/, prediction_output/)]:::storage
    end

    %% Deployment Layer
    subgraph "Deployment" 
        direction TB
        Sync["Cloud Sync Service"]:::compute
    end

    %% External Cloud Providers
    subgraph "Cloud Storage Providers" 
        direction TB
        S3[(AWS S3)]:::cloud
        Azure[(Azure Blob)]:::cloud
        GCS[(GCP Bucket)]:::cloud
    end

    %% Cross-cutting Concerns
    subgraph "Cross-cutting Concerns" 
        direction TB
        Config["YAML Config Schema"]:::config
        Entity["Entity Modules"]:::config
        Logging["Logging Module"]:::config
        Exception["Exception Handling"]:::config
    end

    %% Data Flow Connections
    App -->|trigger| TrainPipe
    App -->|trigger| BatchPipe

    TrainPipe --> MongoDB
    TrainPipe --> LocalCSV

    TrainPipe --> Ingest
    Ingest -->|raw_data.json| Artifacts
    Artifacts --> Validate
    Validate -->|validation_report.json| Artifacts
    Artifacts --> Transform
    Transform -->|clean_encrypted_data.csv| Artifacts
    Artifacts --> Decision

    Decision --yes--> Train
    Train -->|model.pkl\npreprocessor.pkl| Artifacts
    Artifacts --> Eval
    Eval -->|evaluation_report.json| Artifacts
    Eval --approved--> Sync

    Sync --> S3
    Sync --> Azure
    Sync --> GCS

    %% Batch Prediction Flow
    BatchPipe --> Ingest
    Ingest -->|raw_batch.json| Artifacts
    Artifacts --> Transform
    Transform -->|batch_clean.csv| Artifacts
    Artifacts --> Sync

    %% Config and Logging Links
    Config --> Ingest
    Config --> Validate
    Config --> Transform
    Config --> Train
    Config --> Eval

    Entity --> Ingest
    Entity --> Validate

    Logging --> Ingest
    Logging --> Validate
    Logging --> Transform
    Logging --> Train
    Logging --> Eval

    Exception --> Ingest
    Exception --> Validate
    Exception --> Transform
    Exception --> Train
    Exception --> Eval

    %% Click Events
    click Ingest "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/components/data_ingestion.py"
    click Validate "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/components/data_validation.py"
    click Transform "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/components/data_transformation.py"
    click Train "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/components/model_trainer.py"
    click Eval "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/utils/ml_utils/metric/classification_metric.py"
    click TrainPipe "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/pipeline/training_pipeline.py"
    click BatchPipe "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/pipeline/batch_prediction.py"
    click App "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/app.py"
    click App "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/main.py"
    click Sync "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/cloud/s3_syncer.py"
    click Config "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/data_schema/schema.yaml"
    click Entity "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/entity/artifact_entity.py"
    click Entity "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/entity/config_entity.py"
    click Logging "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/logging/logger.py"
    click Exception "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/blob/main/networksecurity/exception/exception.py"
    click Exception "https://github.com/ridabayi/secureflow-a-modular-mlops-pipeline-for-network-security-automation/tree/main/networksecurity/exception/logs/"

    %% Styles
    classDef cylinder fill:#FFE4B5,stroke:#DAA520,stroke-width:2px
    classDef compute fill:#98FB98,stroke:#006400,stroke-width:2px
    classDef storage fill:#FFD700,stroke:#B8860B,stroke-width:2px
    classDef cloud fill:#D8BFD8,stroke:#800080,stroke-width:2px
    classDef decision fill:#FFDAB9,stroke:#FF8C00,stroke-width:2px,shape:diamond
    classDef config fill:#ADD8E6,stroke:#00008B,stroke-width:2px
```
