# rajeshraiml01/networksecurity-mlops architecture

> Architecture diagram of the GitHub repository rajeshraiml01/networksecurity-mlops, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rajeshraiml01/networksecurity-mlops
- Repository: https://github.com/rajeshraiml01/networksecurity-mlops

## Overview

##### Project type and purpose
- This is a machine‐learning–driven network‐security tool focused on phishing detection.
- It provides both:
- A model‐training pipeline (data ingestion → validation → transformation → training).
- A batch/prediction pipeline and a simple Flask‐style web app to display results.

##### File structure
- Top-level orchestration:
- app.py / main.py – entry points for the web app or CLI.
- push_data.py – script to upload or sync data.
- test_mongodb.py – hints at a possible MongoDB integration for logging or storage.
- Core package: networksecurity/
- components/ – data_ingestion, data_validation, transformation, model_trainer.
- pipeline/ – training_pipeline.py and batch_prediction.py orchestrate end-to-end flows.
- utils/ – main_utils for generic helpers; ml_utils for metrics & estimator abstraction.
- cloud/ – s3_syncer.py for AWS S3 interactions.
- constant/ and entity/ – configuration & typed artifacts for pipeline stages.
- logging/ – centralized logger.
- Artifacts and data:
- network_data/phisingData.csv & valid_data/test.csv – raw/validation datasets.
- final_model/{model.pkl, preprocessor.pkl} – trained artifacts.
- prediction_output/output.csv – batch predictions.
- DevOps:
- Dockerfile – container definition.
- .github/workflows/main.yml – CI/CD pipeline.
- setup.py & requirements.txt – packaging and dependencies.
- README.md – AWS ECR, S3 and EC2 Docker deployment instructions.

##### Key insights from README
- Containerized deployment to AWS EC2 (Docker), artifacts and data stored/synced in S3.
- Docker images pushed to ECR (AWS ECR URI in secrets).
- No mention of a complex microservices mesh—single‐container ML service.

a. Main components
1. Data Sources:
– Local CSVs (network_data, valid_data).
– Optionally external sources synced via networksecurity/cloud/s3_syncer.
2. ML Pipeline:
– Data Ingestion → Validation → Transformation → Model Trainer
– Artifacts stored (preprocessor.pkl & model.pkl)
3. Batch Prediction:
– Loads artifacts → Runs on new data → Outputs CSV.
4. Web App (Flask):
– app.py / main.py serve HTML (templates/table.html) for user to upload/run predictions.
5. Cloud/Infra:
– Docker container running on EC2 (Dockerfile)
– AWS S3 for data/artifact storage
– AWS ECR for container registry
– GitHub Actions for CI/CD

b. Relationships & data flow
- From Artifacts → Batch Prediction → Prediction Output.
- The Web App consumes Prediction Output or directly invokes batch_prediction.
- GitHub Actions builds the Docker image → pushes to ECR → deploys to EC2.
- S3 Syncer syncs raw data or artifacts on schedule or on pipeline completion.

c. Architectural patterns & design principles
- Modular pipeline: single‐responsibility components (ingest, validate, transform, train).
- Artifact‐driven: clear separation between training and inference.
- Infrastructure as code / CI‐CD: Docker + GitHub Actions + ECR + EC2.
- Utility layers: central logging, config & entity definitions for type safety.

d. Technologies & frameworks
- Python for core logic, scikit-learn for modeling (implied by .pkl), Flask (templates).
- AWS services: S3, ECR, EC2.
- Docker for containerization.
- GitHub Actions for CI/CD.

##### Essential detail reminders
- Both ML pipeline and deployment pipeline.
- How utilities (logging, config) weave into each component.
- Do not omit the S3 syncer and GitHub Actions steps—they ensure end-to-end reproducibility.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Data Sources"
        DS1["phisingData.csv"]:::datastore
        DS2["test.csv"]:::datastore
        S3["S3 Bucket"]:::datastore
    end

    %% ML Pipeline - Training
    subgraph "ML Pipeline"
        subgraph "Training Flow"
            DI["Data Ingestion"]:::compute
            DV["Data Validation"]:::compute
            DT["Data Transformation"]:::compute
            MT["Model Trainer"]:::compute
            TP["Training Pipeline"]:::compute
        end
    end

    %% Artifacts
    subgraph "Artifacts Store"
        Model["model.pkl"]:::datastore
        Preprocessor["preprocessor.pkl"]:::datastore
    end

    %% Inference
    subgraph "Batch Prediction"
        BP["Batch Prediction Pipeline"]:::compute
        Output["output.csv"]:::datastore
    end

    %% Web App
    subgraph "Web Application"
        Flask["Flask App"]:::ui
        Template["Result Template (table.html)"]:::ui
    end

    %% DevOps
    subgraph "DevOps / CI-CD"
        Code["Source Code"]:::devops
        GH["GitHub Actions"]:::devops
        DockerBuild["Docker Build"]:::devops
        ECR["AWS ECR"]:::infra
        EC2["AWS EC2"]:::infra
    end

    %% Data Flow
    DS1 --> DI --> DV --> DT --> MT --> TP
    TP --> Model
    TP --> Preprocessor
    Model --> BP
    Preprocessor --> BP
    DS2 --> BP
    BP --> Output
    Output --> Flask
    Flask --> Template

    %% Cloud Sync
    Model --> S3_Syncer["S3 Syncer"]:::compute
    Preprocessor --> S3_Syncer
    S3_Syncer --> S3

    %% Deployment Flow
    Code --> GH --> DockerBuild --> ECR --> EC2
    EC2 --> Flask

    %% Click Events
    click DI "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/components/data_ingestion.py"
    click DV "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/components/data_validation.py"
    click DT "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/components/data_transformation.py"
    click MT "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/components/model_trainer.py"
    click TP "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/pipeline/training_pipeline.py"
    click BP "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/pipeline/batch_prediction.py"
    click S3_Syncer "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/cloud/s3_syncer.py"
    click Code "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/app.py"
    click Flask "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/app.py"
    click Template "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/templates/table.html"
    click DS1 "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/network_data/phisingData.csv"
    click DS2 "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/valid_data/test.csv"
    click Model "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/final_model/model.pkl"
    click Preprocessor "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/final_model/preprocessor.pkl"
    click Output "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/prediction_output/output.csv"
    click S3 "https://github.com/rajeshraiml01/networksecurity-mlops/tree/main/networksecurity/constant/training_pipeline"
    click S3 "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/entity/artifact_entity.py"
    click S3 "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/entity/config_entity.py"
    click S3 "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/logging/logger.py"
    click S3 "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/utils/main_utils/utils.py"
    click S3 "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/utils/ml_utils/metric/classification_metric.py"
    click S3 "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/networksecurity/utils/ml_utils/model/estimator.py"
    click Code "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/main.py"
    click Code "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/push_data.py"
    click Code "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/test_mongodb.py"
    click DockerBuild "https://github.com/rajeshraiml01/networksecurity-mlops/tree/main/Dockerfile"
    click GH "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/.github/workflows/main.yml"
    click Code "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/setup.py"
    click Code "https://github.com/rajeshraiml01/networksecurity-mlops/blob/main/requirements.txt"

    %% Styles
    classDef compute fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef datastore fill:#FFD700,stroke:#333,stroke-width:1px
    classDef infra fill:#90EE90,stroke:#333,stroke-width:1px
    classDef ui fill:#FFB6C1,stroke:#333,stroke-width:1px
    classDef devops fill:#D3D3D3,stroke:#333,stroke-width:1px
```
