# thatsuman/phishnet architecture

> Architecture diagram of the GitHub repository thatsuman/phishnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thatsuman/phishnet
- Repository: https://github.com/thatsuman/phishnet

## Overview

##### Project Type and Purpose
- This is an MLOps pipeline for detecting phishing attacks in network data.
- It’s a modular Python project that spans data ingestion, validation, transformation, model training, batch prediction, and serving/deployment.

##### File Structure
- Top-level scripts and configs:
– app.py: prediction service/front-end script
– main.py: orchestrates the training pipeline
– push_data.py: data ingestion or push to a remote store
– Dockerfile & .github/workflows/main.yml: containerization & CI/CD
– data_schema/schema.yaml: defines expected data format
– final_model/: serialized model & preprocessor artifacts
- phishnet package:
– components/: core pipeline steps (ingestion, validation, transformation, trainer)
– pipeline/: high-level orchestrators (training_pipeline, batch_prediction)
– utils/: helper functions (main utilities, ML metrics, estimator abstractions)
– cloud/: hooks for cloud storage or deployment targets
– entity, constant, exception, logging: support layers for configs, artifacts, errors, logging.

##### Extract Architecture Insights from README
- Modular MLOps pipeline with automatic validation, training, evaluation, and deployment.
- Production-grade reliability, data contracts (schema), and versioned artifacts.

##### Components
a. Data Sources & Ingestion
– Raw network_data/phisingData.csv
– Optionally external feeds (e.g. MongoDB tested via test_mongodb.py)
– push_data.py or phishnet.cloud connectors
b. Data Schema & Validation
– schema.yaml → phishnet.components.data_validation
c. Data Transformation
– phishnet.components.data_transformation
d. Model Training
– phishnet.components.model_trainer invoked by phishnet.pipeline.training_pipeline
e. Artifact Storage
– final_model/model.pkl & preprocessor.pkl
– Possibly a model registry or cloud bucket (phishnet.cloud)
f. Batch Prediction
– phishnet.pipeline.batch_prediction
g. Serving Layer
– app.py exposes model inference via REST or CLI
h. CI/CD & Containerization
– Dockerfile builds a container image
– .github/workflows/main.yml runs tests, builds, and deploys
i. Logging & Monitoring
– phishnet.logging.logger
– Any cloud monitoring in phishnet.cloud

##### Relationships & Data Flows
- From Artifact Store → Serving Layer (app.py) and Batch Prediction.
- CI/CD pipeline connects your Git repo to Container Registry → Deployment Environment.

##### Architectural Patterns & Principles
- Pipeline (sequence of discrete steps)
- Layered/Clean architecture:
– Components layer (business logic)
– Pipeline layer (orchestration)
– Utils & Entities (cross-cutting concerns)
– Cloud/Infra abstractions for deployment
- Configuration-driven: schema.yaml & config_entity define contracts.
- Artifact-based reproducibility: model.pkl and preprocessor.pkl.

##### Relevant Technologies & Frameworks
- Python modules/packages
- scikit-learn or custom estimator (phishnet.utils.ml_utils.model.estimator)
- YAML for schema config
- Docker & GitHub Actions for CI/CD
- Potential external DB/service (MongoDB, cloud buckets)

##### Detail Without Overthinking
- Break each Python module into its logical role.
- Cross-cutting elements (logging, exceptions, config) in sidebars or a utilities layer.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        DS_CSV["network_data/phisingData.csv"]:::data
        DS_Mongo["External DB (MongoDB)"]:::data
        PushData["push_data.py"]:::process
        Ingestion["phishnet.components.data_ingestion"]:::process
    end

    subgraph "Schema & Validation"
        Schema["data_schema/schema.yaml"]:::data
        Validation["phishnet.components.data_validation"]:::process
    end

    subgraph "Transformation"
        Transformation["phishnet.components.data_transformation"]:::process
    end

    subgraph "Training & Storage"
        TrainingPipeline["phishnet.pipeline.training_pipeline"]:::process
        ModelTrainer["phishnet.components.model_trainer"]:::process
        MainOrch["main.py"]:::process
        ModelArtifact["final_model/model.pkl"]:::storage
        Preprocessor["final_model/preprocessor.pkl"]:::storage
    end

    subgraph "Batch Prediction"
        BatchPred["phishnet.pipeline.batch_prediction"]:::process
    end

    subgraph "Serving Layer"
        App["app.py"]:::service
    end

    subgraph "Infrastructure"
        Docker["Dockerfile"]:::infra
        CICD[".github/workflows/main.yml"]:::infra
        CloudConn["phishnet.cloud"]:::infra
    end

    subgraph "Utilities & Entities"
        ConfigEnt["phishnet.entity.config_entity"]:::util
        ArtEnt["phishnet.entity.artifact_entity"]:::util
        Consts["phishnet.constant.training_pipeline"]:::util
        Logger["phishnet.logging.logger"]:::util
        Utils["phishnet.utils.main_utils"]:::util
        Metrics["phishnet.utils.ml_utils.metric.classification_metric"]:::util
        Estimator["phishnet.utils.ml_utils.model.estimator"]:::util
    end

    DS_CSV --> PushData
    DS_Mongo --> PushData
    PushData --> Ingestion
    Ingestion --> Schema
    Schema --> Validation
    Validation --> Transformation
    Transformation --> TrainingPipeline
    TrainingPipeline --> ModelTrainer
    ModelTrainer --> ModelArtifact
    ModelTrainer --> Preprocessor
    MainOrch --> TrainingPipeline
    ModelArtifact --> BatchPred
    Preprocessor --> BatchPred
    ModelArtifact --> App
    Preprocessor --> App
    BatchPred --> App

    CICD --> Docker
    Docker --> App
    CICD --> CloudConn
    CloudConn --> ModelArtifact
    CloudConn --> App

    Utils --> Ingestion
    Metrics --> ModelTrainer
    Estimator --> ModelTrainer
    ConfigEnt --> Validation
    ConfigEnt --> Transformation
    ConfigEnt --> ModelTrainer
    ArtEnt --> BatchPred
    Consts --> TrainingPipeline
    Logger -.-> Ingestion
    Logger -.-> Validation
    Logger -.-> Transformation
    Logger -.-> ModelTrainer
    Logger -.-> BatchPred
    Logger -.-> App

    click DS_CSV "https://github.com/thatsuman/phishnet/blob/main/network_data/phisingData.csv"
    click DS_Mongo "https://github.com/thatsuman/phishnet/blob/main/test_mongodb.py"
    click PushData "https://github.com/thatsuman/phishnet/blob/main/push_data.py"
    click Ingestion "https://github.com/thatsuman/phishnet/blob/main/phishnet/components/data_ingestion.py"
    click Schema "https://github.com/thatsuman/phishnet/blob/main/data_schema/schema.yaml"
    click Validation "https://github.com/thatsuman/phishnet/blob/main/phishnet/components/data_validation.py"
    click Transformation "https://github.com/thatsuman/phishnet/blob/main/phishnet/components/data_transformation.py"
    click ModelTrainer "https://github.com/thatsuman/phishnet/blob/main/phishnet/components/model_trainer.py"
    click TrainingPipeline "https://github.com/thatsuman/phishnet/blob/main/phishnet/pipeline/training_pipeline.py"
    click MainOrch "https://github.com/thatsuman/phishnet/blob/main/main.py"
    click BatchPred "https://github.com/thatsuman/phishnet/blob/main/phishnet/pipeline/batch_prediction.py"
    click App "https://github.com/thatsuman/phishnet/blob/main/app.py"
    click ModelArtifact "https://github.com/thatsuman/phishnet/blob/main/final_model/model.pkl"
    click Preprocessor "https://github.com/thatsuman/phishnet/blob/main/final_model/preprocessor.pkl"
    click Docker "https://github.com/thatsuman/phishnet/tree/main/Dockerfile"
    click CICD "https://github.com/thatsuman/phishnet/blob/main/.github/workflows/main.yml"
    click CloudConn "https://github.com/thatsuman/phishnet/blob/main/phishnet/cloud/__init__.py"
    click Logger "https://github.com/thatsuman/phishnet/blob/main/phishnet/logging/logger.py"
    click ConfigEnt "https://github.com/thatsuman/phishnet/blob/main/phishnet/entity/config_entity.py"
    click ArtEnt "https://github.com/thatsuman/phishnet/blob/main/phishnet/entity/artifact_entity.py"
    click Consts "https://github.com/thatsuman/phishnet/blob/main/phishnet/constant/training_pipeline/__init__.py"
    click Utils "https://github.com/thatsuman/phishnet/blob/main/phishnet/utils/main_utils/utils.py"
    click Metrics "https://github.com/thatsuman/phishnet/blob/main/phishnet/utils/ml_utils/metric/classification_metric.py"
    click Estimator "https://github.com/thatsuman/phishnet/blob/main/phishnet/utils/ml_utils/model/estimator.py"

    classDef data fill:#D0E8FF,stroke:#0366D6,color:#0366D6;
    classDef process fill:#E6FFEA,stroke:#28A745,color:#28A745;
    classDef storage fill:#E0F7FA,stroke:#00ACC1,color:#006064;
    classDef service fill:#FFF3E0,stroke:#FB8C00,color:#E65100;
    classDef infra fill:#FFF5F5,stroke:#D73A49,color:#B31D28;
    classDef util fill:#F3E8FF,stroke:#6F42C1,color:#4A148C;
```
