# ishanpandey15/networksecurity architecture

> Architecture diagram of the GitHub repository ishanpandey15/networksecurity, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ishanpandey15/networksecurity
- Repository: https://github.com/ishanpandey15/networksecurity

## Overview

##### Project type and purpose
- This is a Python-based, full-stack ML service for real-time phishing detection.
- It ingests network traffic data, preprocesses it, trains a scikit-learn model, and exposes a FastAPI application that continuously monitors traffic and issues alerts.
- Artifacts (model + preprocessor) are versioned and can be synced to cloud storage (S3).

##### File structure
- Top level:
- app.py: FastAPI service entry point
- main.py: CLI orchestration for training pipelines
- push_data.py: data ingestion helper, likely pushes data to DB or cloud
- Dockerfile / GitHub workflows: containerization and CI/CD
- data_schema/schema.yaml: enforces data format
- Network_Data/*.csv and valid_data/*.csv: raw and test datasets
- final_model/: persisted model artifacts
- templates/table.html: web UI template for results
- networksecurity/:
- components/: modules for data_ingestion, validation, transformation, model_trainer
- pipeline/: scripts for training_pipeline.py (end-to-end training) and batch_prediction.py
- cloud/s3_syncer.py: artifact syncing to AWS S3
- utils/: helper functions, ML metrics, model wrapper, logging
- entity/, constant/: config and artifact definition
- Deployment/config files: requirements.txt, setup.py, Dockerfile, GitHub Actions.

##### README for additional insights
- Layers named: Data Collection, Preprocessing & Schema, Model Training, Real-Time Detection, Alert System.
- Tech stack: Python, scikit-learn, MongoDB, FastAPI.
- Alerts are raised when phishing is detected; possibility to extend to email/SMS.

##### Main components and their interactions
A. Data Collection Layer
- Network traffic source (CSV or real stream)
- push_data.py or FastAPI ingestion endpoint
B. Data Preprocessing & Schema Layer
- data_ingestion → raw DataFrame
- data_validation → enforce schema.yaml
- data_transformation → feature engineering
C. Model Training Pipeline
- training_pipeline.py calls components in sequence
- model_trainer.py fits scikit-learn model
- outputs model.pkl + preprocessor.pkl to final_model/
- optionally sync to AWS S3 via cloud/s3_syncer.py
D. Batch Prediction (optional)
- batch_prediction.py reads new data and writes predictions to prediction_output/output.csv
E. Real-Time Detection Service
- app.py (FastAPI) loads preprocessor + model on startup
- monitors incoming network data
- preprocesses & scores requests
- logs results to MongoDB
- triggers alert logic (could be email/SMS in future)
- serves a basic dashboard via templates/table.html
F. Supporting Services
- MongoDB: storing raw, clean, and prediction logs
- S3 (cloud): artifact storage
- CI/CD: GitHub Actions + Docker for build and deploy

##### Architectural patterns and design principles
- Modular pipeline design: each “component” (ingest, validate, transform, train) is a standalone module.
- Layered architecture: Data → Processing → Training → Deployment → Alerting.
- Separation of concerns: model code, utils, constants, entities, and pipelines each live in their own package.
- Artifact management: model + preprocessor treated as first-class entities, with cloud sync support.
- Monolithic service for real-time detection, but easily extensible into microservices (each component could be its own service).

##### Final checklist
- Arrowheads on all lines to show directionality.
- Related files/modules in labeled containers.
- Technologies next to components (e.g., “MongoDB”, “scikit-learn”, “FastAPI”).
- Every module from the file tree is represented at least once in the appropriate layer.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Collection Layer
    subgraph "Data Collection Layer"
        direction TB
        CSV["Raw Network Traffic\nNetwork_Data/phisingData.csv"]:::data
        PushData(["push_data.py"]):::data
        IngestAPI(["Ingestion Endpoint\napp.py"]):::data
    end

    %% Preprocessing & Schema Layer
    subgraph "Preprocessing & Schema Layer"
        direction TB
        Schema["Schema Definition\nschema.yaml"]:::data
        IngestionMod(["data_ingestion.py"]):::data
        ValidationMod(["data_validation.py"]):::data
        TransformationMod(["data_transformation.py"]):::data
        MongoClean[(MongoDB\nCleaned Data)]:::db
    end

    %% Model Training Pipeline
    subgraph "Model Training Pipeline"
        direction TB
        CLI(["main.py\nTraining CLI"]):::pipeline
        Orchestrator(["training_pipeline.py"]):::pipeline
        Trainer(["model_trainer.py"]):::pipeline
        Artifacts[(Model Artifacts\nmodel.pkl + preprocessor.pkl)]:::storage
        S3Sync(["s3_syncer.py"]):::cloud
    end

    %% Batch Prediction
    subgraph "Batch Prediction"
        direction TB
        BatchPred(["batch_prediction.py"]):::pipeline
        BatchOut["prediction_output/output.csv"]:::data
    end

    %% Real-Time Detection Service
    subgraph "Real-Time Detection Service"
        direction TB
        FastAPIService(["FastAPI Service\napp.py"]):::service
        Dashboard(["Dashboard Template\ntemplates/table.html"]):::service
        MongoLog[(MongoDB\nPrediction Logs)]:::db
        Alerting(["Alert System\n(Future Email/SMS)"]):::cloud
    end

    %% Supporting Modules
    subgraph "Utilities & Config"
        direction TB
        MLUtils(["ml_utils"]):::utils
        GenUtils(["utils.py"]):::utils
        ConfigEntity(["config_entity.py"]):::utils
        ArtEntity(["artifact_entity.py"]):::utils
        Constants(["training_pipeline"]):::utils
        Logger(["logger.py"]):::utils
    end

    subgraph "Deployment & CI/CD"
        direction TB
        DockerfileNode(["Dockerfile"]):::infra
        CI(["GitHub Actions\nmain.yaml"]):::infra
    end

    %% External Services
    MongoDB[(MongoDB)]:::db
    S3[(AWS S3)]:::cloud

    %% Connections Data Collection to Preprocessing
    CSV -->|CSV| IngestionMod
    PushData -->|pushes data| MongoDB
    IngestAPI -->|JSON over HTTP| IngestionMod

    %% Schema enforcement and transformation
    IngestionMod -->|DataFrame| ValidationMod
    ValidationMod -->|Validated| TransformationMod
    TransformationMod -->|Features| MongoClean

    %% Training pipeline flow
    MongoClean -->|Clean Data| Orchestrator
    CLI -->|calls| Orchestrator
    Orchestrator -->|fits model| Trainer
    Trainer -->|outputs| Artifacts
    Artifacts -->|store| S3Sync
    Artifacts -->|local store| Artifacts

    %% Batch prediction flow
    MongoClean -->|new data| BatchPred
    BatchPred -->|predictions| BatchOut

    %% Real-time detection flow
    FastAPIService -->|loads| Artifacts
    FastAPIService -->|reads stream| IngestionMod
    FastAPIService -->|uses| ValidationMod
    FastAPIService -->|uses| TransformationMod
    FastAPIService -->|writes| MongoLog
    FastAPIService -->|triggers| Alerting
    FastAPIService -->|serves| Dashboard

    %% Supporting modules usage
    Orchestrator --> MLUtils
    Trainer --> MLUtils
    ValidationMod --> GenUtils
    ConfigEntity --> Orchestrator
    ArtEntity --> Orchestrator
    Orchestrator --> Constants
    all --> Logger

    %% Infrastructure
    DockerfileNode -.-> FastAPIService
    DockerfileNode -.-> CLI
    CI -.-> DockerfileNode

    %% External services
    MongoLog --> MongoDB
    MongoClean --> MongoDB
    S3Sync --> S3

    %% Click Events
    click PushData "https://github.com/ishanpandey15/networksecurity/blob/main/push_data.py"
    click IngestAPI "https://github.com/ishanpandey15/networksecurity/blob/main/app.py"
    click Schema "https://github.com/ishanpandey15/networksecurity/blob/main/data_schema/schema.yaml"
    click IngestionMod "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/components/data_ingestion.py"
    click ValidationMod "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/components/data_validation.py"
    click TransformationMod "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/components/data_transformation.py"
    click Orchestrator "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/pipeline/training_pipeline.py"
    click Trainer "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/components/model_trainer.py"
    click CLI "https://github.com/ishanpandey15/networksecurity/blob/main/main.py"
    click Artifacts "https://github.com/ishanpandey15/networksecurity/blob/main/final_model/model.pkl"
    click Artifacts "https://github.com/ishanpandey15/networksecurity/blob/main/final_model/preprocessor.pkl"
    click S3Sync "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/cloud/s3_syncer.py"
    click BatchPred "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/pipeline/batch_prediction.py"
    click FastAPIService "https://github.com/ishanpandey15/networksecurity/blob/main/app.py"
    click Dashboard "https://github.com/ishanpandey15/networksecurity/blob/main/templates/table.html"
    click MLUtils "https://github.com/ishanpandey15/networksecurity/tree/main/networksecurity/utils/ml_utils"
    click GenUtils "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/utils/main_utils/utils.py"
    click ConfigEntity "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/entity/config_entity.py"
    click ArtEntity "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/entity/artifact_entity.py"
    click Constants "https://github.com/ishanpandey15/networksecurity/tree/main/networksecurity/constant/training_pipeline"
    click Logger "https://github.com/ishanpandey15/networksecurity/blob/main/networksecurity/logging/logger.py"
    click CI "https://github.com/ishanpandey15/networksecurity/blob/main/.github/workflows/main.yaml"
    click DockerfileNode "https://github.com/ishanpandey15/networksecurity/tree/main/Dockerfile"
    click CSV "https://github.com/ishanpandey15/networksecurity/blob/main/Network_Data/phisingData.csv"
    click BatchOut "https://github.com/ishanpandey15/networksecurity/blob/main/prediction_output/output.csv"

    %% Styles
    classDef data fill:#e0f7fa,stroke:#006064,color:#006064
    classDef pipeline fill:#e8f5e9,stroke:#1b5e20,color:#1b5e20
    classDef service fill:#fff3e0,stroke:#e65100,color:#e65100
    classDef cloud fill:#eceff1,stroke:#37474f,color:#37474f
    classDef db fill:#f1f8e9,stroke:#33691e,color:#33691e
    classDef utils fill:#f3e5f5,stroke:#6a1b9a,color:#6a1b9a
    classDef infra fill:#fbe9e7,stroke:#bf360c,color:#bf360c
```
