# aditya-sharma07/networksecurity architecture

> Architecture diagram of the GitHub repository aditya-sharma07/networksecurity, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aditya-sharma07/networksecurity
- Repository: https://github.com/aditya-sharma07/networksecurity

## Overview

##### Project Type and Purpose
- This is a full-stack, ML-powered phishing-detection system.
- It features an end-to-end training pipeline (ingest → validate → transform → train → persist), a FastAPI backend exposing training/prediction endpoints, HTML rendering of results, and MongoDB Atlas for storage.

##### File-Structure Analysis
- Top level:
- Dockerfile, requirements.txt, setup.py, GitHub Actions workflow → deployment/CI-CD.
- app.py & main.py → FastAPI application entry points.
- push_data.py & test_mongodb.py → data-ingestion helpers and DB connectivity checks.
- Network_Data/valid_data/prediction_output → sample datasets and inference outputs.
- data_schema/schema.yaml → schema definition for CSV validation.
- final_model/{model.pkl, preprocessor.pkl} → serialized artifacts for inference.
- templates/table.html → HTML template for rendering prediction tables.
- networksecurity/ contains the modular pipeline, constants, entities, exception handling, logging, utilities, and pipeline orchestrators:
- components: ingestion, transformation, validation, model training
- pipeline: batch_prediction.py & training_pipeline.py
- utils: main_utils (e.g., file loaders), ml_utils (metrics & estimator)

##### Key Insights from README
- ML stack: scikit-learn for modeling, Pandas/NumPy for data ops.
- FastAPI with interactive Swagger UI.
- MongoDB Atlas for persisting raw & scored data.
- Docker-ready, with GH Actions for CI.

a. Main Components
- Data Sources: Network_Data CSV, valid_data CSV.
- Schema Validator: reads schema.yaml.
- Data Ingestion Module (networksecurity/components/data_ingestion.py).
- Data Validation & Transformation Modules.
- Model Trainer (produces model.pkl & preprocessor.pkl).
- Model Registry/Artifact Store (final_model folder).
- FastAPI Server (app.py/main.py) exposing:
– /train endpoint calls training_pipeline.py
– /predict endpoint loads preprocessor & model for inference
- MongoDB Atlas (ingested data & predictions storage).
- HTML Renderer (templates/table.html) for front-end view.
- CI/CD & Containerization: Docker + GitHub Actions.

b. Relationships & Data Flows
1. CSV → Schema Validator → Data Ingestion → Data Validation → Transformation → Trainer → Model & Preprocessor artifacts.
2. Trainer writes artifacts to final_model and optionally logs metadata.
3. FastAPI /train triggers (1) and persists raw/trained metadata to MongoDB.
4. FastAPI /predict loads artifacts → preprocesses incoming CSV → model.predict() → writes predictions to MongoDB → serves HTML table or JSON.
5. push_data.py can also feed new batches into the pipeline.

c. Architectural Patterns & Principles
- Modular pipeline components (single-responsibility).
- Clear separation: Training pipeline vs. Serving/inference pipeline.
- Configuration via data_schema/schema.yaml and environment variables (.env for MongoDB URI).
- Use of design entities and artifact/config entity classes for strong typing.
- Exception handling and centralized logging.

d. Technologies & Frameworks
- Python 3.x, FastAPI, scikit-learn, Pandas, NumPy
- MongoDB Atlas (via PyMongo or Motor)
- Docker, GitHub Actions
- Jinja2 templates for HTML

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Data Layer
  subgraph "Data Layer"
    RawCSV["Raw CSV\n(phisingData.csv)"]:::data
    ValidCSV["Valid CSV\n(test.csv)"]:::data
    Schema["Schema Definition\n(schema.yaml)"]:::data
  end

  %% Pipeline Layer
  subgraph "Pipeline Layer"
    DataIngest["Data Ingestion\n(data_ingestion.py)"]:::pipeline
    Validation["Data Validation\n(data_validation.py)"]:::pipeline
    Transformation["Data Transformation\n(data_transformation.py)"]:::pipeline
    Trainer["Model Trainer\n(model_trainer.py)"]:::pipeline
    TrainingPipe["Training Pipeline\n(training_pipeline.py)"]:::pipeline
    BatchPred["Batch Prediction\n(batch_prediction.py)"]:::pipeline
  end

  %% Model Registry
  subgraph "Model Registry"
    ModelArt["Artifacts\n(model.pkl & preprocessor.pkl)"]:::storage
  end

  %% Serving Layer
  subgraph "Serving Layer"
    FastAPI["FastAPI Server\n(app.py & main.py)"]:::api
  end

  %% Storage Layer
  subgraph "Storage Layer"
    MongoDB["MongoDB Atlas"]:::storage
  end

  %% UI Layer
  subgraph "UI Layer"
    HTML["HTML Renderer\n(table.html)"]:::ui
  end

  %% Infrastructure
  subgraph "Infrastructure"
    Docker["Dockerfile"]:::infra
    CICD["GitHub Actions\n(.github/workflows/main.yml)"]:::infra
  end

  %% Data Flow
  RawCSV -->|"CSV → DataFrame"| DataIngest
  ValidCSV -->|"CSV → DataFrame"| Validation
  Schema -->|"YAML → Rules"| Validation

  DataIngest -->|"raw DF"| Validation
  Validation -->|"validated DF"| Transformation
  Transformation -->|"features"| Trainer
  Trainer -->|"train"| TrainingPipe
  TrainingPipe -->|"pickle artifacts"| ModelArt
  TrainingPipe -->|"metadata"| MongoDB

  BatchPred -->|"batch CSV → preds"| MongoDB

  %% Serving Flow
  FastAPI -->|"/train"| TrainingPipe
  FastAPI -->|"/predict"| Validation
  Validation -->|"validated DF"| Transformation
  Transformation -->|"features"| FastAPI
  FastAPI -->|"load artifacts"| ModelArt
  FastAPI -->|"predictions"| MongoDB
  FastAPI -->|"results"| HTML

  %% Infra Links
  Docker -->|runs| FastAPI
  CICD -->|deploys| Docker

  %% Click Events
  click RawCSV "https://github.com/aditya-sharma07/networksecurity/blob/main/Network_Data/phisingData.csv"
  click ValidCSV "https://github.com/aditya-sharma07/networksecurity/blob/main/valid_data/test.csv"
  click Schema "https://github.com/aditya-sharma07/networksecurity/blob/main/data_schema/schema.yaml"
  click DataIngest "https://github.com/aditya-sharma07/networksecurity/blob/main/networksecurity/components/data_ingestion.py"
  click Validation "https://github.com/aditya-sharma07/networksecurity/blob/main/networksecurity/components/data_validation.py"
  click Transformation "https://github.com/aditya-sharma07/networksecurity/blob/main/networksecurity/components/data_transformation.py"
  click Trainer "https://github.com/aditya-sharma07/networksecurity/blob/main/networksecurity/components/model_trainer.py"
  click TrainingPipe "https://github.com/aditya-sharma07/networksecurity/blob/main/networksecurity/pipeline/training_pipeline.py"
  click BatchPred "https://github.com/aditya-sharma07/networksecurity/blob/main/networksecurity/pipeline/batch_prediction.py"
  click ModelArt "https://github.com/aditya-sharma07/networksecurity/blob/main/final_model/model.pkl"
  click FastAPI "https://github.com/aditya-sharma07/networksecurity/blob/main/app.py"
  click Docker "https://github.com/aditya-sharma07/networksecurity/tree/main/Dockerfile"
  click CICD "https://github.com/aditya-sharma07/networksecurity/blob/main/.github/workflows/main.yml"
  click HTML "https://github.com/aditya-sharma07/networksecurity/blob/main/templates/table.html"

  %% Styles
  classDef data fill:#f9f,stroke:#333,stroke-width:1px
  classDef pipeline fill:#bbf,stroke:#333,stroke-width:1px
  classDef api fill:#bfb,stroke:#333,stroke-width:1px
  classDef storage fill:#fb3,stroke:#333,stroke-width:1px
  classDef ui fill:#ffb,stroke:#333,stroke-width:1px
  classDef infra fill:#ccc,stroke:#333,stroke-width:1px
```
