# lakku153/network-security-project architecture

> Architecture diagram of the GitHub repository lakku153/network-security-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lakku153/network-security-project
- Repository: https://github.com/lakku153/network-security-project

## Overview

##### Project type and purpose
- This is a machine-learning–based analytics system (phishing-website classifier) with two major modes:
a. A **training pipeline** that ingests raw URL data, validates, transforms, trains multiple classifiers, and serializes a preprocessor + model.
b. A **prediction interface** (batch & online) that applies the saved preprocessor+model to new URLs and outputs classification results.
- The README confirms it’s a Python project using scikit-learn, with features extracted from URLs to detect phishing vs. legitimate sites.

##### File structure
Top-level:
- `networksecurity/` – core python package, subdivided into
– **components** (data_ingestion, data_validation, data_transformation, model_trainer)
– **pipeline** (training_pipeline, batch_prediction)
– **cloud** (S3 sync logic)
– **entity**, **constant**, **exception**, **logging**, **utils** (shared configs, entities, error handling, logging, helper functions)
- `data_schema/schema.yaml` – feature definitions/schema
- `Network_Data/phisingData.csv` – raw training data
- `valid_data/test.csv` – test data for batch runs
- `final_model/` – serialized `preprocessor.pkl` + `model.pkl`
- Entry scripts: `main.py` (invoke training), `app.py` (web/API server), `push_data.py` (data loader), `test_mongo.py` (DB integration check)
- Infrastructure: `Dockerfile`, GitHub Actions workflow, `setup.py`, `requirements.txt`

##### README for additional insights
- Confirms ML models used, evaluation metrics, CLI usage (`python predict.py --url ...`).
- Future roadmap: web-based API, deeper model integration, expanded dataset.

a. Main components:
- Data Storage layer
– Raw CSVs (Network_Data, valid_data)
– Feature schema (schema.yaml)
– Serialized artifacts (final_model)
- ML Training Pipeline
– Data Ingestion → Data Validation → Data Transformation → Model Trainer
– Outputs: preprocessor.pkl + model.pkl
- Batch Prediction Pipeline
– Loads final_model + valid_data/test.csv → outputs prediction_output/output.csv
- Online Prediction API
– `app.py` serving HTTP endpoint → uses the preprocessor & model
– `templates/table.html` for rendering results
- Cloud Storage
– S3 syncer component for pushing/fetching artifacts
- CI/CD
– GitHub Actions (`.github/workflows/main.yml`) for linting, testing, packaging, Docker build
b. Relationships & data flow:
1. Raw data in → ingestion component
2. Cleaned through validation & transformation
3. Transformed data → model trainer → produce and persist artifacts
4. Artifacts pushed to S3 (optional)
5. Batch pred pipeline & online API pull artifacts → perform inference → store/display results
c. Architectural patterns & design principles:
- Modular “pipeline” pattern: clear separation of data ingestion, validation, transformation, training, inference
- Layered utilities: shared `utils`, `constant`, `entity`, `exception`, `logging` across components
- Strategy for extensibility: new models plug into `model_trainer`, new metrics under `ml_utils/metric`
- Cloud integration abstracted into `networksecurity/cloud`
d. Technologies / frameworks highlighted:
- Python packaging (`setup.py`)
- scikit-learn for ML
- Pandas / NumPy for data
- Jinja2 templates (`templates/table.html`) for web UI
- boto3 or equivalent in `s3_syncer.py` for S3 interactions
- Docker for containerization
- GitHub Actions for CI/CD

##### Diagramming recommendations
- Use **clear labels** (e.g., “Data Ingestion”, “Model Trainer”, “Batch Prediction”)
– Blue for data stores
– Green for processing components
– Orange for serving/API
– Grey for infrastructure (CI/CD, Docker, S3)
– Rectangles for compute modules,
– Clouds for external services (S3),

##### Detail
- Shared utilities (logging, exceptions) as supporting services
- Entry-point scripts (`main.py`, `app.py`) as triggers to pipelines
- Config/schema file driving validation/transform rules

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DS1[/"Raw Training Data\n(phisingData.csv)"/]:::dataStore
        DS2[/"Batch Test Data\n(test.csv)"/]:::dataStore
        DS3["Feature Schema\n(schema.yaml)"]:::dataStore
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        Ingest["Data Ingestion"]:::process
        Validate["Data Validation"]:::process
        Transform["Data Transformation"]:::process
        Trainer["Model Trainer"]:::process
        TrainingPipeline["Training Pipeline"]:::process
        BatchPred["Batch Prediction Pipeline"]:::process
        S3Sync["S3 Syncer"]:::process
    end

    %% Serving Layer
    subgraph "Serving Layer"
        direction TB
        API["Online Prediction API"]:::serve
        Template["Web UI Template"]:::serve
    end

    %% Storage & DevOps
    subgraph "Storage & DevOps"
        direction TB
        Preproc(("preprocessor.pkl")):::dataStore
        Model(("model.pkl")):::dataStore
        main["CLI Training Entry\n(main.py)"]:::process
        pushData["Data Loader\n(push_data.py)"]:::process
        testMongo["MongoDB Test\n(test_mongo.py)"]:::process
        CI["CI/CD Workflow\n(.github/workflows/main.yml)"]:::infra
        Dockerf["Docker Configuration\n(Dockerfile)"]:::infra
        Setup["Packaging Config\n(setup.py)"]:::infra
        Req["Dependencies\n(requirements.txt)"]:::infra
        S3Cloud(("S3 Storage")):::infra
    end

    %% Data Flow Connections
    DS1 --> Ingest --> Validate --> Transform --> Trainer --> Preproc
    Trainer --> Model
    main --> TrainingPipeline
    TrainingPipeline --> S3Sync --> S3Cloud
    pushData --> Ingest
    DS2 --> BatchPred
    Preproc --> BatchPred
    Model --> BatchPred
    BatchPred -->|"outputs predictions"|Template
    Preproc --> API
    Model --> API
    API --> Template

    %% Click Events
    click Ingest "https://github.com/lakku153/network-security-project/blob/main/networksecurity/components/data_ingestion.py"
    click Validate "https://github.com/lakku153/network-security-project/blob/main/networksecurity/components/data_validation.py"
    click Transform "https://github.com/lakku153/network-security-project/blob/main/networksecurity/components/data_transformation.py"
    click Trainer "https://github.com/lakku153/network-security-project/blob/main/networksecurity/components/model_trainer.py"
    click TrainingPipeline "https://github.com/lakku153/network-security-project/blob/main/networksecurity/pipeline/training_pipeline.py"
    click BatchPred "https://github.com/lakku153/network-security-project/blob/main/networksecurity/pipeline/batch_prediction.py"
    click API "https://github.com/lakku153/network-security-project/blob/main/app.py"
    click Template "https://github.com/lakku153/network-security-project/blob/main/templates/table.html"
    click S3Sync "https://github.com/lakku153/network-security-project/blob/main/networksecurity/cloud/s3_syncer.py"
    click DS3 "https://github.com/lakku153/network-security-project/blob/main/data_schema/schema.yaml"
    click DS1 "https://github.com/lakku153/network-security-project/blob/main/Network_Data/phisingData.csv"
    click DS2 "https://github.com/lakku153/network-security-project/blob/main/valid_data/test.csv"
    click Preproc "https://github.com/lakku153/network-security-project/blob/main/final_model/preprocessor.pkl"
    click Model "https://github.com/lakku153/network-security-project/blob/main/final_model/model.pkl"
    click main "https://github.com/lakku153/network-security-project/blob/main/main.py"
    click pushData "https://github.com/lakku153/network-security-project/blob/main/push_data.py"
    click testMongo "https://github.com/lakku153/network-security-project/blob/main/test_mongo.py"
    click CI "https://github.com/lakku153/network-security-project/blob/main/.github/workflows/main.yml"
    click Dockerf "https://github.com/lakku153/network-security-project/tree/main/Dockerfile"
    click Setup "https://github.com/lakku153/network-security-project/blob/main/setup.py"
    click Req "https://github.com/lakku153/network-security-project/blob/main/requirements.txt"

    %% Styles
    classDef dataStore fill:#ADD8E6,stroke:#000,color:#000;
    classDef process fill:#90EE90,stroke:#000,color:#000;
    classDef serve fill:#FFA500,stroke:#000,color:#000;
    classDef infra fill:#D3D3D3,stroke:#000,color:#000;
```
