# viraj-gavade/network-security-machine-learning-project architecture

> Architecture diagram of the GitHub repository viraj-gavade/network-security-machine-learning-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/viraj-gavade/network-security-machine-learning-project
- Repository: https://github.com/viraj-gavade/network-security-machine-learning-project

## Overview

This repository is a network-security threat-classification ML product. Its main workflow ingests MongoDB records, validates and transforms them, trains and evaluates candidate classifiers, and saves deployable artifacts; FastAPI provides training and file-based prediction entry points. The README documents experiment tracking and optional S3 storage, while sampled source confirms the S3 sync and MLflow calls. Serving internals are only partly visible in the excerpts, so those links are limited to the app’s indexed imports and README claims.

Main entry points: [`network_security/pipeline/training_pipeline.py`](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/pipeline/training_pipeline.py) (Training orchestration), [`network_security/components/model_trainer.py`](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/components/model_trainer.py) (Model trainer), [`network_security/pipeline/batch_pipeline.py`](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/pipeline/batch_pipeline.py) (Batch orchestration), [`network_security/components/data_validation.py`](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/components/data_validation.py) (Data validation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data preparation"]
  node_ingestion["Data ingestion<br/>[data_ingestion.py]"]
  node_validation["Data validation<br/>[data_validation.py]"]
  node_schema["Validation schema<br/>[schema.yaml]"]
  node_transform["Feature transformation"]
end

subgraph group_train["Model training"]
  node_pipeline["Training orchestration"]
  node_batch["Batch orchestration<br/>[batch_pipeline.py]"]
  node_trainer["Model trainer<br/>[model_trainer.py]"]
  node_metrics["Classification metrics"]
end

subgraph group_serve["Prediction service"]
  node_estimator["Prediction estimator<br/>[estimator.py]"]
  node_api["FastAPI service<br/>[app.py]"]
end

subgraph group_platform["Artifacts and integrations"]
  node_storage_utils["Artifact utilities<br/>[utils.py]"]
  node_artifacts["Model artifacts<br/>[model.pkl]"]
  node_s3_sync["S3 sync<br/>[s3_syncer.py]"]
end

node_user(("Operator"))
node_api_user(("API caller"))
node_mongo[("MongoDB")]
node_training_actor(("Training operator"))
node_s3[("AWS S3")]
node_mlflow["MLflow tracking"]

node_pipeline -->|"starts ingestion"| node_ingestion
node_pipeline -->|"starts validation"| node_validation
node_pipeline -->|"starts transformation"| node_transform
node_pipeline -->|"starts training"| node_trainer
node_batch -->|"starts ingestion"| node_ingestion
node_batch -->|"starts validation"| node_validation
node_batch -->|"starts transformation"| node_transform
node_batch -->|"starts training"| node_trainer
node_ingestion -->|"reads records"| node_mongo
node_validation -->|"checks schema"| node_schema
node_validation -->|"reads and writes YAML"| node_storage_utils
node_transform -->|"saves arrays and objects"| node_storage_utils
node_trainer -->|"loads and saves"| node_storage_utils
node_trainer -->|"computes scores"| node_metrics
node_trainer -->|"wraps model"| node_estimator
node_trainer -->|"logs metrics"| node_mlflow
node_pipeline -->|"syncs artifacts"| node_s3_sync
node_s3_sync -->|"uploads folders"| node_s3
node_training_actor -->|"runs training"| node_pipeline
node_api_user -->|"requests prediction"| node_api
node_api -->|"triggers training"| node_pipeline
node_api -.->|"uses estimator"| node_estimator
node_transform -->|"saves preprocessor"| node_artifacts
node_trainer -->|"saves model"| node_artifacts

click node_ingestion "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/components/data_ingestion.py"
click node_validation "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/components/data_validation.py"
click node_schema "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/data_schema/schema.yaml"
click node_transform "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/components/data_transformation.py"
click node_pipeline "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/pipeline/training_pipeline.py"
click node_batch "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/pipeline/batch_pipeline.py"
click node_trainer "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/components/model_trainer.py"
click node_metrics "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/utils/ml_utils/metric/classification_metric.py"
click node_estimator "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/utils/ml_utils/models/estimator.py"
click node_api "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/app.py"
click node_storage_utils "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/utils/main_utils/utils.py"
click node_artifacts "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/final_model/model.pkl"
click node_s3_sync "https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/cloud/s3_syncer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion,node_validation,node_schema,node_transform toneBlue
class node_pipeline,node_batch,node_trainer,node_metrics,node_mongo,node_s3 toneAmber
class node_estimator,node_api,node_api_user toneMint
class node_storage_utils,node_artifacts,node_s3_sync toneRose
class node_user,node_training_actor,node_mlflow toneIndigo
```

## Components

### Data preparation

- [Data ingestion](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/components/data_ingestion.py): `network_security/components/data_ingestion.py`
- [Data validation](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/components/data_validation.py): `network_security/components/data_validation.py`
- [Validation schema](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/data_schema/schema.yaml): `data_schema/schema.yaml`
- [Feature transformation](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/components/data_transformation.py): `network_security/components/data_transformation.py`

### Model training

- [Training orchestration](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/pipeline/training_pipeline.py): `network_security/pipeline/training_pipeline.py`
- [Batch orchestration](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/pipeline/batch_pipeline.py): `network_security/pipeline/batch_pipeline.py`
- [Model trainer](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/components/model_trainer.py): `network_security/components/model_trainer.py`
- [Classification metrics](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/utils/ml_utils/metric/classification_metric.py): `network_security/utils/ml_utils/metric/classification_metric.py`

### Prediction service

- [Prediction estimator](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/utils/ml_utils/models/estimator.py): `network_security/utils/ml_utils/models/estimator.py`
- [FastAPI service](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/app.py): `app.py`

### Artifacts and integrations

- [Artifact utilities](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/utils/main_utils/utils.py): `network_security/utils/main_utils/utils.py`
- [Model artifacts](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/final_model/model.pkl): `final_model/model.pkl`
- [S3 sync](https://github.com/viraj-gavade/network-security-machine-learning-project/blob/main/network_security/cloud/s3_syncer.py): `network_security/cloud/s3_syncer.py`

### Other components

- **Operator**
- **API caller**
- **MongoDB**
- **Training operator**
- **AWS S3**
- **MLflow tracking**

## Connections

- Training orchestration → Data ingestion: starts ingestion (evidence: `network_security/pipeline/training_pipeline.py`)
- Training orchestration → Data validation: starts validation (evidence: `network_security/pipeline/training_pipeline.py`)
- Training orchestration → Feature transformation: starts transformation (evidence: `network_security/pipeline/training_pipeline.py`)
- Training orchestration → Model trainer: starts training (evidence: `network_security/pipeline/training_pipeline.py`)
- Batch orchestration → Data ingestion: starts ingestion (evidence: `network_security/pipeline/batch_pipeline.py`)
- Batch orchestration → Data validation: starts validation (evidence: `network_security/pipeline/batch_pipeline.py`)
- Batch orchestration → Feature transformation: starts transformation (evidence: `network_security/pipeline/batch_pipeline.py`)
- Batch orchestration → Model trainer: starts training (evidence: `network_security/pipeline/batch_pipeline.py`)
- Data ingestion → MongoDB: reads records (evidence: `network_security/components/data_ingestion.py`)
- Data validation → Validation schema: checks schema (evidence: `network_security/components/data_validation.py`)
- Data validation → Artifact utilities: reads and writes YAML (evidence: `network_security/components/data_validation.py`)
- Feature transformation → Artifact utilities: saves arrays and objects (evidence: `network_security/components/data_transformation.py`)
- Model trainer → Artifact utilities: loads and saves (evidence: `network_security/components/model_trainer.py`)
- Model trainer → Classification metrics: computes scores (evidence: `network_security/components/model_trainer.py`)
- Model trainer → Prediction estimator: wraps model (evidence: `network_security/components/model_trainer.py`)
- Model trainer → MLflow tracking: logs metrics (evidence: `network_security/components/model_trainer.py`)
- Training orchestration → S3 sync: syncs artifacts (evidence: `network_security/pipeline/training_pipeline.py`)
- S3 sync → AWS S3: uploads folders (evidence: `network_security/cloud/s3_syncer.py`)
- Training operator → Training orchestration: runs training (evidence: `README.md`)
- API caller → FastAPI service: requests prediction (evidence: `README.md`)
- FastAPI service → Training orchestration: triggers training (evidence: `README.md`)
- FastAPI service → Prediction estimator: uses estimator (evidence: `app.py`)
- Feature transformation → Model artifacts: saves preprocessor (evidence: `network_security/components/data_transformation.py`)
- Model trainer → Model artifacts: saves model (evidence: `network_security/components/model_trainer.py`)