How viraj-gavade/network-security-machine-learning-project works
This repository is a network-security threat-classification ML product. Its main workflow ingests MongoDB records, validates and transforms them, trains and evaluates candidate classifiers, and saves deployable artifacts; FastAPI provides training and file-based prediction entry points. The README documents experiment tracking and optional S3 storage, while sampled source confirms the S3 sync and MLflow calls. Serving internals are only partly visible in the excerpts, so those links are limited to the app’s indexed imports and README claims.
Main entry points: training_pipeline.py Training orchestration · model_trainer.py Model trainer · batch_pipeline.py Batch orchestration · data_validation.py Data validation
Components 19
Data preparation
- Data ingestion
network_security/components/data_ingestion.py - Data validation
network_security/components/data_validation.py - Validation schema
data_schema/schema.yaml - Feature transformation
network_security/components/data_transformation.py
Model training
- Training orchestration
network_security/pipeline/training_pipeline.py - Batch orchestration
network_security/pipeline/batch_pipeline.py - Model trainer
network_security/components/model_trainer.py - Classification metrics
network_security/utils/ml_utils/metric/classification_metric.py
Prediction service
- Prediction estimator
network_security/utils/ml_utils/models/estimator.py - FastAPI service
app.py
Artifacts and integrations
- Artifact utilities
network_security/utils/main_utils/utils.py - Model artifacts
final_model/model.pkl - S3 sync
network_security/cloud/s3_syncer.py
Other components
- Operator
- API caller
- MongoDB
- Training operator
- AWS S3
- MLflow tracking
Connections 24
- Training orchestration → Data ingestion: starts ingestion
network_security/pipeline/training_pipeline.py - Training orchestration → Data validation: starts validation
network_security/pipeline/training_pipeline.py - Training orchestration → Feature transformation: starts transformation
network_security/pipeline/training_pipeline.py - Training orchestration → Model trainer: starts training
network_security/pipeline/training_pipeline.py - Batch orchestration → Data ingestion: starts ingestion
network_security/pipeline/batch_pipeline.py - Batch orchestration → Data validation: starts validation
network_security/pipeline/batch_pipeline.py - Batch orchestration → Feature transformation: starts transformation
network_security/pipeline/batch_pipeline.py - Batch orchestration → Model trainer: starts training
network_security/pipeline/batch_pipeline.py - Data ingestion → MongoDB: reads records
network_security/components/data_ingestion.py - Data validation → Validation schema: checks schema
network_security/components/data_validation.py - Data validation → Artifact utilities: reads and writes YAML
network_security/components/data_validation.py - Feature transformation → Artifact utilities: saves arrays and objects
network_security/components/data_transformation.py - Model trainer → Artifact utilities: loads and saves
network_security/components/model_trainer.py - Model trainer → Classification metrics: computes scores
network_security/components/model_trainer.py - Model trainer → Prediction estimator: wraps model
network_security/components/model_trainer.py - Model trainer → MLflow tracking: logs metrics
network_security/components/model_trainer.py - Training orchestration → S3 sync: syncs artifacts
network_security/pipeline/training_pipeline.py - S3 sync → AWS S3: uploads folders
network_security/cloud/s3_syncer.py - Training operator → Training orchestration: runs training
README.md - API caller → FastAPI service: requests prediction
README.md - FastAPI service → Training orchestration: triggers training
README.md - FastAPI service → Prediction estimator: uses estimator
app.py - Feature transformation → Model artifacts: saves preprocessor
network_security/components/data_transformation.py - Model trainer → Model artifacts: saves model
network_security/components/model_trainer.py