How wrvarun-96/end-to-end-mlops-project-with-etl-pipeline---building-network-security-system works
Project Type and Purpose
• This is an end-to-end MLOps pipeline for network security (phishing detection).
• It comprises an ETL (data ingestion → validation → transformation), model training & tracking, and deployment on AWS (EC2/ECR/S3), plus CI/CD via GitHub Actions.
File Structure
• Top-level scripts:
• main.py – orchestrates training pipeline
• app.py – runs inference API/UI
• push_data.py/test_mongodb.py – data loading & DB tests
• Core package “networksecurity/”:
• components/: data_ingestion.py, data_validation.py, data_transformation.py, model_trainer.py
• pipeline/: training_pipeline.py, batch_prediction.py
• cloud/: s3_syncer.py
• utils/, constant/, entity/, exception/, logging/
• Artifacts & final_model folders hold intermediate and final outputs.
• Dockerfile + .github/workflows/main.yml define containerization & CI/CD.
README Insights
• Data source: MongoDB Atlas + local phishingData.csv
• MLFlow + Dagshub for experiment tracking
• AWS S3/ECR for artifacts, EC2 for deployment
• Etl pipeline: ingestion → validation → transformation → training → deploy
Main Components & Relationships
A. Data Sources
• External: MongoDB Atlas
• Repository: Network_Data/phisingData.csv
B. ETL Pipeline Components (networksecurity/components)
1. Data Ingestion → Reads CSV or MongoDB → writes raw to feature_store & ingested/
2. Data Validation → Loads ingested → drift_report & validated/
3. Data Transformation → Loads validated → produces transformed .npy + preprocessing.pkl
C. Model Training (networksecurity/components/model_trainer + pipeline/training_pipeline)
• Consumes transformed data → trains model → hyperparameter tuning
• Logs metrics & params to local MLFlow + remote Dagshub
• Outputs final_model/model.pkl & preprocessor.pkl
D. Artifact Storage & Cloud Sync
• Local Artifacts/…
• s3_syncer pushes key folders (feature_store, models, logs) to AWS S3/ECR
E. Deployment
• app.py (Flask/FastAPI) loads model.pkl + preprocessor.pkl → exposes REST API or web UI (templates/table.html)
• batch_prediction.py for bulk inference
• pack via setup.py, containerize with Dockerfile, push to AWS ECR
• Deploy container on AWS EC2
F. CI/CD
• .github/workflows/main.yml builds, tests, pushes Docker image, triggers s3 sync & EC2 deployment
Architectural Patterns & Design Principles
• Layered ETL → clear separation of ingestion/validation/transformation
• Modular components (each step in its own class/module)
• Pipeline orchestration via main.py/training_pipeline.py
• Experiment tracking abstraction (MLFlow + remote)
• Infrastructure as code via Docker + GitHub Actions
• Cloud-native deployment (S3, ECR, EC2)