How ishanpandey15/networksecurity works
Project type and purpose
• This is a Python-based, full-stack ML service for real-time phishing detection.
• It ingests network traffic data, preprocesses it, trains a scikit-learn model, and exposes a FastAPI application that continuously monitors traffic and issues alerts.
• Artifacts (model + preprocessor) are versioned and can be synced to cloud storage (S3).
File structure
• Top level:
• app.py: FastAPI service entry point
• main.py: CLI orchestration for training pipelines
• push_data.py: data ingestion helper, likely pushes data to DB or cloud
• Dockerfile / GitHub workflows: containerization and CI/CD
• data_schema/schema.yaml: enforces data format
• Network_Data/*.csv and valid_data/*.csv: raw and test datasets
• final_model/: persisted model artifacts
• templates/table.html: web UI template for results
• networksecurity/:
• components/: modules for data_ingestion, validation, transformation, model_trainer
• pipeline/: scripts for training_pipeline.py (end-to-end training) and batch_prediction.py
• cloud/s3_syncer.py: artifact syncing to AWS S3
• utils/: helper functions, ML metrics, model wrapper, logging
• entity/, constant/: config and artifact definition
• Deployment/config files: requirements.txt, setup.py, Dockerfile, GitHub Actions.
README for additional insights
• Layers named: Data Collection, Preprocessing & Schema, Model Training, Real-Time Detection, Alert System.
• Tech stack: Python, scikit-learn, MongoDB, FastAPI.
• Alerts are raised when phishing is detected; possibility to extend to email/SMS.
Main components and their interactions
A. Data Collection Layer
• Network traffic source (CSV or real stream)
• push_data.py or FastAPI ingestion endpoint
B. Data Preprocessing & Schema Layer
• data_ingestion → raw DataFrame
• data_validation → enforce schema.yaml
• data_transformation → feature engineering
C. Model Training Pipeline
• training_pipeline.py calls components in sequence
• model_trainer.py fits scikit-learn model
• outputs model.pkl + preprocessor.pkl to final_model/
• optionally sync to AWS S3 via cloud/s3_syncer.py
D. Batch Prediction (optional)
• batch_prediction.py reads new data and writes predictions to prediction_output/output.csv
E. Real-Time Detection Service
• app.py (FastAPI) loads preprocessor + model on startup
• monitors incoming network data
• preprocesses & scores requests
• logs results to MongoDB
• triggers alert logic (could be email/SMS in future)
• serves a basic dashboard via templates/table.html
F. Supporting Services
• MongoDB: storing raw, clean, and prediction logs
• S3 (cloud): artifact storage
• CI/CD: GitHub Actions + Docker for build and deploy
Architectural patterns and design principles
• Modular pipeline design: each “component” (ingest, validate, transform, train) is a standalone module.
• Layered architecture: Data → Processing → Training → Deployment → Alerting.
• Separation of concerns: model code, utils, constants, entities, and pipelines each live in their own package.
• Artifact management: model + preprocessor treated as first-class entities, with cloud sync support.
• Monolithic service for real-time detection, but easily extensible into microservices (each component could be its own service).
Final checklist
• Arrowheads on all lines to show directionality.
• Related files/modules in labeled containers.
• Technologies next to components (e.g., “MongoDB”, “scikit-learn”, “FastAPI”).
• Every module from the file tree is represented at least once in the appropriate layer.