How ds-kushagra/network-security works
Project Type and Purpose
• This is a machine-learning-driven, full-stack pipeline + micro-web-app for phishing detection.
• Core use cases:
• Offline training pipeline (main.py)
• Batch prediction pipeline (batch_prediction.py)
• Web interface for results (app.py + templates)
• Cloud sync of artifacts (s3_syncer.py)
File Structure
• Top-level scripts:
• main.py – orchestrates training pipeline
• app.py – Flask (or similar) web service for UI
• push_data.py – trigger for pushing new data or artifacts
• ML pipelines under networksecurity/pipeline:
• training_pipeline.py – end-to-end training
• batch_prediction.py – runs inference on new data
• Core components in networksecurity/components:
• data_ingestion.py
• data_validation.py
• data_transformation.py
• model_trainer.py
• Utilities & constants under networksecurity/utils and constants
• Schema (data_schema/schema.yaml) used by validation component
• Final_model/*.pkl – saved artifacts
• prediction_output/output.csv – inference results
• cloud sync logic in networksecurity/cloud/s3_syncer.py
README for Technical Stack
• Python, scikit-learn, GridSearchCV, YAML configs
• Logging to logs/ via networksecurity/logging/logger.py
• Exception handling via networksecurity/exception/exception.py
• Optional Dockerfile for containerization
• GitHub Actions workflow (CI) under .github/workflows
Main Components & Their Relationships
a. Data Sources & Storage
• Raw Data (Network_Data/phisingData.csv)
• Schema Definition (data_schema/schema.yaml)
• Validated Data (valid_data/)
• Artifacts (Final_model/, artifacts directory)
• Prediction Output (prediction_output/)
• Cloud Storage (AWS S3 – via s3_syncer)
b. Pipeline Orchestrator
• main.py invokes TrainingPipeline
• batch_prediction.py invokes BatchPredictionPipeline
c. Components
• DataIngestion → DataValidation → DataTransformation → ModelTraining
• Utils & Constants feed into all components
d. Web Service
• app.py serves UI, reads output.csv, renders templates/table.html
e. External & DevOps
• Docker container wraps app & pipeline
• CI (GitHub Actions) triggers tests/builds
• s3_syncer pushes/pulls model artifacts to/from AWS S3
Architectural Patterns & Principles
• Modular pipeline: each step in its own component with clear inputs/outputs
• Configuration-driven via YAML (schema.yaml) and constants modules
• Separation of concerns: ingestion vs. validation vs. transformation vs. training
• Layered structure: core logic under networksecurity/, scripts at root, data & models separated
• Fault tolerance: centralized logging + exception handling