How lokesh-kummari/network-security works
Project type and purpose
• This is a Python-based machine learning training and batch-prediction pipeline for phishing detection (network security use case).
• The code implements end-to-end stages: raw data ingestion, validation, transformation, model training, model artifact storage, batch inference, plus optional web UI and cloud synchronization.
File structure
Top-level scripts and folders:
• app.py / templates/table.html – lightweight web interface to display prediction results.
• main.py – orchestration entry point for training pipeline.
• push_data.py – utility to sync local artifacts to S3 (networksecurity/cloud/s3_syncer.py).
• networksecurity/components – four core components: data_ingestion.py, data_validation.py, data_transformation.py, model_trainer.py.
• networksecurity/pipelines – orchestration classes for training_pipeline.py and batch_prediction.py.
• networksecurity/utils/main_utils/ml_utils – helpers for metrics, estimators, general utilities.
• data_schema/schema.yaml – schema definition for raw CSV.
• final_model – holds final model.pkl and preprocessor.pkl.
• Network_data, valid_data, prediction_output – example input/output folders.
• logs – pipeline execution logs.
• setup.py, requirements.txt, Dockerfile – build and deployment.
Architecture patterns and dependencies
• Layered pipeline: each component (ingestion → validation → transformation → training) is independent, communicates via artifacts on disk or S3.
• CI/CD via GitHub Actions (/.github/workflows/main.yml).
• Cloud integration for artifact backup (S3 syncer).
• Separation of concerns: components, pipelines, utils, entities, exceptions, logging.
Main components and data flows
a. Data Sources
– Local CSV (“Network_data/phisingData.csv”) or S3 bucket.
b. Data Ingestion
– Reads raw CSV, writes feature_store and ingested/train.csv, ingested/test.csv.
c. Data Validation
– Validates ingested data against schema.yaml, generates drift_report (YAML) and validated/train.csv, validated/test.csv.
d. Data Transformation
– Loads validated CSVs, applies preprocessing pipeline (preprocessing.pkl), outputs transformed NumPy arrays.
e. Model Trainer
– Trains a classifier on transformed data, outputs model.pkl.
f. Artifact Management
– final_model and Artifacts/YYYY… directories hold timestamps of each run.
– push_data.py / S3 syncer pushes artifacts to cloud.
g. Batch Prediction Pipeline
– Invoked via networksecurity/pipelines/batch_prediction.py or app.py, reads new CSV, loads preprocessor+model, writes prediction_output/output.csv.
h. Web Interface (optional)
– app.py + templates/table.html serve output.csv as an HTML table.
Architectural principles
• Modular pipeline stages with clear input/output contracts.
• Artifact-driven orchestration—enables reproducibility and auditing.
• Separation of concerns:
– Components (pure processing logic) vs. pipelines (orchestration).
– Entities for config/artifact definitions.
– Utils for cross-cutting concerns (logging, metrics).