How danish0511/network_security works
Project Type and Purpose
• This is a machine‐learning orchestration project (a full “ML pipeline” application) for network security data.
• It provides end-to-end functionality: data ingestion, validation, transformation, model training, evaluation, and deployment—coordinated via Apache Airflow and tracked with MLflow.
• It can perform scheduled batch predictions and push “production” models to storage (local or S3), and offers a Jupyter notebook for EDA.
Key Architectural Components
a. Data Sources
• Raw CSV files under Network_Data/
• External data (potentially fetched via get_data.py)
b. Orchestration Layer
• Airflow (airflow/dags/training_pipeline.py, batch_prediction.py)
• Docker Compose + start.sh to spin up Airflow, MLflow server, etc.
c. Pipeline Modules (networksecurity/components)
• DataIngestion → DataValidation → DataTransformation → ModelTrainer → ModelEvaluation → ModelPusher
d. Metadata & Config
• data_schema/schema.yaml
• networksecurity/constant/… and networksecurity/entity/… for config artifacts
e. Experiment Tracking & Model Registry
• MLflow (mlruns/)
• networksecurity/cloud/s3_syncer.py pushes artifacts/models to S3
f. Persisted Models
• saved_models/…/model.pkl
g. User Interaction
• Jupyter Notebook (Notebook/EDA.ipynb)
• Potential HTML report (templates/table.html)
• CLI entrypoints: main.py, start_training.py, test.py
1. Data Layer (leftmost)
2. Orchestration Layer (Airflow)
3. Compute/Processing Layer (pipeline)
4. Storage & Registry Layer (MLflow, S3, local FS)
5. User & Reporting Layer (Notebook, HTML templates)
• Rectangles for compute modules & DAGs
Raw Data → DataIngestion → DataValidation → DataTransformation → ModelTrainer → ModelEvaluation → ModelPusher → (SavedModels & MLflow & S3)
Airflow DAG (training) orchestrates all pipeline steps.
Airflow DAG (batch_prediction) reads deployed model & raw input → Prediction → Output storage or report.
User (Notebook) reads from raw & model registry to produce EDA reports.
• Blue = Data sources
• Orange = Orchestration (Airflow)
• Green = Pipeline compute
• Purple = Storage & tracking
• Gray = User interfaces
Relationships & Interactions
• Airflow (orange) invokes Python modules in networksecurity/pipeline and networksecurity/components
• The components read/write local file system (Network_Data/ & saved_models/) and MLflow tracking server (mlruns/)
• ModelPusher uses s3_syncer to upload models/artifacts to S3
• Notebook (user) pulls raw data and model artifacts from local FS or S3 for analysis
• Docker Compose wraps Airflow, MLflow server, and mounts volumes for mlruns/ & saved_models/
Architectural Patterns & Principles
• Modular “component‐based” ML pipeline: each step separately testable and reusable
• Orchestration via DAGs in Airflow to manage dependencies and scheduling
• Centralized experiment tracking with MLflow
• Separation of concerns: ingestion, validation, transformation, training, evaluation, deployment
• Infrastructure as code: DockerFile + docker-compose.yaml + start.sh for reproducible environments