How aditya-sharma07/networksecurity works
Project Type and Purpose
• This is a full-stack, ML-powered phishing-detection system.
• It features an end-to-end training pipeline (ingest → validate → transform → train → persist), a FastAPI backend exposing training/prediction endpoints, HTML rendering of results, and MongoDB Atlas for storage.
File-Structure Analysis
• Top level:
• Dockerfile, requirements.txt, setup.py, GitHub Actions workflow → deployment/CI-CD.
• app.py & main.py → FastAPI application entry points.
• push_data.py & test_mongodb.py → data-ingestion helpers and DB connectivity checks.
• Network_Data/valid_data/prediction_output → sample datasets and inference outputs.
• data_schema/schema.yaml → schema definition for CSV validation.
• final_model/{model.pkl, preprocessor.pkl} → serialized artifacts for inference.
• templates/table.html → HTML template for rendering prediction tables.
• networksecurity/ contains the modular pipeline, constants, entities, exception handling, logging, utilities, and pipeline orchestrators:
• components: ingestion, transformation, validation, model training
• pipeline: batch_prediction.py & training_pipeline.py
• utils: main_utils (e.g., file loaders), ml_utils (metrics & estimator)
Key Insights from README
• ML stack: scikit-learn for modeling, Pandas/NumPy for data ops.
• FastAPI with interactive Swagger UI.
• MongoDB Atlas for persisting raw & scored data.
• Docker-ready, with GH Actions for CI.
a. Main Components
• Data Sources: Network_Data CSV, valid_data CSV.
• Schema Validator: reads schema.yaml.
• Data Ingestion Module (networksecurity/components/data_ingestion.py).
• Data Validation & Transformation Modules.
• Model Trainer (produces model.pkl & preprocessor.pkl).
• Model Registry/Artifact Store (final_model folder).
• FastAPI Server (app.py/main.py) exposing:
– /train endpoint calls training_pipeline.py
– /predict endpoint loads preprocessor & model for inference
• MongoDB Atlas (ingested data & predictions storage).
• HTML Renderer (templates/table.html) for front-end view.
• CI/CD & Containerization: Docker + GitHub Actions.
b. Relationships & Data Flows
1. CSV → Schema Validator → Data Ingestion → Data Validation → Transformation → Trainer → Model & Preprocessor artifacts.
2. Trainer writes artifacts to final_model and optionally logs metadata.
3. FastAPI /train triggers (1) and persists raw/trained metadata to MongoDB.
4. FastAPI /predict loads artifacts → preprocesses incoming CSV → model.predict() → writes predictions to MongoDB → serves HTML table or JSON.
5. push_data.py can also feed new batches into the pipeline.
c. Architectural Patterns & Principles
• Modular pipeline components (single-responsibility).
• Clear separation: Training pipeline vs. Serving/inference pipeline.
• Configuration via data_schema/schema.yaml and environment variables (.env for MongoDB URI).
• Use of design entities and artifact/config entity classes for strong typing.
• Exception handling and centralized logging.
d. Technologies & Frameworks
• Python 3.x, FastAPI, scikit-learn, Pandas, NumPy
• MongoDB Atlas (via PyMongo or Motor)
• Docker, GitHub Actions
• Jinja2 templates for HTML