How lakku153/network-security-project works
Project type and purpose
• This is a machine-learning–based analytics system (phishing-website classifier) with two major modes:
a. A training pipeline that ingests raw URL data, validates, transforms, trains multiple classifiers, and serializes a preprocessor + model.
b. A prediction interface (batch & online) that applies the saved preprocessor+model to new URLs and outputs classification results.
• The README confirms it’s a Python project using scikit-learn, with features extracted from URLs to detect phishing vs. legitimate sites.
File structure
Top-level:
• networksecurity/ – core python package, subdivided into
– components (data_ingestion, data_validation, data_transformation, model_trainer)
– pipeline (training_pipeline, batch_prediction)
– cloud (S3 sync logic)
– entity, constant, exception, logging, utils (shared configs, entities, error handling, logging, helper functions)
• data_schema/schema.yaml – feature definitions/schema
• Network_Data/phisingData.csv – raw training data
• valid_data/test.csv – test data for batch runs
• final_model/ – serialized preprocessor.pkl + model.pkl
• Entry scripts: main.py (invoke training), app.py (web/API server), push_data.py (data loader), test_mongo.py (DB integration check)
• Infrastructure: Dockerfile, GitHub Actions workflow, setup.py, requirements.txt
README for additional insights
• Confirms ML models used, evaluation metrics, CLI usage (python predict.py --url ...).
• Future roadmap: web-based API, deeper model integration, expanded dataset.
a. Main components:
• Data Storage layer
– Raw CSVs (Network_Data, valid_data)
– Feature schema (schema.yaml)
– Serialized artifacts (final_model)
• ML Training Pipeline
– Data Ingestion → Data Validation → Data Transformation → Model Trainer
– Outputs: preprocessor.pkl + model.pkl
• Batch Prediction Pipeline
– Loads final_model + valid_data/test.csv → outputs prediction_output/output.csv
• Online Prediction API
– app.py serving HTTP endpoint → uses the preprocessor & model
– templates/table.html for rendering results
• Cloud Storage
– S3 syncer component for pushing/fetching artifacts
• CI/CD
– GitHub Actions (.github/workflows/main.yml) for linting, testing, packaging, Docker build
b. Relationships & data flow:
1. Raw data in → ingestion component
2. Cleaned through validation & transformation
3. Transformed data → model trainer → produce and persist artifacts
4. Artifacts pushed to S3 (optional)
5. Batch pred pipeline & online API pull artifacts → perform inference → store/display results
c. Architectural patterns & design principles:
• Modular “pipeline” pattern: clear separation of data ingestion, validation, transformation, training, inference
• Layered utilities: shared utils, constant, entity, exception, logging across components
• Strategy for extensibility: new models plug into model_trainer, new metrics under ml_utils/metric
• Cloud integration abstracted into networksecurity/cloud
d. Technologies / frameworks highlighted:
• Python packaging (setup.py)
• scikit-learn for ML
• Pandas / NumPy for data
• Jinja2 templates (templates/table.html) for web UI
• boto3 or equivalent in s3_syncer.py for S3 interactions
• Docker for containerization
• GitHub Actions for CI/CD
Diagramming recommendations
• Use clear labels (e.g., “Data Ingestion”, “Model Trainer”, “Batch Prediction”)
– Blue for data stores
– Green for processing components
– Orange for serving/API
– Grey for infrastructure (CI/CD, Docker, S3)
– Rectangles for compute modules,
– Clouds for external services (S3),
Detail
• Shared utilities (logging, exceptions) as supporting services
• Entry-point scripts (main.py, app.py) as triggers to pipelines
• Config/schema file driving validation/transform rules