How rajeshraiml01/networksecurity-mlops works
Project type and purpose
• This is a machine‐learning–driven network‐security tool focused on phishing detection.
• It provides both:
• A model‐training pipeline (data ingestion → validation → transformation → training).
• A batch/prediction pipeline and a simple Flask‐style web app to display results.
File structure
• Top-level orchestration:
• app.py / main.py – entry points for the web app or CLI.
• push_data.py – script to upload or sync data.
• test_mongodb.py – hints at a possible MongoDB integration for logging or storage.
• Core package: networksecurity/
• components/ – data_ingestion, data_validation, transformation, model_trainer.
• pipeline/ – training_pipeline.py and batch_prediction.py orchestrate end-to-end flows.
• utils/ – main_utils for generic helpers; ml_utils for metrics & estimator abstraction.
• cloud/ – s3_syncer.py for AWS S3 interactions.
• constant/ and entity/ – configuration & typed artifacts for pipeline stages.
• logging/ – centralized logger.
• Artifacts and data:
• network_data/phisingData.csv & valid_data/test.csv – raw/validation datasets.
• final_model/{model.pkl, preprocessor.pkl} – trained artifacts.
• prediction_output/output.csv – batch predictions.
• DevOps:
• Dockerfile – container definition.
• .github/workflows/main.yml – CI/CD pipeline.
• setup.py & requirements.txt – packaging and dependencies.
• README.md – AWS ECR, S3 and EC2 Docker deployment instructions.
Key insights from README
• Containerized deployment to AWS EC2 (Docker), artifacts and data stored/synced in S3.
• Docker images pushed to ECR (AWS ECR URI in secrets).
• No mention of a complex microservices mesh—single‐container ML service.
a. Main components
1. Data Sources:
– Local CSVs (network_data, valid_data).
– Optionally external sources synced via networksecurity/cloud/s3_syncer.
2. ML Pipeline:
– Data Ingestion → Validation → Transformation → Model Trainer
– Artifacts stored (preprocessor.pkl & model.pkl)
3. Batch Prediction:
– Loads artifacts → Runs on new data → Outputs CSV.
4. Web App (Flask):
– app.py / main.py serve HTML (templates/table.html) for user to upload/run predictions.
5. Cloud/Infra:
– Docker container running on EC2 (Dockerfile)
– AWS S3 for data/artifact storage
– AWS ECR for container registry
– GitHub Actions for CI/CD
b. Relationships & data flow
• From Artifacts → Batch Prediction → Prediction Output.
• The Web App consumes Prediction Output or directly invokes batch_prediction.
• GitHub Actions builds the Docker image → pushes to ECR → deploys to EC2.
• S3 Syncer syncs raw data or artifacts on schedule or on pipeline completion.
c. Architectural patterns & design principles
• Modular pipeline: single‐responsibility components (ingest, validate, transform, train).
• Artifact‐driven: clear separation between training and inference.
• Infrastructure as code / CI‐CD: Docker + GitHub Actions + ECR + EC2.
• Utility layers: central logging, config & entity definitions for type safety.
d. Technologies & frameworks
• Python for core logic, scikit-learn for modeling (implied by .pkl), Flask (templates).
• AWS services: S3, ECR, EC2.
• Docker for containerization.
• GitHub Actions for CI/CD.
Essential detail reminders
• Both ML pipeline and deployment pipeline.
• How utilities (logging, config) weave into each component.
• Do not omit the S3 syncer and GitHub Actions steps—they ensure end-to-end reproducibility.