How thatsuman/phishnet works
Project Type and Purpose
• This is an MLOps pipeline for detecting phishing attacks in network data.
• It’s a modular Python project that spans data ingestion, validation, transformation, model training, batch prediction, and serving/deployment.
File Structure
• Top-level scripts and configs:
– app.py: prediction service/front-end script
– main.py: orchestrates the training pipeline
– push_data.py: data ingestion or push to a remote store
– Dockerfile & .github/workflows/main.yml: containerization & CI/CD
– data_schema/schema.yaml: defines expected data format
– final_model/: serialized model & preprocessor artifacts
• phishnet package:
– components/: core pipeline steps (ingestion, validation, transformation, trainer)
– pipeline/: high-level orchestrators (training_pipeline, batch_prediction)
– utils/: helper functions (main utilities, ML metrics, estimator abstractions)
– cloud/: hooks for cloud storage or deployment targets
– entity, constant, exception, logging: support layers for configs, artifacts, errors, logging.
Extract Architecture Insights from README
• Modular MLOps pipeline with automatic validation, training, evaluation, and deployment.
• Production-grade reliability, data contracts (schema), and versioned artifacts.
Components
a. Data Sources & Ingestion
– Raw network_data/phisingData.csv
– Optionally external feeds (e.g. MongoDB tested via test_mongodb.py)
– push_data.py or phishnet.cloud connectors
b. Data Schema & Validation
– schema.yaml → phishnet.components.data_validation
c. Data Transformation
– phishnet.components.data_transformation
d. Model Training
– phishnet.components.model_trainer invoked by phishnet.pipeline.training_pipeline
e. Artifact Storage
– final_model/model.pkl & preprocessor.pkl
– Possibly a model registry or cloud bucket (phishnet.cloud)
f. Batch Prediction
– phishnet.pipeline.batch_prediction
g. Serving Layer
– app.py exposes model inference via REST or CLI
h. CI/CD & Containerization
– Dockerfile builds a container image
– .github/workflows/main.yml runs tests, builds, and deploys
i. Logging & Monitoring
– phishnet.logging.logger
– Any cloud monitoring in phishnet.cloud
Relationships & Data Flows
• From Artifact Store → Serving Layer (app.py) and Batch Prediction.
• CI/CD pipeline connects your Git repo to Container Registry → Deployment Environment.
Architectural Patterns & Principles
• Pipeline (sequence of discrete steps)
• Layered/Clean architecture:
– Components layer (business logic)
– Pipeline layer (orchestration)
– Utils & Entities (cross-cutting concerns)
– Cloud/Infra abstractions for deployment
• Configuration-driven: schema.yaml & config_entity define contracts.
• Artifact-based reproducibility: model.pkl and preprocessor.pkl.
Relevant Technologies & Frameworks
• Python modules/packages
• scikit-learn or custom estimator (phishnet.utils.ml_utils.model.estimator)
• YAML for schema config
• Docker & GitHub Actions for CI/CD
• Potential external DB/service (MongoDB, cloud buckets)
Detail Without Overthinking
• Break each Python module into its logical role.
• Cross-cutting elements (logging, exceptions, config) in sidebars or a utilities layer.