How pwskills-datascienceteam/spam-detection works
This repository implements a spam-versus-ham classifier with a web-facing training and prediction workflow. Training retrieves labeled messages, splits and validates data, transforms text, trains and evaluates a model, then can publish an accepted model; prediction loads the stored model and classifies supplied text. The graph emphasizes these domain stages and the storage/model-serving layer. Some implementation files were not excerpted, so their subsystem relationships are included only where the source index or README supports them.
Main entry points: train_pipeline.py Training orchestrator · artifact_entity.py Pipeline artifacts · aws_storage.py S3 storage service · app.py Web application
Components 19
Web workflows
- Web application
app.py - Prediction pipeline
src/pipeline/prediction_pipeline.py
Model training
- Training orchestrator
src/pipeline/train_pipeline.py - Data ingestion
src/components/data_ingestion.py - Data validation
src/components/data_validation.py - Text transformation
src/components/data_transformation.py - Model training
src/components/model_trainer.py - Model evaluation
src/components/model_evaluation.py - Model publishing
src/components/model_pusher.py
Model serving
- S3 model adapter
src/ml/model/s3_estimator.py - Classifier model
src/ml/model/estimator.py
Shared infrastructure
- Mongo data access
src/data_access/spamham_data.py - Pipeline artifacts
src/entity/artifact_entity.py - ML utilities
src/utils/main_utils.py - S3 storage service
src/cloud_storage/aws_storage.py - AWS connection
src/configuration/aws_connection.py
Other components
- User
- MongoDB
- AWS S3
Connections 23
- User → Web application: submits requests
README.md - Web application → Training orchestrator: starts training
README.md - Web application → Prediction pipeline: requests predictions
README.md - Training orchestrator → Data ingestion: runs ingestion
src/pipeline/train_pipeline.py - Data ingestion → Mongo data access: exports collection
src/components/data_ingestion.py - Mongo data access → MongoDB: reads messages
src/data_access/spamham_data.py - Training orchestrator → Data validation: runs validation
src/pipeline/train_pipeline.py - Training orchestrator → Text transformation: runs transformation
src/pipeline/train_pipeline.py - Training orchestrator → Model training: runs training
src/pipeline/train_pipeline.py - Training orchestrator → Model evaluation: evaluates model
src/pipeline/train_pipeline.py - Training orchestrator → Model publishing: publishes accepted
src/pipeline/train_pipeline.py - Data ingestion → Pipeline artifacts: returns paths
src/components/data_ingestion.py - Data validation → Pipeline artifacts: returns status
src/components/data_validation.py - Text transformation → Pipeline artifacts: returns outputs
src/components/data_transformation.py - Model training → Pipeline artifacts: returns metrics
src/components/model_trainer.py - Model evaluation → Pipeline artifacts: returns decision
src/components/model_evaluation.py - Model publishing → S3 storage service: uploads model
src/components/model_pusher.py - S3 storage service → AWS connection: creates client
src/cloud_storage/aws_storage.py - S3 storage service → AWS S3: reads and writes
src/cloud_storage/aws_storage.py - Prediction pipeline → S3 model adapter: loads and predicts
src/pipeline/prediction_pipeline.py - S3 model adapter → S3 storage service: loads model
src/ml/model/s3_estimator.py - S3 model adapter → Classifier model: delegates prediction
src/ml/model/s3_estimator.py - Model training → ML utilities: tunes models
src/components/model_trainer.py