How saad1912/spam_classification_ml_pipeline_dvc works
Main Components
• ML Pipeline Stages: Represent each stage in the pipeline such as Data Ingestion, Data Preprocessing, Feature Engineering, Model Training, and Model Evaluation. These are clearly defined in the src folder as:
• 01_data_ingestion.py
• 02_data_preprocessing.py
• 03_feature_engineering.py
• 04_model_training.py
• 05_model_evaluation.py
• Experiment Tracking: Highlight DVC and dvclive components. You have DVC configurations (dvc.yaml, dvc.lock, .dvc folder) and dvclive directories showing logged metrics, parameters, and plots.
• Data Storage: Include AWS S3 integration as mentioned in the README (this is an external dependency where data is stored/managed).
• Experiments and Artifacts: Illustrate the experiments directory where notebooks and supplemental data (spam.csv) reside.
• Project Flow Documentation: Represent the project_flow.txt, which captures an overview of the process.
Relationships and Interactions
• ML pipeline stages to the DVC component to indicate that these stages are version-controlled and tracked.
• That each stage could act as a separate execution node that takes input and produces outputs that are versioned (link each stage with dvc.yaml and dvc.lock files).
• Interactions between the local environment (the src scripts and experiments) with remote storage (AWS S3) via DVC.
• Dvclive as a side component that logs selected metrics (accuracy, auc, precision, recall), reinforcing the experiment tracking aspect.
Architectural Patterns
• The pipeline is designed as a sequential or staged workflow, typical in ML projects.
• The separation between data processing, model training, and evaluation allows clear modularity.
Technologies, Frameworks, and Tools
• DVC for data versioning and pipeline orchestration (use of dvc.yaml and dvc.lock).
• dvclive for live tracking of metrics and experiment parameters.
• AWS S3 for remote data and model storage.
• Jupyter Notebook in the experiments folder for interactive analysis (my_notebook.ipynb).
• Standard Python scripting for implementing ML pipeline stages.