How ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow works
Project Type and Purpose
• This is an end-to-end machine learning pipeline for binary classification (Pima Diabetes prediction) that emphasizes reproducibility (DVC), experiment tracking (MLflow), and modular stages (preprocess → train → evaluate).
Key Architectural Components
a. Version Control & Orchestration
• Git: source code versioning
• DVC: data & model versioning, pipeline orchestration (dvc.yaml / dvc.lock)
• Remote Storage: S3/GCS (or any DVC remote) for raw data, processed data, and model artifacts
b. Data Layer
• Raw data (data/raw/diabetes.csv)
• Processed data (data/processed/diabetes_clean.csv)
c. Pipeline Stages (src/)
• preprocess.py: reads raw data, cleans/transforms → writes processed CSV
• train.py: reads processed data, trains RandomForest → writes model.pkl
• evaluate.py: reads model & processed data → computes metrics
d. Experiment Tracking
• MLflow server/ UI: log hyperparameters, metrics, and artifacts per run
e. Parameters & Dependencies
• params.yaml: all hyperparameters (e.g., random_state, n_estimators, max_depth)
• requirements.txt: Python dependencies (scikit-learn, pandas, numpy, mlflow, dvc)
Relationships & Data Flow
• A directed pipeline DAG:
Raw Data ──(dvc)──> Preprocess ──> Clean Data ──> Train ──> Model Artifact ──> Evaluate
• How MLflow hooks into train & evaluate steps to log runs
Architectural Patterns & Design Principles
• Pipeline as Code: dvc.yaml defines stages, inputs, outputs, params
• Modularity: Each stage encapsulated in its own script under src/
• Reproducibility: Data + model versioned under DVC, parameters centralized in params.yaml
• Experiment Tracking: MLflow decoupled from code via API calls in train.py / evaluate.py