How vaibhavsingh6952/mlpipeline works
Project Type and Purpose
• This is a data-engineering/machine-learning pipeline that uses:
• DVC for data and model versioning
• MLflow for experiment tracking
• It orchestrates four main stages on the Pima Indians Diabetes dataset: data ingestion, preprocessing, training, and evaluation.
File Structure
• Top-level:
• dvc.yaml, dvc.lock, .dvc/ → DVC pipeline definitions & config
• data/raw, data/processed → raw and processed datasets under DVC control
• models/ → stored trained models
• src/ → pipeline scripts (preprocess.py, train.py, evaluate.py, evaluate.py)
• params.yaml → pipeline hyperparameters
• requirements.txt → Python dependencies
• The presence of dvc.yaml indicates a DAG of pipeline stages.
Extract Insights from README
• Confirms DVC pipeline + MLflow integration.
• Two experiments are logged to DagsHub: one for training, one for evaluation.
• Data artifacts (raw and processed) and model artifacts are versioned in DagsHub.
Main Components
a. Data Stores
• Raw Data Store (data/raw, DVC remote)
• Processed Data Store (data/processed, DVC remote)
• Model Artifact Store (models/, DVC remote)
b. Compute/Processing Steps
• Preprocessing (src/preprocess.py)
• Training (src/train.py, RF classifier)
• Evaluation (src/evaluate.py)
c. Orchestration & Versioning
• DVC Pipeline (dvc.yaml)
• params.yaml for hyperparameters
d. Experiment Tracking
• MLflow Tracking Server (hosted on DagsHub)
e. External Services
• DVC Remote Storage on DagsHub
• MLflow Server on DagsHub
Relationships & Data Flow
• At each Compute step (Training & Evaluation), send metrics/logs to MLflow Tracking Server
• DVC pipeline orchestrates these steps, tracking inputs/outputs in dvc.lock
Architectural Patterns and Principles
• Pipeline as DAG (defined by dvc.yaml)
• Data versioning and reproducibility via DVC
• Experiment tracking decoupled from compute via MLflow
• Parameterization via params.yaml to enable configurable runs