How shashank975/mlops-pipeline-dvc-mlflow works
This repository implements a spam-classification ML pipeline: a user runs scripts to ingest and split a hosted CSV, preprocess text, create TF-IDF features, train a classifier, and evaluate it. Intermediate datasets and the model are persisted as files; evaluation also records metrics through DVCLive. The README documents the ingestion and preprocessing workflow, while sampled source shows the later stages. DVC configuration and unsampled files are not mapped beyond the visible tracking artifacts.
Main entry points: model_evaluation.py Evaluate model · data_ingestion.py Ingest and split · params.yaml Pipeline parameters · feature_engineering.py TF-IDF features
Components 14
Data preparation
- Ingest and split
src/data_ingestion.py - Raw splits
data - Clean and transform
src/data_preprocessing.py - Processed text
data - TF-IDF features
src/feature_engineering.py - Feature datasets
data
Model lifecycle
- Pipeline parameters
params.yaml - Train classifier
src/model_training.py - Serialized model
models - Evaluate model
src/model_evaluation.py
Metrics tracking
- Metric report
reports - DVCLive metrics
dvclive
Other components
- Pipeline user
- Spam CSV source
Connections 18
- Pipeline user → Ingest and split: runs first
README.md - Spam CSV source → Ingest and split: provides CSV
src/data_ingestion.py - Ingest and split → Pipeline parameters: reads settings
src/data_ingestion.py - Ingest and split → Raw splits: writes splits
src/data_ingestion.py - Pipeline user → Clean and transform: runs next
README.md - Clean and transform → Raw splits: reads splits
src/data_preprocessing.py - Clean and transform → Processed text: writes transformed data
src/data_preprocessing.py - TF-IDF features → Pipeline parameters: reads settings
src/feature_engineering.py - TF-IDF features → Processed text: reads text data
src/feature_engineering.py - TF-IDF features → Feature datasets: writes features
src/feature_engineering.py - Train classifier → Pipeline parameters: reads settings
src/model_training.py - Train classifier → Feature datasets: reads training set
src/model_training.py - Train classifier → Serialized model: saves model
src/model_training.py - Evaluate model → Pipeline parameters: reads settings
src/model_evaluation.py - Evaluate model → Serialized model: loads model
src/model_evaluation.py - Evaluate model → Feature datasets: reads test set
src/model_evaluation.py - Evaluate model → Metric report: writes metrics
src/model_evaluation.py - Evaluate model → DVCLive metrics: logs metrics
src/model_evaluation.py