# shashank975/mlops-pipeline-dvc-mlflow architecture

> Architecture diagram of the GitHub repository shashank975/mlops-pipeline-dvc-mlflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shashank975/mlops-pipeline-dvc-mlflow
- Repository: https://github.com/shashank975/mlops-pipeline-dvc-mlflow

## Overview

This repository implements a spam-classification ML pipeline: a user runs scripts to ingest and split a hosted CSV, preprocess text, create TF-IDF features, train a classifier, and evaluate it. Intermediate datasets and the model are persisted as files; evaluation also records metrics through DVCLive. The README documents the ingestion and preprocessing workflow, while sampled source shows the later stages. DVC configuration and unsampled files are not mapped beyond the visible tracking artifacts.

Main entry points: [`src/model_evaluation.py`](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/model_evaluation.py) (Evaluate model), [`src/data_ingestion.py`](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/data_ingestion.py) (Ingest and split), [`params.yaml`](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/params.yaml) (Pipeline parameters), [`src/feature_engineering.py`](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/feature_engineering.py) (TF-IDF features)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data_prep["Data preparation"]
  node_ingestion["Ingest and split<br/>[data_ingestion.py]"]
  node_raw_data[("Raw splits")]
  node_preprocessing["Clean and transform"]
  node_interim_data[("Processed text")]
  node_feature_engineering["TF-IDF features"]
  node_feature_data[("Feature datasets")]
end

subgraph group_modeling["Model lifecycle"]
  node_parameters["Pipeline parameters<br/>[params.yaml]"]
  node_training["Train classifier<br/>[model_training.py]"]
  node_model_data[("Serialized model")]
  node_evaluation["Evaluate model"]
end

subgraph group_tracking["Metrics tracking"]
  node_metric_report["Metric report"]
  node_dvclive[("DVCLive metrics")]
end

node_user(("Pipeline user"))
node_dataset_source["Spam CSV source"]

node_user -->|"runs first"| node_ingestion
node_dataset_source -->|"provides CSV"| node_ingestion
node_ingestion -->|"reads settings"| node_parameters
node_ingestion -->|"writes splits"| node_raw_data
node_user -->|"runs next"| node_preprocessing
node_preprocessing -->|"reads splits"| node_raw_data
node_preprocessing -->|"writes transformed data"| node_interim_data
node_feature_engineering -->|"reads settings"| node_parameters
node_feature_engineering -->|"reads text data"| node_interim_data
node_feature_engineering -->|"writes features"| node_feature_data
node_training -->|"reads settings"| node_parameters
node_training -->|"reads training set"| node_feature_data
node_training -->|"saves model"| node_model_data
node_evaluation -->|"reads settings"| node_parameters
node_evaluation -->|"loads model"| node_model_data
node_evaluation -->|"reads test set"| node_feature_data
node_evaluation -->|"writes metrics"| node_metric_report
node_evaluation -->|"logs metrics"| node_dvclive

click node_ingestion "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/data_ingestion.py"
click node_raw_data "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/data"
click node_preprocessing "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/data_preprocessing.py"
click node_interim_data "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/data"
click node_feature_engineering "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/feature_engineering.py"
click node_feature_data "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/data"
click node_parameters "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/params.yaml"
click node_training "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/model_training.py"
click node_model_data "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/models"
click node_evaluation "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/model_evaluation.py"
click node_metric_report "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/reports"
click node_dvclive "https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/dvclive"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion,node_raw_data,node_preprocessing,node_interim_data,node_feature_engineering,node_feature_data,node_user toneBlue
class node_parameters,node_training,node_model_data,node_evaluation toneAmber
class node_metric_report,node_dvclive toneMint
class node_dataset_source toneIndigo
```

## Components

### Data preparation

- [Ingest and split](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/data_ingestion.py): `src/data_ingestion.py`
- [Raw splits](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/data): `data`
- [Clean and transform](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/data_preprocessing.py): `src/data_preprocessing.py`
- [Processed text](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/data): `data`
- [TF-IDF features](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/feature_engineering.py): `src/feature_engineering.py`
- [Feature datasets](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/data): `data`

### Model lifecycle

- [Pipeline parameters](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/params.yaml): `params.yaml`
- [Train classifier](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/model_training.py): `src/model_training.py`
- [Serialized model](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/models): `models`
- [Evaluate model](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/blob/main/src/model_evaluation.py): `src/model_evaluation.py`

### Metrics tracking

- [Metric report](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/reports): `reports`
- [DVCLive metrics](https://github.com/shashank975/mlops-pipeline-dvc-mlflow/tree/main/dvclive): `dvclive`

### Other components

- **Pipeline user**
- **Spam CSV source**

## Connections

- Pipeline user → Ingest and split: runs first (evidence: `README.md`)
- Spam CSV source → Ingest and split: provides CSV (evidence: `src/data_ingestion.py`)
- Ingest and split → Pipeline parameters: reads settings (evidence: `src/data_ingestion.py`)
- Ingest and split → Raw splits: writes splits (evidence: `src/data_ingestion.py`)
- Pipeline user → Clean and transform: runs next (evidence: `README.md`)
- Clean and transform → Raw splits: reads splits (evidence: `src/data_preprocessing.py`)
- Clean and transform → Processed text: writes transformed data (evidence: `src/data_preprocessing.py`)
- TF-IDF features → Pipeline parameters: reads settings (evidence: `src/feature_engineering.py`)
- TF-IDF features → Processed text: reads text data (evidence: `src/feature_engineering.py`)
- TF-IDF features → Feature datasets: writes features (evidence: `src/feature_engineering.py`)
- Train classifier → Pipeline parameters: reads settings (evidence: `src/model_training.py`)
- Train classifier → Feature datasets: reads training set (evidence: `src/model_training.py`)
- Train classifier → Serialized model: saves model (evidence: `src/model_training.py`)
- Evaluate model → Pipeline parameters: reads settings (evidence: `src/model_evaluation.py`)
- Evaluate model → Serialized model: loads model (evidence: `src/model_evaluation.py`)
- Evaluate model → Feature datasets: reads test set (evidence: `src/model_evaluation.py`)
- Evaluate model → Metric report: writes metrics (evidence: `src/model_evaluation.py`)
- Evaluate model → DVCLive metrics: logs metrics (evidence: `src/model_evaluation.py`)
