# saad1912/spam_classification_ml_pipeline_dvc architecture

> Architecture diagram of the GitHub repository saad1912/spam_classification_ml_pipeline_dvc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saad1912/spam_classification_ml_pipeline_dvc
- Repository: https://github.com/saad1912/spam_classification_ml_pipeline_dvc

## Overview

##### Main Components
- ML Pipeline Stages: Represent each stage in the pipeline such as Data Ingestion, Data Preprocessing, Feature Engineering, Model Training, and Model Evaluation. These are clearly defined in the src folder as:
- 01_data_ingestion.py
- 02_data_preprocessing.py
- 03_feature_engineering.py
- 04_model_training.py
- 05_model_evaluation.py
- Experiment Tracking: Highlight DVC and dvclive components. You have DVC configurations (dvc.yaml, dvc.lock, .dvc folder) and dvclive directories showing logged metrics, parameters, and plots.
- Data Storage: Include AWS S3 integration as mentioned in the README (this is an external dependency where data is stored/managed).
- Experiments and Artifacts: Illustrate the experiments directory where notebooks and supplemental data (spam.csv) reside.
- Project Flow Documentation: Represent the project_flow.txt, which captures an overview of the process.

##### Relationships and Interactions
- ML pipeline stages to the DVC component to indicate that these stages are version-controlled and tracked.
- That each stage could act as a separate execution node that takes input and produces outputs that are versioned (link each stage with dvc.yaml and dvc.lock files).
- Interactions between the local environment (the src scripts and experiments) with remote storage (AWS S3) via DVC.
- Dvclive as a side component that logs selected metrics (accuracy, auc, precision, recall), reinforcing the experiment tracking aspect.

##### Architectural Patterns
- The pipeline is designed as a sequential or staged workflow, typical in ML projects.
- The separation between data processing, model training, and evaluation allows clear modularity.

##### Technologies, Frameworks, and Tools
- DVC for data versioning and pipeline orchestration (use of dvc.yaml and dvc.lock).
- dvclive for live tracking of metrics and experiment parameters.
- AWS S3 for remote data and model storage.
- Jupyter Notebook in the experiments folder for interactive analysis (my_notebook.ipynb).
- Standard Python scripting for implementing ML pipeline stages.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "ML Pipeline"
        A["Data Ingestion"]:::pipeline
        B["Data Preprocessing"]:::pipeline
        C["Feature Engineering"]:::pipeline
        D["Model Training"]:::pipeline
        E["Model Evaluation"]:::pipeline
        A -->|"ingested data"| B
        B -->|"preprocessed data"| C
        C -->|"engineered features"| D
        D -->|"trained model"| E
    end

    subgraph "Experiment Management"
        F["DVC Experiment Tracking"]:::tracking
        G["dvclive Metrics Logging"]:::tracking
        I["Experiments & Artifacts"]:::artifact
        J["Project Flow Documentation"]:::doc
        F -->|"experiments tracked"| I
        I -->|"documented flow"| J
    end

    H["AWS S3"]:::external

    A -->|"versioned"| F
    B -->|"versioned"| F
    C -->|"versioned"| F
    D -->|"versioned"| F
    E -->|"versioned"| F

    E -->|"evaluation metrics"| G

    F -->|"remote storage"| H

    click A "https://github.com/saad1912/spam_classification_ml_pipeline_dvc/blob/main/src/01_data_ingestion.py"
    click B "https://github.com/saad1912/spam_classification_ml_pipeline_dvc/blob/main/src/02_data_preprocessing.py"
    click C "https://github.com/saad1912/spam_classification_ml_pipeline_dvc/blob/main/src/03_feature_engineering.py"
    click D "https://github.com/saad1912/spam_classification_ml_pipeline_dvc/blob/main/src/04_model_training.py"
    click E "https://github.com/saad1912/spam_classification_ml_pipeline_dvc/blob/main/src/05_model_evaluation.py"
    click F "https://github.com/saad1912/spam_classification_ml_pipeline_dvc/blob/main/dvc.yaml"
    click G "https://github.com/saad1912/spam_classification_ml_pipeline_dvc/tree/main/dvclive"
    click I "https://github.com/saad1912/spam_classification_ml_pipeline_dvc/tree/main/experiments"
    click J "https://github.com/saad1912/spam_classification_ml_pipeline_dvc/blob/main/project_flow.txt"

    classDef pipeline fill:#f9f,stroke:#333,stroke-width:2px;
    classDef tracking fill:#bbf,stroke:#333,stroke-width:2px;
    classDef external fill:#cfc,stroke:#333,stroke-width:2px;
    classDef artifact fill:#fc9,stroke:#333,stroke-width:2px;
    classDef doc fill:#ffc,stroke:#333,stroke-width:2px;
```
