# oluwatiremilekun/data-engineering-for-mlops architecture

> Architecture diagram of the GitHub repository oluwatiremilekun/data-engineering-for-mlops, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oluwatiremilekun/data-engineering-for-mlops
- Repository: https://github.com/oluwatiremilekun/data-engineering-for-mlops

## Overview

##### Project type and purpose
- This is a batch data-engineering pipeline for an MLOps project (not a web app or microservice).
- Purpose: clean and validate raw diabetic patient data, add metadata, and produce ready-to-use datasets (including slices) for downstream model training.

##### File structure
- Top-level items:
– Jupyter notebooks:
- C2W1_Assignment.ipynb
- C2W1_Assignment-checkpoint.ipynb
- C2_W1_Lab_1_TFDV_Exercise.ipynb
– Python script: add_metadata.py
– Data files: diabetic_data.csv, IDs_mapping.csv, slice_sample.csv
– Schema definition: schema.pbtxt
– Configuration/Mapping: IDs_mapping.csv
– README.md
- No service folders (frontend/backend), indicates a linear ETL pipeline implemented in notebooks and a helper script.

##### Extract insights from the README
- Confirms that this is an end-to-end “data engineering cleaning” notebook suite to ensure accurate model inputs.
- Mentions “engineer your data correctly” implying use of TensorFlow Data Validation (TFDV) and metadata augmentation.

##### Main components
a. Data Sources
- Raw CSVs: diabetic_data.csv, IDs_mapping.csv
b. Schema & Validation
- schema.pbtxt (TFDV schema)
c. Data Processing
- Jupyter notebooks (Extraction, Cleaning, Validation)
- add_metadata.py (metadata injection)
d. Slicing & Sampling
- slice_sample.csv generation
e. Outputs for ML
- Cleaned/validated dataset ready for model training
f. (Optional) Downstream ML pipeline (outside this repo)

##### Relationships & data flow
1. Raw CSVs → Notebook#1: Exploratory data analysis & schema inference (TFDV).
2. Schema (schema.pbtxt) → Notebook#2: Validation; detect anomalies.
3. IDs_mapping.csv + raw data → add_metadata.py: augment records with additional metadata.
4. Cleaned data → Notebook#3: Generate sliced datasets → outputs slice_sample.csv.
5. Final cleaned data + slices → passed to ML training component.

##### Key architectural patterns & technologies
- Batch ETL pipeline
- Data validation with TensorFlow Data Validation (TFDV)
- Metadata management via TensorFlow Metadata (TFMD)
- Orchestration via Jupyter notebooks and a standalone Python script

##### Detail
- Critical transformation (e.g., “detect anomalies,” “add timestamp metadata”).
- Libraries used (Pandas, TFDV, TFMD) next to each processing component.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Sources"
        RawDataCSV["diabetic_data.csv"]:::data
        IDMapCSV["IDs_mapping.csv"]:::data
    end

    subgraph "Configuration"
        Schema["schema.pbtxt"]:::config
    end

    subgraph "Data Processing"
        Notebook1["C2W1_Assignment.ipynb"]:::processing
        Checkpoint["C2W1_Assignment-checkpoint.ipynb"]:::processing
        Notebook2["C2_W1_Lab_1_TFDV_Exercise.ipynb"]:::processing
        Script["add_metadata.py"]:::processing
    end

    subgraph "Outputs"
        Cleaned["Cleaned Dataset"]:::output
        Slice["slice_sample.csv"]:::output
    end

    ML["ML Training Component"]:::processing
    Doc["README.md"]:::doc

    RawDataCSV -->|"raw CSV"| Notebook1
    Notebook1 -->|"inferred schema"| Schema
    Schema -->|"schema"| Notebook2
    RawDataCSV -->|"raw CSV"| Notebook2
    IDMapCSV -->|"ID mapping"| Script
    RawDataCSV -->|"raw CSV"| Script
    Notebook2 -->|"validated records"| Script
    Script -->|"augmented records"| Cleaned
    Cleaned -->|"cleaned data"| Checkpoint
    Cleaned -->|"cleaned data"| Slice
    Cleaned -->|"cleaned dataset"| ML
    Slice -->|"sliced dataset"| ML

    click RawDataCSV "https://github.com/oluwatiremilekun/data-engineering-for-mlops/blob/main/diabetic_data.csv"
    click IDMapCSV "https://github.com/oluwatiremilekun/data-engineering-for-mlops/blob/main/IDs_mapping.csv"
    click Schema "https://github.com/oluwatiremilekun/data-engineering-for-mlops/blob/main/schema.pbtxt"
    click Notebook1 "https://github.com/oluwatiremilekun/data-engineering-for-mlops/blob/main/C2W1_Assignment.ipynb"
    click Checkpoint "https://github.com/oluwatiremilekun/data-engineering-for-mlops/blob/main/C2W1_Assignment-checkpoint.ipynb"
    click Notebook2 "https://github.com/oluwatiremilekun/data-engineering-for-mlops/blob/main/C2_W1_Lab_1_TFDV_Exercise.ipynb"
    click Script "https://github.com/oluwatiremilekun/data-engineering-for-mlops/blob/main/add_metadata.py"
    click Slice "https://github.com/oluwatiremilekun/data-engineering-for-mlops/blob/main/slice_sample.csv"
    click Doc "https://github.com/oluwatiremilekun/data-engineering-for-mlops/blob/main/README.md"

    classDef data fill:#3498db,stroke:#000,color:#fff;
    classDef processing fill:#2ecc71,stroke:#000,color:#000;
    classDef config fill:#e67e22,stroke:#000,color:#000;
    classDef output fill:#9b59b6,stroke:#000,color:#000;
    classDef doc fill:#95a5a6,stroke:#000,color:#000;
```
