How oluwatiremilekun/data-engineering-for-mlops works
Project type and purpose
• This is a batch data-engineering pipeline for an MLOps project (not a web app or microservice).
• Purpose: clean and validate raw diabetic patient data, add metadata, and produce ready-to-use datasets (including slices) for downstream model training.
File structure
• Top-level items:
– Jupyter notebooks:
• C2W1_Assignment.ipynb
• C2W1_Assignment-checkpoint.ipynb
• C2_W1_Lab_1_TFDV_Exercise.ipynb
– Python script: add_metadata.py
– Data files: diabetic_data.csv, IDs_mapping.csv, slice_sample.csv
– Schema definition: schema.pbtxt
– Configuration/Mapping: IDs_mapping.csv
– README.md
• No service folders (frontend/backend), indicates a linear ETL pipeline implemented in notebooks and a helper script.
Extract insights from the README
• Confirms that this is an end-to-end “data engineering cleaning” notebook suite to ensure accurate model inputs.
• Mentions “engineer your data correctly” implying use of TensorFlow Data Validation (TFDV) and metadata augmentation.
Main components
a. Data Sources
• Raw CSVs: diabetic_data.csv, IDs_mapping.csv
b. Schema & Validation
• schema.pbtxt (TFDV schema)
c. Data Processing
• Jupyter notebooks (Extraction, Cleaning, Validation)
• add_metadata.py (metadata injection)
d. Slicing & Sampling
• slice_sample.csv generation
e. Outputs for ML
• Cleaned/validated dataset ready for model training
f. (Optional) Downstream ML pipeline (outside this repo)
Relationships & data flow
1. Raw CSVs → Notebook#1: Exploratory data analysis & schema inference (TFDV).
2. Schema (schema.pbtxt) → Notebook#2: Validation; detect anomalies.
3. IDs_mapping.csv + raw data → add_metadata.py: augment records with additional metadata.
4. Cleaned data → Notebook#3: Generate sliced datasets → outputs slice_sample.csv.
5. Final cleaned data + slices → passed to ML training component.
Key architectural patterns & technologies
• Batch ETL pipeline
• Data validation with TensorFlow Data Validation (TFDV)
• Metadata management via TensorFlow Metadata (TFMD)
• Orchestration via Jupyter notebooks and a standalone Python script
Detail
• Critical transformation (e.g., “detect anomalies,” “add timestamp metadata”).
• Libraries used (Pandas, TFDV, TFMD) next to each processing component.