How nnassili-z0/life-style-mlops works
Project Type and Purpose
• This is a full-fledged MLOps pipeline project (“Life Style MLOps”) that ingests raw data from Kaggle, validates and preprocesses it, trains a RandomForest regression model, evaluates and registers it with MLflow, archives artifacts, and optionally pushes data to Databricks—all orchestrated by Apache Airflow and containerized via Docker Compose.
High-Level Components
a. Orchestration Layer (Apache Airflow)
• Airflow Scheduler & Webserver (in Docker Compose)
• Airflow Workers/Executors running tasks defined in dags/mlops_level1_pipeline.py
b. Data Sources & Storage
• Kaggle API (external)
• Local file system volumes:
• Raw data: data/raw/*.csv
• Processed artifacts: demo_artifacts/ and ai_playground_data/
• Logs: logs/
• MLflow tracking store: mlflow/
c. Data Processing & Validation
• Python scripts using Pandas & Pandera (data/ingest_kaggle.py, validation tasks in DAG)
d. Model Training & Evaluation
• Scikit-learn (RandomForestRegressor)
• Evaluation metrics computed and logged (MSE, MAE, R²)
e. Experiment Tracking & Model Registry
• MLflow Server (container, default URI localhost:5000)
f. Cloud Integration (Optional)
• Databricks connector for pushing processed tables to serverless SQL warehouses
g. Containerization & Deployment
• Dockerfiles in docker/ and root Dockerfile
• docker-compose.yml and docker-compose-airflow.yml
h. Monitoring (future/target)
• Prometheus + Grafana for metrics and dashboards
Component Interactions & Data Flow
1) Orchestrator triggers tasks in order: Download → Validate Raw → Preprocess → Split → Train → Evaluate → Validate Model → Register → Archive.
2) Each task reads from and writes to shared host volumes or mounted directories.
3) The model registration step communicates with the MLflow server.
4) The optional Databricks push interacts via JDBC/REST to the Databricks SQL warehouse.
5) All logs flow into the logs/ directory, which can feed Prometheus exporters.
Architectural Patterns & Principles
• Pipeline (DAG) pattern: clearly delineated sequential tasks with retry/error-handling logic.
• Layered separation:
• Orchestration layer (Airflow)
• Processing layer (Python scripts, ML libraries)
• Persistence layer (file system volumes, MLflow DB)
• External services layer (Kaggle API, Databricks)
• Containerized microservices: each major service (Airflow, MLflow) in its own container, defined in Docker Compose.
• Idempotent ETL: tasks write timestamped artifacts to avoid collisions.
Final Checklist
• [ ] Clear labels on all services, tasks, and directories
• [ ] Annotations with technology names and protocols