# nnassili-z0/life-style-mlops architecture

> Architecture diagram of the GitHub repository nnassili-z0/life-style-mlops, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nnassili-z0/life-style-mlops
- Repository: https://github.com/nnassili-z0/life-style-mlops

## Overview

##### Project Type and Purpose
- This is a full-fledged MLOps pipeline project (“Life Style MLOps”) that ingests raw data from Kaggle, validates and preprocesses it, trains a RandomForest regression model, evaluates and registers it with MLflow, archives artifacts, and optionally pushes data to Databricks—all orchestrated by Apache Airflow and containerized via Docker Compose.

##### High-Level Components
a. Orchestration Layer (Apache Airflow)
- Airflow Scheduler & Webserver (in Docker Compose)
- Airflow Workers/Executors running tasks defined in `dags/mlops_level1_pipeline.py`
b. Data Sources & Storage
- Kaggle API (external)
- Local file system volumes:
- Raw data: `data/raw/*.csv`
- Processed artifacts: `demo_artifacts/` and `ai_playground_data/`
- Logs: `logs/`
- MLflow tracking store: `mlflow/`
c. Data Processing & Validation
- Python scripts using Pandas & Pandera (`data/ingest_kaggle.py`, validation tasks in DAG)
d. Model Training & Evaluation
- Scikit-learn (RandomForestRegressor)
- Evaluation metrics computed and logged (MSE, MAE, R²)
e. Experiment Tracking & Model Registry
- MLflow Server (container, default URI `localhost:5000`)
f. Cloud Integration (Optional)
- Databricks connector for pushing processed tables to serverless SQL warehouses
g. Containerization & Deployment
- Dockerfiles in `docker/` and root `Dockerfile`
- `docker-compose.yml` and `docker-compose-airflow.yml`
h. Monitoring (future/target)
- Prometheus + Grafana for metrics and dashboards

##### Component Interactions & Data Flow
1) Orchestrator triggers tasks in order: Download → Validate Raw → Preprocess → Split → Train → Evaluate → Validate Model → Register → Archive.
2) Each task reads from and writes to shared host volumes or mounted directories.
3) The model registration step communicates with the MLflow server.
4) The optional Databricks push interacts via JDBC/REST to the Databricks SQL warehouse.
5) All logs flow into the `logs/` directory, which can feed Prometheus exporters.

##### Architectural Patterns & Principles
- Pipeline (DAG) pattern: clearly delineated sequential tasks with retry/error-handling logic.
- Layered separation:
- Orchestration layer (Airflow)
- Processing layer (Python scripts, ML libraries)
- Persistence layer (file system volumes, MLflow DB)
- External services layer (Kaggle API, Databricks)
- Containerized microservices: each major service (Airflow, MLflow) in its own container, defined in Docker Compose.
- Idempotent ETL: tasks write timestamped artifacts to avoid collisions.

##### Final Checklist
- [ ] Clear labels on all services, tasks, and directories
- [ ] Annotations with technology names and protocols

## Diagram (Mermaid)

```mermaid
graph TD
    %% Orchestration Layer
    subgraph "Orchestration Layer" 
        direction TB
        Airflow_Containers["Airflow Containers"]:::container
        subgraph "Airflow Components"
            direction TB
            Scheduler["Scheduler"]:::orchestration
            Webserver["Webserver"]:::orchestration
            Workers["Workers/Executors"]:::orchestration
        end
        DAG["mlops_level1_pipeline.py"]:::orchestration
        Additional_DAGs["Additional DAGs Directory"]:::orchestration
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        Ingest["Data Ingestion"]:::processing
        Validate["Data Validation"]:::processing
        Preprocess["Preprocessing"]:::processing
        Split["Train/Test Split"]:::processing
        Train["Model Training"]:::processing
        Evaluate["Model Evaluation"]:::processing
        Register["Model Registration"]:::processing
        Archive["Artifact Archival"]:::processing
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        RawData["Raw Data\n(data/raw/)"]:::storage
        ProcArtifacts["Processed Artifacts\n(demo_artifacts/)"]:::storage
        AIPlayground["AI Playground Data\n(ai_playground_data/)"]:::storage
        MLflowStore["MLflow Store\n(mlflow/)"]:::storage
        LogsStore["Logs\n(logs/)"]:::storage
    end

    %% External Services Layer
    subgraph "External Services" 
        direction TB
        KaggleAPI["Kaggle API"]:::external
        Databricks["Databricks\n(databricks.yml)"]:::external
    end

    %% Containerization & CI/CD
    subgraph "Infrastructure Files"
        direction TB
        RootDockerfile["Dockerfile (root)"]:::infra
        Dockerfile1["docker/Dockerfile"]:::infra
        Dockerfile2["docker/Dockerfile_new"]:::infra
        ComposeMain["docker-compose.yml"]:::infra
        ComposeAirflow["docker-compose-airflow.yml"]:::infra
        CI_CD[".gitlab-ci.yml"]:::infra
        Deps["Dependencies\n(requirements*.txt)"]:::infra
    end

    %% Control Flow
    Scheduler -->|triggers| Ingest
    Scheduler -->|triggers| Validate
    Scheduler -->|triggers| Preprocess
    Scheduler -->|triggers| Split
    Scheduler -->|triggers| Train
    Scheduler -->|triggers| Evaluate
    Scheduler -->|triggers| Register
    Scheduler -->|triggers| Archive

    %% Data Flow
    KaggleAPI -->|downloads CSV| RawData
    RawData -->|reads raw CSV| Ingest
    Ingest -->|writes validated CSV| RawData
    RawData -->|feeds| Validate
    Validate -->|writes cleaned CSV| ProcArtifacts
    ProcArtifacts -->|reads| Preprocess
    Preprocess -->|writes preprocessed CSV| ProcArtifacts
    ProcArtifacts -->|reads| Split
    Split -->|writes train/test CSVs| ProcArtifacts
    ProcArtifacts -->|reads train/test| Train
    Train -->|writes model artifact| ProcArtifacts
    Train -->|logs metrics| LogsStore
    Evaluate -->|reads model & data| Evaluate
    Evaluate -->|writes metrics| LogsStore
    Register -->|registers model| MLflowStore
    Archive -->|archives artifacts| ProcArtifacts
    Archive -->|archives logs| LogsStore
    Register -->|communicates| MLflowServer[MLflow Server]:::container

    %% Optional Cloud Integration
    ProcArtifacts -->|push tables via JDBC| Databricks
    Databricks -->|ack| ProcArtifacts

    %% Shared Volumes (Mounts)
    Airflow_Containers --- RawData
    Airflow_Containers --- ProcArtifacts
    Airflow_Containers --- MLflowStore
    Airflow_Containers --- LogsStore

    %% Click Events
    click DAG "https://github.com/nnassili-z0/life-style-mlops/blob/main/dags/mlops_level1_pipeline.py"
    click Additional_DAGs "https://github.com/nnassili-z0/life-style-mlops/tree/main/airflow/dags/"
    click Ingest "https://github.com/nnassili-z0/life-style-mlops/blob/main/data/ingest_kaggle.py"
    click RawData "https://github.com/nnassili-z0/life-style-mlops/tree/main/data/raw/"
    click ProcArtifacts "https://github.com/nnassili-z0/life-style-mlops/tree/main/demo_artifacts/"
    click AIPlayground "https://github.com/nnassili-z0/life-style-mlops/tree/main/ai_playground_data/"
    click MLflowStore "https://github.com/nnassili-z0/life-style-mlops/tree/main/mlflow/"
    click RootDockerfile "https://github.com/nnassili-z0/life-style-mlops/tree/main/Dockerfile"
    click Dockerfile1 "https://github.com/nnassili-z0/life-style-mlops/tree/main/docker/Dockerfile"
    click Dockerfile2 "https://github.com/nnassili-z0/life-style-mlops/tree/main/docker/Dockerfile_new"
    click ComposeMain "https://github.com/nnassili-z0/life-style-mlops/blob/main/docker-compose.yml"
    click ComposeAirflow "https://github.com/nnassili-z0/life-style-mlops/blob/main/docker-compose-airflow.yml"
    click Databricks "https://github.com/nnassili-z0/life-style-mlops/blob/main/databricks.yml"
    click CI_CD "https://github.com/nnassili-z0/life-style-mlops/blob/main/.gitlab-ci.yml"
    click Deps "https://github.com/nnassili-z0/life-style-mlops/blob/main/requirements.txt"

    %% Styles
    classDef orchestration fill:#AED6F1,stroke:#2980B9,color:#154360
    classDef processing fill:#D5F5E3,stroke:#27AE60,color:#145A32
    classDef storage fill:#F9E79F,stroke:#B7950B,color:#7D6608
    classDef external fill:#F5B7B1,stroke:#C0392B,color:#641E16
    classDef container fill:#D6EAF8,stroke:#1F618D,color:#1B2631,stroke-dasharray: 5 5
    classDef infra fill:#E8DAEF,stroke:#8E44AD,color:#512E5F asunto
```
