# kachiann/project-mlops architecture

> Architecture diagram of the GitHub repository kachiann/project-mlops, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kachiann/project-mlops
- Repository: https://github.com/kachiann/project-mlops

## Overview

##### Project Type and Purpose
- This is an end-to-end MLOps pipeline for bike-sharing demand prediction. It combines data ingestion, ML experiment tracking, workflow orchestration, model deployment, and monitoring.
- Key goals: train regression models on historical data, track experiments in MLflow, orchestrate tasks with Prefect, deploy a Flask API, and monitor data/model drift with Evidently and Grafana.

##### File Structure Analysis
- Top-level folders:
- data & production/reference CSVs
- notebooks for EDA and prototype training
- src: Python modules for experiment tracking, model registry, Prefect pipeline, monitoring dataset preparation
- web_service: Flask API, Docker/Kubernetes deployment manifests, tests
- monitoring: Evidently notebooks, Docker Compose, Grafana configs, metrics calculators
- monitoring_reports: drift reports
- Configuration files: Dockerfile, docker-compose.yml, Kubernetes YAMLs, Makefile, CI/CD workflows, lint/test configs

##### README Highlights
- MLflow for experiment tracking & model registry
- Prefect for workflow orchestration (data prep → training → registration → deployment)
- Flask + MLflow for REST API model serving
- Evidently + Grafana for production monitoring

a. Main Components (nodes)
- Data Storage – raw CSVs, reference & production data
- Prefect Orchestrator – executes pipeline flow in src/ml_pipeline.py
- MLflow Server – backend store (SQLite) + artifact store (mlruns), UI at port 5000
- Training & Registry – src/experiment_tracking.py, src/model_registry.py
- Model Artifacts – persisted in MLflow artifact store
- Web Service – Flask app in web_service/deploy.py, containerized via Dockerfile
- Kubernetes Cluster – deployment.yaml and service.yaml under web_service/kubernetes
- Monitoring Stack
– Evidently AI (data/model drift checks)
– Grafana + Prometheus (dashboards, metrics ingestion)
– Docker Compose orchestrating monitoring services
1. Raw data → Prefect → data ingestion & preprocessing tasks
2. Prefect → MLflow (logs parameters, metrics, artifacts)
3. Prefect → calls experiment_tracking → trains models → registers best model in MLflow Registry
4. Web Service → loads latest “Production” model from MLflow → serves REST predictions
5. In production: Web Service logs requests & outputs → monitoring datasets generated → Evidently calculates drift metrics → Grafana dashboards display alerts
6. CI/CD pipelines (GitHub Actions) push code → Docker images → Kubernetes cluster → auto-deploy new versions
c. Architectural Patterns & Principles
- Pipeline Orchestration (Prefect flow with tasks)
- Experiment Tracking & Reproducibility (MLflow + versioned dependencies)
- Microservice for model serving (Flask + Docker + K8s)
- Infrastructure as Code (Docker Compose, Kubernetes YAML)
- Observability (Evidently + Grafana dashboards)
d. Key Technologies & Frameworks
- Python, scikit-learn, pandas
- Prefect, MLflow
- Flask for REST API
- Docker & Docker Compose
- Kubernetes (deployment.yaml, service.yaml)
- Evidently AI, Grafana (dashboards, datasources YAML)
- CI/CD: GitHub Actions, pylint, pytest

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DS_raw["Raw Data CSVs"]:::storage
        DS_ref["Reference & Production CSVs"]:::storage
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        PF["Prefect Flow"]:::orchestration
        PFD["Prefect Deployment"]:::orchestration
    end

    %% Experiment Layer
    subgraph "Experiment Layer"
        ET["Experiment Tracking"]:::experiment
        MR["Model Registry"]:::experiment
        MS["MLflow Server"]:::experiment_storage
    end

    %% Serving Layer
    subgraph "Serving Layer"
        FC["Flask Service"]:::serving
        DC["Docker Container"]:::infra
        K8s["Kubernetes Cluster"]:::infra
    end

    %% Monitoring Layer
    subgraph "Monitoring Layer"
        EV["Evidently AI"]:::monitor
        GD["Grafana"]:::monitor
        MC["Monitoring Compose"]:::infra
        DR["Drift Report"]:::monitor
    end

    %% CI/CD & Testing
    subgraph "CI/CD & Quality Gates"
        CI["CI/CD Pipelines"]:::infra
        TS["Tests & Linting"]:::infra
    end

    %% Flows
    DS_raw -->|"raw data"| PF
    DS_ref -->|"reference data"| PF
    PF -->|"log params/metrics"| MS
    PF -->|"call training"| ET
    ET -->|"store artifacts"| MS
    PF -->|"register best model"| MR
    MR -->|"store model"| MS
    FC -->|"load model"| MS
    FC -->|"serve predictions"| K8s
    DC -->|build image| K8s
    CI -->|"deploy image"| K8s
    FC -->|"log requests"| EV
    EV -->|"metrics output"| GD
    MC -->|"orchestrate"| EV
    MC -->|"orchestrate"| GD
    EV -->|"generate report"| DR

    %% Click Events
    click DS_raw "https://github.com/kachiann/project-mlops/blob/main/data/hour.csv"
    click DS_raw "https://github.com/kachiann/project-mlops/blob/main/data/reference.csv"
    click DS_ref "https://github.com/kachiann/project-mlops/blob/main/reference_data.csv"
    click DS_ref "https://github.com/kachiann/project-mlops/blob/main/production_data.csv"
    click PF "https://github.com/kachiann/project-mlops/blob/main/src/ml_pipeline.py"
    click PFD "https://github.com/kachiann/project-mlops/blob/main/ml_pipeline-deployment.yaml"
    click PF "https://github.com/kachiann/project-mlops/blob/main/.prefectignore"
    click ET "https://github.com/kachiann/project-mlops/blob/main/src/experiment_tracking.py"
    click ET "https://github.com/kachiann/project-mlops/blob/main/src/constants.py"
    click MR "https://github.com/kachiann/project-mlops/blob/main/src/model_registry.py"
    click MR "https://github.com/kachiann/project-mlops/blob/main/constants.py"
    click MS "https://github.com/kachiann/project-mlops/blob/main/src/model_registry.py"
    click FC "https://github.com/kachiann/project-mlops/blob/main/web_service/deploy.py"
    click FC "https://github.com/kachiann/project-mlops/blob/main/web_service/utils.py"
    click FC "https://github.com/kachiann/project-mlops/blob/main/web_service/constants.py"
    click DC "https://github.com/kachiann/project-mlops/tree/main/Dockerfile"
    click DC "https://github.com/kachiann/project-mlops/blob/main/docker-compose.yml"
    click K8s "https://github.com/kachiann/project-mlops/blob/main/web_service/kubernetes/deployment.yaml"
    click K8s "https://github.com/kachiann/project-mlops/blob/main/web_service/kubernetes/service.yaml"
    click MC "https://github.com/kachiann/project-mlops/blob/main/monitoring/docker-compose.yml"
    click EV "https://github.com/kachiann/project-mlops/blob/main/monitoring/evidently_metrics_calculations.py"
    click EV "https://github.com/kachiann/project-mlops/blob/main/monitoring/create_monitoring_datasets.py"
    click GD "https://github.com/kachiann/project-mlops/blob/main/monitoring/config/grafana_dashboards.yaml"
    click GD "https://github.com/kachiann/project-mlops/blob/main/monitoring/config/grafana_datasources.yaml"
    click DR "https://github.com/kachiann/project-mlops/blob/main/monitoring_reports/data_drift_report.html"
    click CI "https://github.com/kachiann/project-mlops/blob/main/.github/workflows/ci.yml"
    click CI "https://github.com/kachiann/project-mlops/blob/main/.github/workflows/pylint.yml"
    click TS "https://github.com/kachiann/project-mlops/blob/main/pytest.ini"
    click TS "https://github.com/kachiann/project-mlops/tree/main/tests/unit-tests"
    click TS "https://github.com/kachiann/project-mlops/tree/main/tests/integration-test"
    click notebook1 "https://github.com/kachiann/project-mlops/blob/main/notebooks/Model_training.ipynb"
    click notebook2 "https://github.com/kachiann/project-mlops/blob/main/notebooks/baseline_model_evidently.ipynb"

    %% Styles
    classDef storage fill:#d9edf7,stroke:#31708f,color:#31708f;
    classDef orchestration fill:#d0e9c6,stroke:#3c763d,color:#3c763d;
    classDef experiment fill:#fcf8e3,stroke:#8a6d3b,color:#8a6d3b;
    classDef experiment_storage fill:#f5f5f5,stroke:#333,color:#333;
    classDef serving fill:#d9f2d9,stroke:#3c763d,color:#3c763d;
    classDef infra fill:#f2dede,stroke:#a94442,color:#a94442;
    classDef monitor fill:#fcf8e3,stroke:#8a6d3b,color:#8a6d3b;
```
