# mozilla-ai/lumigator architecture

> Architecture diagram of the GitHub repository mozilla-ai/lumigator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mozilla-ai/lumigator
- Repository: https://github.com/mozilla-ai/lumigator

## Overview

##### Relationships and Interactions
- Backend connected to the Database (for reading/writing metadata) and also communicating with Minio (for dataset storage/retrieval).
- Backend sending jobs (or tasks) to the Ray Cluster (which in turn might trigger jobs from the jobs evaluator/inference modules).
- Optionally, indicate how the SDK interacts with the Backend API as an external client library that can be used in notebooks or other applications.
- For deployment flexibility, note that the project supports both local (Docker Compose) and Kubernetes deployments (via Helm charts in the “infra/helm” directory).

##### Architectural Patterns and Design Principles
- Separation of Concerns: The clear division between the UI (frontend), REST API (backend), and compute (Ray cluster) makes the architecture modular.
- Containerization: With Dockerfiles for both frontend and backend, along with multiple Compose override files and Kubernetes Helm charts, the project is designed to be easily deployed and scaled.
- Extensibility: The existence of separate directories for jobs, SDKs, and schemas, as well as an API built using FastAPI, suggests that the system is designed to be extended (e.g., adding more model evaluation tasks or integrating with external APIs like OpenAI).

##### Being Detailed Yet Clear
- Mention the “jobs” components (evaluator, inference) as part of the overall job processing pipeline, linking them to both backend and Ray.
- External dependencies that might be configured later via environment variables (e.g., the possibility of connecting to external S3 services instead of Minio or remote Ray clusters).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Interaction"
        U("User")
    end

    subgraph "Frontend Layer"
        FE("Frontend UI (Vue/Vite)"):::frontend
    end

    subgraph "Backend Services"
        BE("Backend API (FastAPI)"):::backend
        DB("Database (SQLite)"):::database
        MS("Minio Object Storage"):::storage
    end

    subgraph "Compute Layer"
        RC("Ray Cluster"):::cluster
        RH("Ray Cluster Helm Chart"):::helm
        JE("Jobs - Evaluator"):::job
        JI("Jobs - Inference"):::job
    end

    subgraph "Client Libraries"
        SDK("SDK"):::sdk
        SCH("Schemas"):::schemas
    end

    %% Data Flow
    U -->|"HTTP"| FE
    FE -->|"REST"| BE
    SDK -->|"REST"| BE
    BE -->|"DB Query"| DB
    BE -->|"StorageOps"| MS
    BE -->|"SubmitJob"| RC
    RC -->|"Executes"| JE
    RC -->|"Executes"| JI
    RH --> RC

    %% Styles
    classDef frontend fill:#f9d5bb,stroke:#e5780d,stroke-width:2px;
    classDef backend fill:#bbf9d5,stroke:#0eac78,stroke-width:2px;
    classDef database fill:#d5bbf9,stroke:#780ea1,stroke-width:2px;
    classDef storage fill:#f9fbbd,stroke:#ac780e,stroke-width:2px;
    classDef cluster fill:#bbd5f9,stroke:#0e78ac,stroke-width:2px;
    classDef helm fill:#f5bbf9,stroke:#ac0e78,stroke-width:2px;
    classDef job fill:#d5f9bb,stroke:#78ac0e,stroke-width:2px;
    classDef sdk fill:#f9d5f9,stroke:#ac078a,stroke-width:2px;
    classDef schemas fill:#d5f9f9,stroke:#078aac,stroke-width:2px;

    %% Click Events
    click FE "https://github.com/mozilla-ai/lumigator/tree/main/lumigator/frontend"
    click BE "https://github.com/mozilla-ai/lumigator/tree/main/lumigator/backend"
    click RC "https://github.com/mozilla-ai/lumigator/blob/main/lumigator/rayserve/serve.py"
    click RH "https://github.com/mozilla-ai/lumigator/blob/main/infra/helm/lumigator/charts/backend/charts/ray-cluster-1.2.2.tgz"
    click JE "https://github.com/mozilla-ai/lumigator/tree/main/lumigator/jobs/evaluator"
    click JI "https://github.com/mozilla-ai/lumigator/tree/main/lumigator/jobs/inference"
    click DB "https://github.com/mozilla-ai/lumigator/blob/main/lumigator/backend/backend/db.py"
    click SDK "https://github.com/mozilla-ai/lumigator/tree/main/lumigator/sdk"
    click SCH "https://github.com/mozilla-ai/lumigator/tree/main/lumigator/schemas"
```
