# data-infra/cube-studio architecture

> Architecture diagram of the GitHub repository data-infra/cube-studio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/data-infra/cube-studio
- Repository: https://github.com/data-infra/cube-studio

## Overview

##### Project type and purpose
- Cube-Studio is a full-stack, cloud-native machine-learning platform.
- Backend: Python Flask (myapp/), Celery tasks (myapp/tasks/), SQLAlchemy models/migrations (myapp/models, myapp/migrations).
- Frontend: React-based UI under myapp/frontend, plus two specialized React apps: vision and visionPlus.
- Auxiliary modules: AIHub integrations (aihub/), job-template repos for pipeline tasks, Docker/Kubernetes deployment under install/.

##### Main components
a. Client-side UIs (Browser)
- Cube-Studio Web Console (myapp/frontend)
- Vision Flow Editor (myapp/vision)
- VisionPlus Flow Editor (myapp/visionPlus)
b. API Layer & Web Server
- Flask app (myapp/run.py → myapp/views/, myapp/cli.py)
- Authentication/SSO (myapp/security.py)
c. Database
- Relational DB (MySQL/PostgreSQL) storing metadata, user/role info, pipelines, jobs
- SQLAlchemy + Alembic migrations
d. Task Queue & Workers
- Celery (myapp/tasks/) with Redis/RabbitMQ broker for asynchronous jobs (notebook launches, pipeline execution, model training)
- Worker images pulled from job-template/ for distributed frameworks (Ray, Spark, TFJob, PyTorchJob, etc.)
e. Storage
- Persistent volumes (NFS, Ceph, OSS, S3/Minio) mounted into pods
- PVC/PV templates defined under install/kubernetes/…
f. Cluster Orchestration
- Kubernetes clusters (multi-cluster support)
- CRDs/operators for TFJob, PyTorchJob, PaddleJob, Mindspore, Volcano, NNI, etc.
- Helm charts under install/kubernetes/alluxio, argo, ceph, kubeflow/train-operator, etc.
g. Monitoring & Logging
- Prometheus + Grafana (install/kubernetes/prometheus)
- EFK stack (install/kubernetes/efk)
- Zipkin/tracing (optional)
h. External Integrations
- AIHub model marketplace (aihub/)
- Data-platform (sqllab, hive, presto) via adapter code in myapp/utils/py_prometheus, py_k8s
- SSO/OAuth/LDAP, webhooks for alerting

##### Relationships and data flows
1. User → Web UI (frontend/vision/visionPlus)
- HTTP(S) requests → API Layer
2. API Layer ↔ Database
- CRUD operations for users, projects, pipelines, jobs
3. API Layer → Celery Broker
- Enqueue background tasks: deploy notebook, start pipeline, launch training job
4. Celery Workers ↔ Kubernetes API
- Workers create Kubernetes Job/CRD objects for distributed workloads
5. Kubernetes Workloads ↔ Storage
- Mount PVCs for code, data, model artifacts
6. Monitoring
- Prometheus scrapes metrics from API Layer, Celery, Kubernetes, node exporters
7. AIHub & Job-Template
- API Layer fetches AIHub model info, pipelines download container images from job-template repositories

##### Architectural patterns & design principles
- Micro-frontends: three independent React SPAs (main console, vision, visionPlus) served by Flask static endpoints
- Modular backend: Blueprints/controllers for each domain (view_pipeline, view_job_template, view_nni, etc.)
- Asynchronous processing: Celery for heavy tasks; non-blocking UI
- Infrastructure as code: Helm/Kustomize YAML under install/kubernetes/…
- Template-driven pipelines: reusable Docker images under job-template/

##### Elements
- Labels: “React Console”, “Vision Editor”, “Flask API”, “Celery Broker/Workers”, “MySQL/Postgres”, “Redis/RabbitMQ”, “Kubernetes Cluster(s)”, “PVC/PV Storage”, “Prometheus/Grafana”
- HTTP(S) → API
- API → DB
- API → Broker
- Broker → Worker
- Worker → K8s API
- Workload Pods → Storage
- Monitoring → Scrape Targets

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% UI Layer
    subgraph "UI Layer"
        ReactConsole["React Console UI<br/>(React)"]:::ui
        VisionEditor["Vision Flow Editor<br/>(React)"]:::ui
        VisionPlus["VisionPlus Flow Editor<br/>(React)"]:::ui
    end

    %% API Layer
    subgraph "API Layer"
        FlaskAPI["Flask API<br/>(Flask)"]:::backend
        Auth["Authentication/SSO<br/>(SSO/OAuth/LDAP)"]:::backend
        Controllers["Controllers<br/>(Blueprints)"]:::backend
        CLI["CLI Commands"]:::backend
        Models["SQLAlchemy Models"]:::backend
        Migrations["Alembic Migrations"]:::backend
    end

    %% Data Layer
    subgraph "Data Layer"
        DB["Relational DB<br/>(MySQL/Postgres)"]:::db
        Broker["Redis/RabbitMQ<br/>(Broker)"]:::db
    end

    %% Worker Layer
    subgraph "Worker Layer"
        CeleryApp["Celery App<br/>(Celery)"]:::worker
        CeleryUtils["Celery Utils"]:::worker
        Workers["Celery Workers"]:::worker
        AIHubMod["AIHub Integration"]:::worker
        JobTemplate["Job Template Repos"]:::worker
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        Helm["Helm Charts/CRDs"]:::infra
        K8sAPI["Kubernetes API"]:::infra
    end

    %% Compute Layer
    subgraph "Compute Layer"
        K8sClusters["Kubernetes Clusters<br/>(Training Jobs)"]:::infra
    end

    %% Storage Layer
    subgraph "Storage Layer"
        Storage["PVC/PV & Object Stores"]:::infra
    end

    %% Monitoring Layer
    subgraph "Monitoring Layer"
        Prom["Prometheus"]:::infra
        Grafana["Grafana"]:::infra
        EFK["EFK Stack"]:::infra
    end

    %% External Adapters
    subgraph "External Integrations"
        PyK8s["py_k8s.py"]:::infra
        PyRay["py_ray.py"]:::infra
        PyProm["py_prometheus.py"]:::infra
        SQLLab["SQLLab Adapter"]:::infra
    end

    %% Connections
    ReactConsole -->|HTTPS| FlaskAPI
    VisionEditor -->|HTTPS| FlaskAPI
    VisionPlus -->|HTTPS| FlaskAPI

    FlaskAPI -->|CRUD| DB
    FlaskAPI -->|Enqueue Tasks| Broker
    FlaskAPI -->|Uses| Controllers
    FlaskAPI -->|Uses| Auth
    FlaskAPI -->|Uses| CLI
    FlaskAPI -->|Uses| Models
    FlaskAPI -->|Uses| Migrations

    Broker -->|Dispatch| CeleryApp
    CeleryApp -->|Import| CeleryUtils
    CeleryApp -->|Spawn| Workers
    Workers -->|Invoke| K8sAPI
    AIHubMod -->|Fetch Models| FlaskAPI
    JobTemplate -->|Provide Images| Workers

    K8sAPI -->|Orchestrate| K8sClusters
    K8sClusters -->|Mount| Storage

    Prom -->|Scrape| FlaskAPI
    Prom -->|Scrape| CeleryApp
    Prom -->|Scrape| K8sClusters
    Grafana -->|Visualize| Prom
    EFK -->|Aggregate Logs| FlaskAPI

    FlaskAPI -->|Calls| PyK8s
    FlaskAPI -->|Calls| PyRay
    FlaskAPI -->|Calls| PyProm
    FlaskAPI -->|Calls| SQLLab

    %% Click Events
    click ReactConsole "https://github.com/data-infra/cube-studio/tree/main/myapp/frontend/"
    click VisionEditor "https://github.com/data-infra/cube-studio/tree/main/myapp/vision/"
    click VisionPlus "https://github.com/data-infra/cube-studio/tree/main/myapp/visionPlus/"
    click FlaskAPI "https://github.com/data-infra/cube-studio/blob/main/myapp/run.py"
    click CLI "https://github.com/data-infra/cube-studio/blob/main/myapp/cli.py"
    click Controllers "https://github.com/data-infra/cube-studio/tree/main/myapp/views/"
    click Auth "https://github.com/data-infra/cube-studio/blob/main/myapp/security.py"
    click Models "https://github.com/data-infra/cube-studio/tree/main/myapp/models/"
    click Migrations "https://github.com/data-infra/cube-studio/tree/main/myapp/migrations/"
    click CeleryApp "https://github.com/data-infra/cube-studio/tree/main/myapp/tasks/"
    click CeleryUtils "https://github.com/data-infra/cube-studio/blob/main/myapp/utils/celery.py"
    click AIHubMod "https://github.com/data-infra/cube-studio/tree/main/aihub/"
    click JobTemplate "https://github.com/data-infra/cube-studio/tree/main/job-template/"
    click Helm "https://github.com/data-infra/cube-studio/tree/main/install/kubernetes/"
    click K8sAPI "https://github.com/data-infra/cube-studio/tree/main/install/kubernetes/"
    click K8sClusters "https://github.com/data-infra/cube-studio/tree/main/install/kubernetes/cube/"
    click Helm "https://github.com/data-infra/cube-studio/tree/main/install/kubernetes/alluxio/"
    click Helm "https://github.com/data-infra/cube-studio/tree/main/install/kubernetes/kubeflow/train-operator/"
    click Helm "https://github.com/data-infra/cube-studio/tree/main/install/kubernetes/volcano/"
    click Storage "https://github.com/data-infra/cube-studio/blob/main/install/kubernetes/pv-pvc-*.yaml"
    click Prom "https://github.com/data-infra/cube-studio/tree/main/install/kubernetes/prometheus/"
    click EFK "https://github.com/data-infra/cube-studio/tree/main/install/kubernetes/efk/"
    click PyK8s "https://github.com/data-infra/cube-studio/blob/main/myapp/utils/py/py_k8s.py"
    click PyRay "https://github.com/data-infra/cube-studio/blob/main/myapp/utils/py/py_ray.py"
    click PyProm "https://github.com/data-infra/cube-studio/blob/main/myapp/utils/py/py_prometheus.py"
    click SQLLab "https://github.com/data-infra/cube-studio/tree/main/myapp/utils/sqllab/"

    %% Styles
    classDef ui fill:#b2f5ba,stroke:#2f855a,color:#1a202c
    classDef backend fill:#bee3f8,stroke:#2b6cb0,color:#2a4365
    classDef worker fill:#fbd38d,stroke:#dd6b20,color:#7b341e
    classDef db fill:#e2e8f0,stroke:#4a5568,color:#2d3748
    classDef infra fill:#f7fafc,stroke:#a0aec0,color:#4a5568
```
