How data-infra/cube-studio works
Project type and purpose
• Cube-Studio is a full-stack, cloud-native machine-learning platform.
• Backend: Python Flask (myapp/), Celery tasks (myapp/tasks/), SQLAlchemy models/migrations (myapp/models, myapp/migrations).
• Frontend: React-based UI under myapp/frontend, plus two specialized React apps: vision and visionPlus.
• Auxiliary modules: AIHub integrations (aihub/), job-template repos for pipeline tasks, Docker/Kubernetes deployment under install/.
Main components
a. Client-side UIs (Browser)
• Cube-Studio Web Console (myapp/frontend)
• Vision Flow Editor (myapp/vision)
• VisionPlus Flow Editor (myapp/visionPlus)
b. API Layer & Web Server
• Flask app (myapp/run.py → myapp/views/, myapp/cli.py)
• Authentication/SSO (myapp/security.py)
c. Database
• Relational DB (MySQL/PostgreSQL) storing metadata, user/role info, pipelines, jobs
• SQLAlchemy + Alembic migrations
d. Task Queue & Workers
• Celery (myapp/tasks/) with Redis/RabbitMQ broker for asynchronous jobs (notebook launches, pipeline execution, model training)
• Worker images pulled from job-template/ for distributed frameworks (Ray, Spark, TFJob, PyTorchJob, etc.)
e. Storage
• Persistent volumes (NFS, Ceph, OSS, S3/Minio) mounted into pods
• PVC/PV templates defined under install/kubernetes/…
f. Cluster Orchestration
• Kubernetes clusters (multi-cluster support)
• CRDs/operators for TFJob, PyTorchJob, PaddleJob, Mindspore, Volcano, NNI, etc.
• Helm charts under install/kubernetes/alluxio, argo, ceph, kubeflow/train-operator, etc.
g. Monitoring & Logging
• Prometheus + Grafana (install/kubernetes/prometheus)
• EFK stack (install/kubernetes/efk)
• Zipkin/tracing (optional)
h. External Integrations
• AIHub model marketplace (aihub/)
• Data-platform (sqllab, hive, presto) via adapter code in myapp/utils/py_prometheus, py_k8s
• SSO/OAuth/LDAP, webhooks for alerting
Relationships and data flows
1. User → Web UI (frontend/vision/visionPlus)
• HTTP(S) requests → API Layer
2. API Layer ↔ Database
• CRUD operations for users, projects, pipelines, jobs
3. API Layer → Celery Broker
• Enqueue background tasks: deploy notebook, start pipeline, launch training job
4. Celery Workers ↔ Kubernetes API
• Workers create Kubernetes Job/CRD objects for distributed workloads
5. Kubernetes Workloads ↔ Storage
• Mount PVCs for code, data, model artifacts
6. Monitoring
• Prometheus scrapes metrics from API Layer, Celery, Kubernetes, node exporters
7. AIHub & Job-Template
• API Layer fetches AIHub model info, pipelines download container images from job-template repositories
Architectural patterns & design principles
• Micro-frontends: three independent React SPAs (main console, vision, visionPlus) served by Flask static endpoints
• Modular backend: Blueprints/controllers for each domain (view_pipeline, view_job_template, view_nni, etc.)
• Asynchronous processing: Celery for heavy tasks; non-blocking UI
• Infrastructure as code: Helm/Kustomize YAML under install/kubernetes/…
• Template-driven pipelines: reusable Docker images under job-template/
Elements
• Labels: “React Console”, “Vision Editor”, “Flask API”, “Celery Broker/Workers”, “MySQL/Postgres”, “Redis/RabbitMQ”, “Kubernetes Cluster(s)”, “PVC/PV Storage”, “Prometheus/Grafana”
• HTTP(S) → API
• API → DB
• API → Broker
• Broker → Worker
• Worker → K8s API
• Workload Pods → Storage
• Monitoring → Scrape Targets