How hfailab/hai-platform works
Project type and purpose
• Project Type: A full‐stack, time-sharing training orchestration platform with a Python REST API server, a CLI client, a scheduling engine, task managers, and k8s integration.
• Purpose: Provide multi-tenant training task scheduling on Kubernetes, manage Jupyter development containers, expose a CLI and UI, and support pluggable storage backends.
File structure
• api/ & uvicorn_server.py – the FastAPI HTTP server (routes, dependency injection, query/operation modules)
• client/ & plugins/ – the CLI (hai-cli) and its extension points (haienv, haiworkspace)
• scheduler/ & roman_parliament/ & server_model/ – the scheduling engine, message broker abstractions, server‐side models and implementations
• k8s/ & k8s_watcher/ – Kubernetes API client and event watchers to create/delete pods, configmaps, deployments
• experiment_manager/ – long-running processes that manage task lifecycle (checking logs, resource release, client handling)
• cloud_storage/ – pluggable storage provider interface (NFS, Ceph, OSS)
• db/ & db_schemas/ – database access layer for PostgreSQL and Redis, plus SQL schema files
• one/ & marsv2/ – container entrypoints and scripts used inside the all-in-one Docker/K8s image
• conf/ – configuration modules and toml overrides
• docs/ – static documentation site (not part of runtime)
README for architectural insights
• Components to deploy: webservice, Postgres, Redis, shared filesystem, Kubernetes cluster
• Deployment methods: docker-compose or k8s (one/up scripts)
• Users interact via hai-cli or the Studio UI (separate repo)
• override.toml drives runtime configuration for DB, scheduler, launcher, jupyter ingress
Main components & their interactions
• Client Layer
– hai-cli (Python CLI) and optional Studio UI
– communicates over HTTP/HTTPS to the API server
• API Server (FastAPI + Uvicorn)
– Exposes REST endpoints under api/app.py
– Uses server_model/* to implement domain logic
– Persists state in Postgres (user, quota, storage, etc.) and Redis (caching, locking)
– Publishes scheduling requests on an internal bus (roman_parliament)
• Scheduler Engine
– scheduler/base_model + modules configure assigners, matchers, feedbackers
– Subscribes to tasks on the bus, decides next resource assignments
– Feeds back to the API and launches via launcher or k8s client
• Experiment Manager
– A set of worker processes that watch task status, collect logs, free resources
– Subscribes/publishes via the same message broker abstractions
• Kubernetes Integration
– k8s/* and k8s_watcher/* handle pod creation, deletion, state watching
– Uses ConfigMaps (for scripts in marsv2), secrets, and RBAC to manage job pods
• Storage Layer
– cloud_storage/provider implements interfaces to NFS, Ceph, OSS
– Shared filesystem mounted into pods for code, logs, and config
• Database & Cache
– PostgreSQL stores system state, user info, quota, task metadata
– Redis used for locks, lightweight queues, and pub/sub in roman_parliament
• Configuration & Orchestration
– one/ and marsv2/ scripts prepare the container image, supervisord config, and entrypoints
– deploy/ scripts initialize DB schemas and override.toml templates
Architectural patterns & design principles
• Modular Layering: clear separation between HTTP API, domain logic (server_model), scheduler, and infrastructure (k8s, storage, DB)
• Adapter Pattern: cloud_storage/provider, db/implement, server_model/task_impl implement pluggable back-ends
• Event-Driven: roman_parliament and Redis pub/sub for decoupled communication between API, scheduler, and managers
• Dependency Injection: FastAPI dependencies in api/depends
• Plugin Architecture: client/commands and plugins/haienv, haiworkspace allow extending CLI functionality
• Infrastructure as Code: one/ and deploy/ generate k8s manifests and docker images