How simerusm/arceus works
Project type and purpose
• Arceus is a distributed deep-learning training framework with a full-stack architecture.
• Purpose: Orchestrate model-parallel neural‐network training across Apple M-series devices on a local network, and support single-machine transformer (GPT) training.
File structure
• Top-level directories:
• api/ → Flask REST API server for job registration, device registration, orchestration
• nn/ → Core Python modules: coordinator, device_client, device_server for distributed training
• protos/ → gRPC service definitions and generated stubs for device-to-server RPC
• gpt/ → Transformer model code and training pipeline
• frontend/ → Next.js + Tauri desktop UI for monitoring and job setup
• Configuration/build: requirements.txt, Cargo.toml (Tauri), next.config.mjs, postcss.config.mjs, tailwind.config.ts
README for additional insights
• REST endpoints under api/server.py for jobs, network initialization, and training
• Devices register themselves via HTTP, then orchestrator uses gRPC (device_service.proto) to push work
• Two training modes:
• Model-parallel across devices (requires device_client processes on each node)
• Transformer training (runs entirely within the Flask process + gpt/model.py)
a. Main components
• User interface (Next.js + Tauri app)
• REST API server (Flask app.py & server.py)
• Orchestrator / Scheduler (nn/coordinator.py)
• Device workers (nn/device_client.py + nn/device_server.py)
• gRPC layer (protos/device_service.proto)
• Data sources: local dataset files (e.g., MNIST), gpt/data.txt
• (Optional) In-memory job registry or future database
b. Relationships & interactions
1. User clicks “Create Job” in Frontend → HTTP POST to Flask REST API
2. API server stores job configuration and returns a job ID
3. Device workers call HTTP POST /api/devices/register to register with the server
4. Coordinator (invoked by REST endpoint) sends gRPC Initialize messages to each device’s gRPC server
5. For each training epoch, Coordinator invokes gRPC Train/Forward/Backward calls to devices in pipeline order
6. Devices compute segment forward/backward on MPS, return activations/gradients via gRPC
7. Coordinator aggregates results; REST API returns progress back to Frontend via polling or WebSocket (future enhancement)
8. For transformer jobs, API directly invokes gpt/model.py training loop (no gRPC)
c. Architectural patterns & design principles
• Separation of concerns: UI ←→ REST API ←→ Orchestrator ←→ Workers
• Micro-service style: independent device worker processes communicating over gRPC
• Model parallel pipeline: each layer segment deployed to a different device
• Pluggable training modes (add new model types by extending gpt/ or nn/)
d. Key technologies & frameworks
• Python 3, Flask (REST API)
• Protobuf + gRPC (device_service.proto)
• Apple MPS (device compute backend)
• Next.js + React + Tailwind CSS + Tauri (desktop UI)
Elements
• Clear, concise labels for each component (e.g., “Flask API Server”, “Device Worker (gRPC)” )
Detailed capture of essential architecture
• Don’t just list files—group by function: UI vs. API vs. Compute
• How protos tie into device_server & device_client
• Both training modes side-by-side, with shared and unique flows