How datura-ai/lium-io works
Project Type and Purpose
• This is a decentralized, peer-to-peer GPU rental marketplace built on Bittensor Subnet 51.
• It comprises three microservices – Executor, Miner, Validator – plus a Python SDK/CLI library (Datura) and a frontend UI (lium.io).
File Structure
• Top‐level services under neurons/:
• neurons/executor: handles task execution on GPU pods
• neurons/miners: manages miner registration, pricing, and portal API interactions
• neurons/validators: performs hardware verification and scoring pipelines
• Shared library under datura/: Python package providing request/consumer abstractions for both CLI and service clients.
• Infrastructure: Dockerfiles, docker-compose configs, Alembic migrations, CI/CD workflows (.github/workflows), scripts for installation and deployment.
• Each service uses a relational database (via SQLAlchemy/Alembic), REST or RPC routes, and middleware components.
README Insights
• High-level architecture calls out Miners, Validators, Renters (frontend & CLI), and the Bittensor blockchain as external component.
• The frontend (lium.io) and CLI (lium-cli / Datura) are consumer clients of the services.
Main Components
a. Frontend & CLI Clients
• lium.io Web App (not in this repo)
• Datura-powered lium-cli
b. Microservices
• Miner Service (neurons/miners)
• Validator Service (neurons/validators)
• Executor Service (neurons/executor)
c. Datastore Layer
• PostgreSQL (or chosen RDBMS) per service (each has its own schema/migrations)
d. External Systems
• Bittensor Subnet blockchain (for token transfers)
• SSH endpoints on GPU machines
e. Messaging/Protocol
• REST / RPC APIs between clients and microservices
• Custom protocol modules in protocol/ folders
Relationships and Data Flows
• Renters → Web UI / CLI → Miner Service: browse & request GPU pods
• Miner Service ↔ Executor Service: assign tasks to pods, track logs
• Validator Service → Miner Service: pull miner details; Validator connects via SSH to GPU hosts
• Executor Service → Validator Service: report task results or metrics
• All services → blockchain API: validators pay miners using $TAO tokens
• Microservices ↔ their local DBs for state, logs, pricing, and tasks
Architectural Patterns & Principles
• Microservices: each domain (miner, validator, executor) runs in its own container, own database
• CQRS‐style separation: read models (DAOs) vs write operations (services/payloads)
• Middleware pipelines in Validator for modular “checks” (GPU count, fingerprint, etc.)
• Containerized deployments using Docker & docker-compose for local/dev/prod
• CI/CD per service via GitHub Actions
Technologies & Frameworks
• Python 3 / FastAPI or Flask (implied by routes/ folders)
• SQLAlchemy + Alembic for ORM & migrations
• Docker + docker-compose for container orchestration
• PDM / pyproject.toml for dependency management
• Datura Python package for client abstractions