How nickel5-inc/gaia works
Project type and purpose
• Gaia is a distributed, Python-based platform for geospatial ML modeling. It consists of two primary node types:
• Miners – run predictive models (soil moisture, geomagnetism, weather)
• Validators – fetch external data, store it, validate miners’ outputs, and compute scores
• It uses Fiber for networking, PostgreSQL for storage, Alembic for migrations, Cloudflare R2 for backups, and a proxy (nginx) to mediate connections.
File structure
• Top-level:
• alembic_migrations_miner/… and alembic_migrations_validator/… – DB migration scripts
• docs/ – high-level guides and protocols
• gaia/ – core code, subdivided into:
– APIcalls – HTTP clients for proxy/chain interactions
– database – DB managers and schema definitions
– miner – model runners, inference service (Docker)
– validator – data ingest, validator DB manager, sync/backup logic, basemodel evaluator
– tasks – task framework (base components, decorators, job ID logic) and domain-specific task definitions (weather, soil, geomagnetic)
– utils – shared helpers (health checks, profiling, consensus metrics)
• scripts/ – setup, monitoring, maintenance scripts
• tests/ – test cases for core functionality
Key README insights
• Miners pull data from validators or external sources, run models, and push predictions.
• Validators connect to external geospatial data APIs, populate the DB, serve miners, and score predictions.
• A proxy server fronts both miners and validators.
• Automated DB sync uses pgBackRest to R2 with primary/replica modes.
• Fiber is the underlying network layer.
Main components and interactions
1. External Data Sources (ECMWF, GFS, SMAP, Dst, Sentinel-2, etc.)
2. Proxy Server (nginx)
3. Validator Service
– validator API (Fiber)
– Validator Database Manager (SQLAlchemy + Alembic)
– Automated Sync Manager (pgBackRest <> R2)
– Basemodel Evaluator (gaia/validator/basemodel_evaluator.py)
– Sync/Backup scripts & health monitors
4. Shared PostgreSQL Database
5. Backup Storage (Cloudflare R2)
6. Miner Service
– inference runner (Dockerized app)
– Miner Database Manager
– Task framework & scoring module (miner_score_sender)
7. Chain / Subtensor Network (btcli, fiber-post-ip)
8. CLI & Setup Scripts
1. Validator Service pulls raw geospatial data → stores into PostgreSQL (via DB Manager)
2. Miner Service requests data via Validator API (through Proxy) → runs model → writes predictions back to PostgreSQL
3. Validator Service watches for new predictions → triggers Basemodel Evaluator
4. Basemodel Evaluator retrieves ground-truth & predictions → computes scores → writes scores/metrics to PostgreSQL
5. Automated Sync Manager reads DB → pushes backups to R2 (primary) and restores on replicas
6. APIcalls modules handle communication with the Subtensor chain (register node, post IP)
Architectural patterns & design principles
• Modular “task” framework: base components + defined_tasks per domain (plug-and-play)
• Layered service: API layer → business logic → persistence → backup
• Event-driven scoring: new predictions trigger validation jobs
• Infrastructure as code: Docker for inference, scripts for setup, Alembic for migrations
Relevant technologies & libraries
• Python 3.8+
• Fiber (network library)
• SQLAlchemy & Alembic (DB ORM & migrations)
• PostgreSQL + pgBackRest for reliable backups
• Docker (inference service)
• Cloudflare R2 (backup storage)
• pytest for testing
Validator Scoring Workflow (zoom-in view)
• New miner predictions inserted into DB (predictions table)
• DB Manager emits a “new_prediction” event (Fiber callback)
• ValidatorEvaluator (gaia/validator/basemodel_evaluator.py) subscribes to event
• Fetch predictions + corresponding ground-truth from DB
• Compute metrics (RMSE, bias, etc.)
• Persist results to scores/metrics tables
• Trigger any post-score hooks (e.g., update weights via gaia/validator/weights)