# nickel5-inc/gaia architecture

> Architecture diagram of the GitHub repository nickel5-inc/gaia, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nickel5-inc/gaia
- Repository: https://github.com/nickel5-inc/gaia

## Overview

##### Project type and purpose
- Gaia is a distributed, Python-based platform for geospatial ML modeling. It consists of two primary node types:
- Miners – run predictive models (soil moisture, geomagnetism, weather)
- Validators – fetch external data, store it, validate miners’ outputs, and compute scores
- It uses Fiber for networking, PostgreSQL for storage, Alembic for migrations, Cloudflare R2 for backups, and a proxy (nginx) to mediate connections.

##### File structure
- Top-level:
- alembic_migrations_miner/… and alembic_migrations_validator/… – DB migration scripts
- docs/ – high-level guides and protocols
- gaia/ – core code, subdivided into:
– APIcalls – HTTP clients for proxy/chain interactions
– database – DB managers and schema definitions
– miner – model runners, inference service (Docker)
– validator – data ingest, validator DB manager, sync/backup logic, basemodel evaluator
– tasks – task framework (base components, decorators, job ID logic) and domain-specific task definitions (weather, soil, geomagnetic)
– utils – shared helpers (health checks, profiling, consensus metrics)
- scripts/ – setup, monitoring, maintenance scripts
- tests/ – test cases for core functionality

##### Key README insights
- Miners pull data from validators or external sources, run models, and push predictions.
- Validators connect to external geospatial data APIs, populate the DB, serve miners, and score predictions.
- A proxy server fronts both miners and validators.
- Automated DB sync uses pgBackRest to R2 with primary/replica modes.
- Fiber is the underlying network layer.

##### Main components and interactions
1. External Data Sources (ECMWF, GFS, SMAP, Dst, Sentinel-2, etc.)
2. Proxy Server (nginx)
3. Validator Service
– validator API (Fiber)
– Validator Database Manager (SQLAlchemy + Alembic)
– Automated Sync Manager (pgBackRest <> R2)
– Basemodel Evaluator (gaia/validator/basemodel_evaluator.py)
– Sync/Backup scripts & health monitors
4. Shared PostgreSQL Database
5. Backup Storage (Cloudflare R2)
6. Miner Service
– inference runner (Dockerized app)
– Miner Database Manager
– Task framework & scoring module (miner_score_sender)
7. Chain / Subtensor Network (btcli, fiber-post-ip)
8. CLI & Setup Scripts

1. Validator Service pulls raw geospatial data → stores into PostgreSQL (via DB Manager)
2. Miner Service requests data via Validator API (through Proxy) → runs model → writes predictions back to PostgreSQL
3. Validator Service watches for new predictions → triggers Basemodel Evaluator
4. Basemodel Evaluator retrieves ground-truth & predictions → computes scores → writes scores/metrics to PostgreSQL
5. Automated Sync Manager reads DB → pushes backups to R2 (primary) and restores on replicas
6. APIcalls modules handle communication with the Subtensor chain (register node, post IP)

##### Architectural patterns & design principles
- Modular “task” framework: base components + defined_tasks per domain (plug-and-play)
- Layered service: API layer → business logic → persistence → backup
- Event-driven scoring: new predictions trigger validation jobs
- Infrastructure as code: Docker for inference, scripts for setup, Alembic for migrations

##### Relevant technologies & libraries
- Python 3.8+
- Fiber (network library)
- SQLAlchemy & Alembic (DB ORM & migrations)
- PostgreSQL + pgBackRest for reliable backups
- Docker (inference service)
- Cloudflare R2 (backup storage)
- pytest for testing

##### Validator Scoring Workflow (zoom-in view)
- New miner predictions inserted into DB (predictions table)
- DB Manager emits a “new_prediction” event (Fiber callback)
- ValidatorEvaluator (gaia/validator/basemodel_evaluator.py) subscribes to event
- Fetch predictions + corresponding ground-truth from DB
- Compute metrics (RMSE, bias, etc.)
- Persist results to scores/metrics tables
- Trigger any post-score hooks (e.g., update weights via gaia/validator/weights)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "External Systems"
        ExtData["External Data Sources"]:::external
        ChainNet["Subtensor Network"]:::external
    end

    Proxy["Proxy Server (nginx)"]:::core

    subgraph "Validator Service"
        VAPI["Validator API"]:::core
        VDBM["Validator DB Manager"]:::core
        BME["Basemodel Evaluator"]:::scoring
        ASM["Automated Sync Manager"]:::core
        BMBackup["Backup Manager"]:::core
        RM["Restore Manager"]:::core
        subgraph "pgBackRest Scripts"
            PDiag["diagnose.sh"]:::scripts
            PMon["monitor-sync.sh"]:::scripts
            PPrim["setup-primary.sh"]:::scripts
            Prep["setup-replica.sh"]:::scripts
            PTmpl["pg_hba.conf.template"]:::scripts
        end
        HCVal["Health Check Utility"]:::core
        IngestTok["External Data Ingest Tokens"]:::core
        SubMgr["Subtensor Chain Manager"]:::core
    end

    subgraph "Miner Service"
        MProc["Miner Core Process"]:::core
        subgraph "Inference Runner"
            MF_Docker["Dockerfile"]:::core
            MF_Main["main.py"]:::core
            MF_Run["inference_runner.py"]:::core
            MF_Data["data_preprocessor.py"]:::core
            MF_Config["settings.yaml"]:::core
        end
        MDBM["Miner DB Manager"]:::core
        HCMin["Health Check Utility"]:::core
        JWTGen["JWT Secret Generator"]:::core
        Subnet["Subnet Utility"]:::core
    end

    subgraph "Shared Database"
        Postgres["PostgreSQL DB"]:::storage
        DBMgr["Database Manager"]:::core
        subgraph "Schemas & Constants"
            VSchema["validator_schema.py"]:::core
            SSchema["miner_schema.py"]:::core
            Consts["constants.py"]:::core
        end
    end

    R2["Cloudflare R2"]:::storage

    subgraph "Task Framework"
        Inputs["inputs.py"]:::core
        Outputs["outputs.py"]:::core
        Preproc["preprocessing.py"]:::core
        ScoringMech["scoring_mechanism.py"]:::core
        Decorators["decorators.py"]:::core
        DJID["deterministic_job_id.py"]:::core
        TaskBase["task.py"]:::core
    end

    subgraph "Domain Tasks"
        Geom["geomagnetic"]:::core
        Soil["soilmoisture"]:::core
        Weather["weather"]:::core
    end

    subgraph "API Clients"
        ScoreSender["miner_score_sender.py"]:::core
        ChainAPI["website_api.py"]:::core
    end

    subgraph "Scripts & Migrations"
        ProxySetup["setup_proxy_server.sh"]:::scripts
        SetupScript["scripts/setup.py"]:::scripts
        ValMon["scripts/validator_monitor.py"]:::scripts
        WipeR2["scripts/wipe_r2_bucket.py"]:::scripts
        AMigs["alembic_migrations_miner/"]:::scripts
        AVMigs["alembic_migrations_validator/"]:::scripts
    end

    ExtData -->|HTTP| Proxy
    Proxy -->|HTTP| VAPI
    MProc -->|HTTP| Proxy
    Proxy -->|HTTP| MProc

    VAPI -->|calls| VDBM
    VDBM -->|SQL| Postgres
    DBMgr -->|SQL| Postgres
    Postgres -->|new_prediction event| BME
    Postgres -->|ground-truth tables| BME
    BME -->|writes scores| Postgres

    ASM -->|reads DB| Postgres
    ASM -->|pgBackRest API| BMBackup
    BMBackup -->|stores backup| R2
    RM -->|restores| Postgres

    ChainNet -->|RPC| SubMgr
    SubMgr -->|Fiber| ChainNet
    ChainAPI -->|HTTP| ChainNet
    ScoreSender -->|HTTP| ChainNet
    MProc -->|calls| ScoreSender

    VAPI -->|uses| HCVal
    VAPI -->|uses| IngestTok

    MProc -->|uses| HCMin
    MProc -->|uses| JWTGen
    MProc -->|uses| Subnet

    click VAPI "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/validator.py"
    click VDBM "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/database/validator_database_manager.py"
    click BME "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/basemodel_evaluator.py"
    click ASM "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/sync/auto_sync_manager.py"
    click BMBackup "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/sync/backup_manager.py"
    click RM "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/sync/restore_manager.py"
    click PDiag "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/sync/pgbackrest/diagnose.sh"
    click PMon "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/sync/pgbackrest/monitor-sync.sh"
    click PPrim "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/sync/pgbackrest/setup-primary.sh"
    click Prep "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/sync/pgbackrest/setup-replica.sh"
    click PTmpl "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/sync/pgbackrest/templates/pg_hba.conf.template"
    click HCVal "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/utils/health_check_validator.py"
    click IngestTok "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/utils/earthdata_tokens.py"
    click SubMgr "https://github.com/nickel5-inc/gaia/blob/main/gaia/validator/utils/substrate_manager.py"
    click DBMgr "https://github.com/nickel5-inc/gaia/blob/main/gaia/database/database_manager.py"
    click VSchema "https://github.com/nickel5-inc/gaia/blob/main/gaia/database/validator_schema.py"
    click SSchema "https://github.com/nickel5-inc/gaia/blob/main/gaia/database/miner_schema.py"
    click Consts "https://github.com/nickel5-inc/gaia/blob/main/gaia/database/constants.py"
    click MProc "https://github.com/nickel5-inc/gaia/blob/main/gaia/miner/miner.py"
    click MF_Docker "https://github.com/nickel5-inc/gaia/tree/main/gaia/miner/inference_service/Dockerfile"
    click MF_Main "https://github.com/nickel5-inc/gaia/blob/main/gaia/miner/inference_service/app/main.py"
    click MF_Run "https://github.com/nickel5-inc/gaia/blob/main/gaia/miner/inference_service/app/inference_runner.py"
    click MF_Data "https://github.com/nickel5-inc/gaia/blob/main/gaia/miner/inference_service/app/data_preprocessor.py"
    click MF_Config "https://github.com/nickel5-inc/gaia/blob/main/gaia/miner/inference_service/config/settings.yaml"
    click MDBM "https://github.com/nickel5-inc/gaia/blob/main/gaia/miner/database/miner_database_manager.py"
    click HCMin "https://github.com/nickel5-inc/gaia/blob/main/gaia/miner/utils/health_check_miner.py"
    click JWTGen "https://github.com/nickel5-inc/gaia/blob/main/gaia/miner/utils/generate_jwt_secret.py"
    click Subnet "https://github.com/nickel5-inc/gaia/blob/main/gaia/miner/utils/subnet.py"
    click Inputs "https://github.com/nickel5-inc/gaia/blob/main/gaia/tasks/base/components/inputs.py"
    click Outputs "https://github.com/nickel5-inc/gaia/blob/main/gaia/tasks/base/components/outputs.py"
    click Preproc "https://github.com/nickel5-inc/gaia/blob/main/gaia/tasks/base/components/preprocessing.py"
    click ScoringMech "https://github.com/nickel5-inc/gaia/blob/main/gaia/tasks/base/components/scoring_mechanism.py"
    click Decorators "https://github.com/nickel5-inc/gaia/blob/main/gaia/tasks/base/decorators.py"
    click DJID "https://github.com/nickel5-inc/gaia/blob/main/gaia/tasks/base/deterministic_job_id.py"
    click TaskBase "https://github.com/nickel5-inc/gaia/blob/main/gaia/tasks/base/task.py"
    click Geom "https://github.com/nickel5-inc/gaia/tree/main/gaia/tasks/defined_tasks/geomagnetic"
    click Soil "https://github.com/nickel5-inc/gaia/tree/main/gaia/tasks/defined_tasks/soilmoisture"
    click Weather "https://github.com/nickel5-inc/gaia/tree/main/gaia/tasks/defined_tasks/weather"
    click ScoreSender "https://github.com/nickel5-inc/gaia/blob/main/gaia/APIcalls/miner_score_sender.py"
    click ChainAPI "https://github.com/nickel5-inc/gaia/blob/main/gaia/APIcalls/website_api.py"
    click ProxySetup "https://github.com/nickel5-inc/gaia/blob/main/setup_proxy_server.sh"
    click SetupScript "https://github.com/nickel5-inc/gaia/blob/main/scripts/setup.py"
    click ValMon "https://github.com/nickel5-inc/gaia/blob/main/scripts/validator_monitor.py"
    click WipeR2 "https://github.com/nickel5-inc/gaia/blob/main/scripts/wipe_r2_bucket.py"
    click AMigs "https://github.com/nickel5-inc/gaia/tree/main/alembic_migrations_miner/"
    click AVMigs "https://github.com/nickel5-inc/gaia/tree/main/alembic_migrations_validator/"

    classDef core fill:#ADD8E6,stroke:#000
    classDef storage fill:#90EE90,stroke:#000
    classDef external fill:#FFA500,stroke:#000
    classDef scoring fill:#DDA0DD,stroke:#000
    classDef scripts fill:#D3D3D3,stroke:#000
```
