# psychefoundation/psyche architecture

> Architecture diagram of the GitHub repository psychefoundation/psyche, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/psychefoundation/psyche
- Repository: https://github.com/psychefoundation/psyche

## Overview

##### Project type and purpose
- Psyche is a monorepo for a distributed, transformer‐based ML training framework. It provides both (a) a “centralized” reference architecture (Rust clients, server, local testnet) and (b) a “decentralized” variant built on Solana Anchor programs (authorizer, coordinator, mining‐pool, treasurer) plus off‐chain tooling, Python bindings, a web frontend/backend, telemetry, Docker/Nix infrastructure, documentation, and shared libraries.

##### File structure
- Top-level “architectures” folder splits into:
· centralized: client, server, shared protocol, local testnet, testing harness
· decentralized: on-chain programs (authorizer, coordinator, mining-pool, treasurer), client CLI, tooling, chaos/testing
- “shared” folder: core libraries for client, coordinator, data‐provider, modeling, network (P2P), metrics, eval, tui, watcher
- “python” folder for Python extension (PyO3 sidecar), dtensor helpers, high-level APIs
- “website” folder: Node.js backend (TypeScript + MikroORM), React/TypeScript frontend, shared TypeScript types, WebAssembly (Rust→wasm)
- “telemetry”: Prometheus, Grafana, Loki, OpenTelemetry collector configs
- “docker”, “nix”, “scripts”, “tools”: build & deployment automation

##### README insights
- Core goal: enable trustless, collaborative training over the Internet via off‐chain compute nodes and on‐chain coordination & reward distribution.

## Diagram (Mermaid)

```mermaid
flowchart TB
  subgraph "Centralized Architecture"
    ClientRust["Rust Client"]:::offchain
    ServerRust["Server"]:::offchain
    SharedProtocol["Shared Protocol Lib"]:::shared
    LocalTestnet["Local Testnet"]:::offchain
    TestingHarness["Testing Harness"]:::offchain
  end

  subgraph "Decentralized Architecture"
    subgraph "Solana Programs"
      Authorizer["Authorizer Program"]:::onchain
      CoordinatorProg["Coordinator Program"]:::onchain
      MiningPool["Mining Pool Program"]:::onchain
      Treasurer["Treasurer Program"]:::onchain
    end
    CLIClient["CLI Client"]:::offchain
    Tooling["Tooling Lib"]:::offchain
    ChaosTests["Chaos Tests"]:::offchain
  end

  subgraph "Shared Libraries"
    CLib["Client Lib"]:::shared
    CoordLib["Coordinator Logic Lib"]:::shared
    CoreLib["Core Util Lib"]:::shared
    DataProvider["Data Provider"]:::shared
    ModelLib["Modeling Lib"]:::shared
    NetworkLib["P2P Network Lib"]:::shared
    MetricsLib["Metrics Lib"]:::shared
    EvalLib["Evaluation Suite"]:::shared
    TUILib["TUI Lib"]:::shared
    WatcherLib["Watcher Service"]:::shared
  end

  subgraph "Python"
    PythonSidecar["Sidecar/API"]:::python
    Dtensor["dtensor Helpers & Models"]:::python
    PyExtension["PyO3 Extension"]:::python
  end

  subgraph "Website"
    Frontend["React Frontend"]:::frontend
    BackendAPI["Node Backend API"]:::frontend
    WebTypes["Shared Web Types"]:::frontend
    WasmUtils["WASM Utilities"]:::frontend
  end

  subgraph "Telemetry"
    Prometheus["Prometheus"]:::telemetry
    Grafana["Grafana"]:::telemetry
    Loki["Loki"]:::telemetry
    OTelCollector["OTel Collector"]:::telemetry
  end

  subgraph "Infrastructure"
    Docker["Docker Compose & Scripts"]:::infra
    Nix["Nix Flakes & DevShell"]:::infra
    CI["CI/CD Workflows"]:::infra
    Scripts["Helper Scripts"]:::infra
  end

  %% Relationships
  ClientRust -->|"fetch batches"| DataProvider
  ClientRust -->|"exchange updates"| NetworkLib
  ClientRust -->|"register/join run"| BackendAPI

  BackendAPI -->|"serve UI"| Frontend
  BackendAPI -->|"read on-chain state"| CoordinatorProg
  BackendAPI -->|"read on-chain state"| Authorizer
  BackendAPI -->|"read on-chain state"| Treasurer
  BackendAPI -->|"read on-chain state"| MiningPool
  BackendAPI -->|"listen events"| CoordinatorProg

  CLIClient -->|"submit txs"| CoordinatorProg
  CLIClient -->|"uses tooling"| Tooling
  ChaosTests -->|"test integrations"| Authorizer

  PythonSidecar -->|"calls lib"| CLib
  PythonSidecar -->|"uses helpers"| Dtensor

  Frontend --> BackendAPI
  Frontend --> WasmUtils

  Dtensor --> ModelLib

  CoordLib --> CoordinatorProg
  CoreLib --> Authorizer
  CoreLib --> Treasurer

  DataProvider --> ClientRust

  NetworkLib --> ClientRust

  ClientRust --> Prometheus
  CLIClient --> Prometheus
  BackendAPI --> Prometheus
  ClientRust --> Loki
  CLIClient --> Loki
  BackendAPI --> Loki

  Prometheus --> Grafana
  Loki --> Grafana
  OTelCollector --> Prometheus
  OTelCollector --> Loki

  Docker --> ClientRust
  Docker --> BackendAPI
  Docker --> CoordinatorProg
  Nix --> ClientRust
  Nix --> BackendAPI
  CI --> Docker
  CI --> Nix
  CI --> Scripts

  %% Click Events
  click ClientRust "https://github.com/psychefoundation/psyche/blob/main/architectures/centralized/client/src/main.rs"
  click ServerRust "https://github.com/psychefoundation/psyche/blob/main/architectures/centralized/server/src/main.rs"
  click SharedProtocol "https://github.com/psychefoundation/psyche/blob/main/architectures/centralized/shared/src/protocol.rs"
  click LocalTestnet "https://github.com/psychefoundation/psyche/blob/main/architectures/centralized/local-testnet/src/main.rs"
  click TestingHarness "https://github.com/psychefoundation/psyche/blob/main/architectures/centralized/testing/src/lib.rs"
  click Authorizer "https://github.com/psychefoundation/psyche/blob/main/architectures/decentralized/solana-authorizer/programs/solana-authorizer/src/lib.rs"
  click CoordinatorProg "https://github.com/psychefoundation/psyche/blob/main/architectures/decentralized/solana-coordinator/programs/solana-coordinator/src/lib.rs"
  click MiningPool "https://github.com/psychefoundation/psyche/blob/main/architectures/decentralized/solana-mining-pool/programs/solana-mining-pool/src/lib.rs"
  click Treasurer "https://github.com/psychefoundation/psyche/blob/main/architectures/decentralized/solana-treasurer/programs/solana-treasurer/src/lib.rs"
  click CLIClient "https://github.com/psychefoundation/psyche/blob/main/architectures/decentralized/solana-client/src/main.rs"
  click Tooling "https://github.com/psychefoundation/psyche/blob/main/architectures/decentralized/solana-tooling/src/lib.rs"
  click ChaosTests "https://github.com/psychefoundation/psyche/blob/main/architectures/decentralized/testing/src/chaos.rs"
  click CLib "https://github.com/psychefoundation/psyche/blob/main/shared/client/src/lib.rs"
  click CoordLib "https://github.com/psychefoundation/psyche/blob/main/shared/coordinator/src/lib.rs"
  click CoreLib "https://github.com/psychefoundation/psyche/blob/main/shared/core/src/lib.rs"
  click DataProvider "https://github.com/psychefoundation/psyche/blob/main/shared/data-provider/src/data_provider.rs"
  click ModelLib "https://github.com/psychefoundation/psyche/blob/main/shared/modeling/src/lib.rs"
  click NetworkLib "https://github.com/psychefoundation/psyche/blob/main/shared/network/src/lib.rs"
  click MetricsLib "https://github.com/psychefoundation/psyche/blob/main/shared/metrics/src/lib.rs"
  click EvalLib "https://github.com/psychefoundation/psyche/blob/main/shared/eval/src/lib.rs"
  click TUILib "https://github.com/psychefoundation/psyche/blob/main/shared/tui/src/lib.rs"
  click WatcherLib "https://github.com/psychefoundation/psyche/blob/main/shared/watcher/src/lib.rs"
  click PythonSidecar "https://github.com/psychefoundation/psyche/blob/main/python/python/psyche/sidecar/api.py"
  click Dtensor "https://github.com/psychefoundation/psyche/blob/main/python/python/psyche/dtensor_helpers.py"
  click PyExtension "https://github.com/psychefoundation/psyche/blob/main/python/extension-impl/src/extension.rs"
  click Frontend "https://github.com/psychefoundation/psyche/blob/main/website/frontend/src/main.tsx"
  click BackendAPI "https://github.com/psychefoundation/psyche/blob/main/website/backend/src/index.ts"
  click WebTypes "https://github.com/psychefoundation/psyche/blob/main/website/shared/formats/index.ts"
  click WasmUtils "https://github.com/psychefoundation/psyche/blob/main/website/wasm/src/lib.rs"
  click Prometheus "https://github.com/psychefoundation/psyche/blob/main/telemetry/prometheus.yml"
  click Grafana "https://github.com/psychefoundation/psyche/blob/main/telemetry/grafana/dashboards/clients_dashboard.json"
  click Loki "https://github.com/psychefoundation/psyche/blob/main/telemetry/loki-config.yaml"
  click OTelCollector "https://github.com/psychefoundation/psyche/blob/main/telemetry/otel-collector-config.yaml"
  click Docker "https://github.com/psychefoundation/psyche/blob/main/docker/docker-compose.yml"
  click Nix "https://github.com/psychefoundation/psyche/blob/main/flake.nix"
  click CI "https://github.com/psychefoundation/psyche/blob/main/.github/workflows/build.yml"
  click Scripts "https://github.com/psychefoundation/psyche/blob/main/scripts/deploy.sh"

  %% Styles
  classDef onchain fill:#fdd835,stroke:#f57f17,stroke-width:2px
  classDef offchain fill:#42a5f5,stroke:#1e88e5,stroke-width:2px
  classDef shared fill:#b0bec5,stroke:#546e7a,stroke-width:2px
  classDef python fill:#80deea,stroke:#26c6da,stroke-width:2px
  classDef frontend fill:#9ccc65,stroke:#7cb342,stroke-width:2px
  classDef telemetry fill:#ab47bc,stroke:#8e24aa,stroke-width:2px
  classDef infra fill:#ef5350,stroke:#e53935,stroke-width:2px
```
