# openmined/pysyft architecture

> Architecture diagram of the GitHub repository openmined/pysyft, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openmined/pysyft
- Repository: https://github.com/openmined/pysyft

## Overview

##### Project type and purpose
- This is a full‐stack, monorepo open‐source platform (PySyft) for “remote data science.”
- It delivers:
- A core Python library (`packages/syft`) providing client & server SDKs.
- A grid system (`packages/grid`) that you deploy as microservices (backend, frontend, enclave, blob‐store, queue, workers).
- CLI tools (`packages/syftcli`, `packages/syft-client`, `packages/syftbox`).
- Deployment tooling (Dockerfiles, Helm charts under `packages/grid/helm`, Traefik, seaweedfs statefulset, rathole for reverse tunneling).
- The README highlights two “actors”: data scientist (client) and datasite server (grid), showing they interact over network APIs and spawn isolated compute workers against private data.

##### High‐level directory/architecture cues
- `packages/syft`: library + FastAPI/Uvicorn server + client API + custom‐worker builders.
- `packages/grid`:
- `backend`: Python API endpoints, business logic, queue producer.
- `frontend`: Svelte UI served by its own container.
- `enclave/attestation`: TEE service to bootstrap secure enclaves.
- `helm`: K8s manifests & Helm charts for deploying all pieces.
- `seaweedfs`: blob‐storage service for data.
- `rathole`: reverse‐tunnel service for secure network ingress into private sites.
- `packages/syft-client` & `packages/syftcli`: Python CLI/SDK for end‐users, container images.
- Scripts & workflows automate CI/CD, deployment, versioning.

1. Client side
– Syft Python client & CLI (`syft` library, `syftcli`, `syftbox`)
– Browser UI (Svelte frontend)
2. Ingress & API gateway
– Traefik (K8s ingress), optionally rathole for reverse‐tunnel
3. Grid backend
– FastAPI/Uvicorn server (routes in `packages/grid/backend/grid/api`)
– Service registry & router
4. Persistence & storage
– PostgreSQL (statefulset)
– SeaweedFS blob store (statefulset)
– Container registry (for custom worker images)
5. Task orchestration
– ZMQ‐based message queue (internal producer/consumer in `syft` & grid)
– Worker pool (Docker or K8s pods) running user code in isolation
6. Enclave service (optional TEE)
– Attestation servers & enclave containers
7. DevOps & deployment
– Helm charts (templates for each microservice)
– Docker images for each component
– CI/CD pipelines (`.github/workflows`)

– Client → Ingress/Gateway → Frontend UI / Backend API
– Backend API ↔ Database
– Backend API ↔ Blob‐store
– Backend API → Enqueue task → ZMQ queue → Worker
– Worker → fetch data from blob‐store, run code, write results back
– Attestation service is called by backend to spin up secure enclaves
– CLI & Python SDK communicate directly with Backend API (HTTP/gRPC)
– Helm/Traefik manage external DNS, TLS, routing into services
– Reverse‐tunnel (rathole) can sit in front of gateway for private‐network datasites

C. Architectural patterns & principles
– Microservices: each functional area (API server, UI, queue, workers, blob‐store, DB, enclave) is its own container/deployment
– Asynchronous event‐driven tasks: client requests spawn jobs via message queue, decoupling request‐submission from computation
– Secure enclaves & attestation for private code execution
– Infrastructure as code: Helm charts parameterize resources, Kubernetes operators deploy statefulsets
– Separation of concerns: library vs. server vs. UI vs. deployment bundles

D. Key technologies & libraries
– Python 3.9+, FastAPI/Uvicorn, Cap’n Proto (for serialization), ZMQ messaging
– PostgreSQL, SeaweedFS
– Docker, Kubernetes (Helm), Traefik ingress controller
– Svelte + Playwright tests for frontend
– Cap’n Proto JS for client serialization
– CI/CD: GitHub Actions, pre‐commit, bumpversion, Justfile

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Tier
    subgraph "Client Tier" 
        direction TB
        PythonSDK["Core Python SDK\n(packages/syft/src/syft)"]:::client
        PythonCLI["Python CLI Tool\n(syftcli)"]:::client
        DesktopCLI["Desktop App CLI\n(syftbox)"]:::client
        BrowserUI["Browser UI\n(Svelte frontend)"]:::client
    end

    %% Ingress & Gateway Tier
    subgraph "Ingress & API Gateway" 
        direction TB
        Traefik["Traefik Ingress\n(packages/grid/traefik)"]:::infra
        Rathole["Reverse-Tunnel Service\n(rathole)"]:::infra
    end

    %% Backend Tier
    subgraph "Backend Tier"
        direction TB
        FastAPI["Grid Backend API Server\n(packages/grid/backend/grid)"]:::backend
    end

    %% Storage Tier
    subgraph "Storage Tier"
        direction TB
        Postgres["PostgreSQL\n(statefulset)"]:::storage
        SeaweedFS["SeaweedFS Blob Store\n(packages/grid/seaweedfs)"]:::storage
    end

    %% Compute Tier
    subgraph "Compute Tier"
        direction TB
        ZMQQueue["ZMQ Message Queue"]:::compute
        WorkerPool["Worker Pool\n(K8s pods)"]:::compute
        Enclave["Enclave Attestation Service\n(packages/grid/enclave/attestation)"]:::compute
    end

    %% Deployment Overlay
    subgraph "Deployment & DevOps"
        direction TB
        HelmCharts["Helm Charts\n(packages/grid/helm/syft)"]:::infra
    end

    %% Connections
    PythonSDK -->|"HTTP request"| Traefik
    PythonCLI -->|"HTTP request"| Traefik
    DesktopCLI -->|"HTTP request"| Traefik
    BrowserUI -->|"HTTP request"| Traefik

    Rathole -->|"SSH tunnel"| Traefik

    Traefik -->|"Route to UI"| BrowserUI
    Traefik -->|"Route to API"| FastAPI

    FastAPI -->|"SQL query"| Postgres
    FastAPI -->|"Blob GET/PUT"| SeaweedFS
    FastAPI -->|"Enqueue task"| ZMQQueue
    FastAPI -->|"Attestation call"| Enclave

    ZMQQueue -->|"ZMQ task"| WorkerPool
    WorkerPool -->|"Blob GET/PUT"| SeaweedFS
    WorkerPool -->|"Write results"| SeaweedFS

    HelmCharts --> Traefik
    HelmCharts --> FastAPI
    HelmCharts --> SeaweedFS
    HelmCharts --> Postgres
    HelmCharts --> ZMQQueue
    HelmCharts --> WorkerPool
    HelmCharts --> Enclave

    %% Click Events
    click PythonSDK "https://github.com/openmined/pysyft/tree/dev/packages/syft/src/syft"
    click FastAPI "https://github.com/openmined/pysyft/tree/dev/packages/grid/backend/grid"
    click Traefik "https://github.com/openmined/pysyft/tree/dev/packages/grid/traefik"
    click Rathole "https://github.com/openmined/pysyft/tree/dev/packages/grid/rathole"
    click SeaweedFS "https://github.com/openmined/pysyft/tree/dev/packages/grid/seaweedfs"
    click Enclave "https://github.com/openmined/pysyft/tree/dev/packages/grid/enclave/attestation"
    click HelmCharts "https://github.com/openmined/pysyft/tree/dev/packages/grid/helm/syft"
    click PythonCLI "https://github.com/openmined/pysyft/tree/dev/packages/syftcli"
    click WorkerPool "https://github.com/openmined/pysyft/tree/dev/packages/grid/syft-client"
    click DesktopCLI "https://github.com/openmined/pysyft/tree/dev/packages/syftbox"

    %% Styles
    classDef client fill:#cce5ff,stroke:#0275d8,color:#01244d;
    classDef backend fill:#d4edda,stroke:#28a745,color:#155724;
    classDef storage fill:#fff3cd,stroke:#ffc107,color:#856404;
    classDef compute fill:#e2e3e5,stroke:#6f42c1,color:#3f2854;
    classDef infra fill:#f8f9fa,stroke:#6c757d,color:#343a40;
```
