How openmined/pysyft works
Project type and purpose
• This is a full‐stack, monorepo open‐source platform (PySyft) for “remote data science.”
• It delivers:
• A core Python library (packages/syft) providing client & server SDKs.
• A grid system (packages/grid) that you deploy as microservices (backend, frontend, enclave, blob‐store, queue, workers).
• CLI tools (packages/syftcli, packages/syft-client, packages/syftbox).
• Deployment tooling (Dockerfiles, Helm charts under packages/grid/helm, Traefik, seaweedfs statefulset, rathole for reverse tunneling).
• The README highlights two “actors”: data scientist (client) and datasite server (grid), showing they interact over network APIs and spawn isolated compute workers against private data.
High‐level directory/architecture cues
• packages/syft: library + FastAPI/Uvicorn server + client API + custom‐worker builders.
• packages/grid:
• backend: Python API endpoints, business logic, queue producer.
• frontend: Svelte UI served by its own container.
• enclave/attestation: TEE service to bootstrap secure enclaves.
• helm: K8s manifests & Helm charts for deploying all pieces.
• seaweedfs: blob‐storage service for data.
• rathole: reverse‐tunnel service for secure network ingress into private sites.
• packages/syft-client & packages/syftcli: Python CLI/SDK for end‐users, container images.
• Scripts & workflows automate CI/CD, deployment, versioning.
1. Client side
– Syft Python client & CLI (syft library, syftcli, syftbox)
– Browser UI (Svelte frontend)
2. Ingress & API gateway
– Traefik (K8s ingress), optionally rathole for reverse‐tunnel
3. Grid backend
– FastAPI/Uvicorn server (routes in packages/grid/backend/grid/api)
– Service registry & router
4. Persistence & storage
– PostgreSQL (statefulset)
– SeaweedFS blob store (statefulset)
– Container registry (for custom worker images)
5. Task orchestration
– ZMQ‐based message queue (internal producer/consumer in syft & grid)
– Worker pool (Docker or K8s pods) running user code in isolation
6. Enclave service (optional TEE)
– Attestation servers & enclave containers
7. DevOps & deployment
– Helm charts (templates for each microservice)
– Docker images for each component
– CI/CD pipelines (.github/workflows)
– Client → Ingress/Gateway → Frontend UI / Backend API
– Backend API ↔ Database
– Backend API ↔ Blob‐store
– Backend API → Enqueue task → ZMQ queue → Worker
– Worker → fetch data from blob‐store, run code, write results back
– Attestation service is called by backend to spin up secure enclaves
– CLI & Python SDK communicate directly with Backend API (HTTP/gRPC)
– Helm/Traefik manage external DNS, TLS, routing into services
– Reverse‐tunnel (rathole) can sit in front of gateway for private‐network datasites
C. Architectural patterns & principles
– Microservices: each functional area (API server, UI, queue, workers, blob‐store, DB, enclave) is its own container/deployment
– Asynchronous event‐driven tasks: client requests spawn jobs via message queue, decoupling request‐submission from computation
– Secure enclaves & attestation for private code execution
– Infrastructure as code: Helm charts parameterize resources, Kubernetes operators deploy statefulsets
– Separation of concerns: library vs. server vs. UI vs. deployment bundles
D. Key technologies & libraries
– Python 3.9+, FastAPI/Uvicorn, Cap’n Proto (for serialization), ZMQ messaging
– PostgreSQL, SeaweedFS
– Docker, Kubernetes (Helm), Traefik ingress controller
– Svelte + Playwright tests for frontend
– Cap’n Proto JS for client serialization
– CI/CD: GitHub Actions, pre‐commit, bumpversion, Justfile