# factdroid/rds architecture

> Architecture diagram of the GitHub repository factdroid/rds, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/factdroid/rds
- Repository: https://github.com/factdroid/rds

## Overview

##### Project type and purpose
- This is an open-source Python package and CLI tool (“syft-rds”) that layers a Remote Data Science (RDS) service on top of SyftBox.
- It provides:
- A server component (app, routers, RPC endpoints) that runs on a data-owner’s datasite
- A client library/CLI that lives in a data-scientist’s environment (local_store, rds_client)
- Notebook-based demos (quickstart, internal, API) and a BigQuery example
- Use cases: allow data scientists to execute compute on remote data (owner’s side) without exposing raw data.

##### File structure
- Top-level folders:
- bigquery/            → example service to publish/query Google BigQuery
- notebooks/           → Jupyter demos: quickstart, API, internal
- runtimes/            → Dockerfile and entrypoint for containerized Python runtime
- syft-rds/            → main package: CLI, client, server, orchestration, store, runtime, utils
- tests/               → unit, integration, end-to-end tests
- CI configs, justfile → build, install, run scripts
- In syft-rds/src/syft_rds:
- cli.py               → entrypoint for CLI
- client/              → client APIs, connection, local_store abstractions, RPC
- server/              → FastAPI-style app, routers for job/runtime/user_code
- store/               → low-level store interface
- syft_runtime/        → code that launches the remote execution runtime
- orchestra.py        → glue between client, server, store
- External dependencies: uv framework, just, uv.lock (dependency lockfile).

##### README for additional insights
- “full example with SyftBox” where you launch:
1) SyftBox server
2) SyftBox clients (data owner + data scientist)
3) RDS app on data-owner datasite
4) (opt.) RDS app on data-scientist datasite
5) Run notebooks

##### Important details
- Orchestration layer (orchestra.py) as the glue
- Test harness (tests/) as a separate lane or note to illustrate coverage
- BigQuery example as an optional external service

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Swimlanes
    subgraph "Data Scientist Environment"
        direction TB
        DS_CLI["RDS CLI"]:::client
        DS_Client["RDS Client Library"]:::client
        Local_Store["Local Store"]:::client
        Notebook["Jupyter Notebooks"]:::aux
    end

    subgraph "SyftBox Infrastructure"
        direction TB
        Broker["SyftBox Orchestrator"]:::infra
    end

    subgraph "Data Owner Datasite"
        direction TB
        Server["RDS Server"]:::server
        Runtime["Execution Runtime"]:::server
        Store["Store Interface"]:::server
        BigQuery["BigQuery Service"]:::infra
    end

    %% Data Scientist flow
    Notebook -->|"uses"| DS_CLI
    DS_CLI -->|"invokes"| DS_Client
    DS_Client -->|"uses"| Local_Store

    %% RPC over broker
    DS_Client -->|"RPC"| Broker
    Broker -->|"RPC"| Server
    Server -->|"responds over RPC"| Broker
    Broker -->|"responds over RPC"| DS_Client

    %% Server internals
    Server -->|"executes code via"| Runtime
    Runtime -->|"persists data to"| Store

    %% External service
    Server -->|"queries"| BigQuery
    BigQuery -->|"returns data"| Server

    %% Click Events
    click Broker "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/orchestra.py"
    click DS_CLI "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/cli.py"
    click DS_Client "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/client/rds_client.py"
    click DS_Client "https://github.com/factdroid/rds/tree/main/syft-rds/src/syft_rds/client/rds_clients/"
    click Local_Store "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/client/local_store.py"
    click Local_Store "https://github.com/factdroid/rds/tree/main/syft-rds/src/syft_rds/client/local_stores/"
    click DS_Client "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/client/connection.py"
    click DS_Client "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/client/rpc.py"
    click DS_Client "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/client/client_registry.py"
    click Server "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/server/app.py"
    click Server "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/server/router.py"
    click Server "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/server/routers/job_router.py"
    click Server "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/server/routers/runtime_router.py"
    click Server "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/server/routers/user_code_router.py"
    click Server "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/server/user_file_service.py"
    click Runtime "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/syft_runtime/main.py"
    click Store "https://github.com/factdroid/rds/blob/main/syft-rds/src/syft_rds/store/store.py"
    click Notebook "https://github.com/factdroid/rds/tree/main/syft-rds/src/syft_rds/jupyter_utils/"
    classDef infra fill:#AED6F1,stroke:#21618C,color:#1B2631
    classDef server fill:#ABEBC6,stroke:#1D8348,color:#1B2631
    classDef client fill:#F9E79F,stroke:#B7950B,color:#1B2631
    classDef aux fill:#D5D8DC,stroke:#566573,color:#1B2631
```
