# apple/embedding-atlas architecture

> Architecture diagram of the GitHub repository apple/embedding-atlas, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/embedding-atlas
- Repository: https://github.com/apple/embedding-atlas

## Overview

##### Project type and purpose
- This is a polyglot monorepo/workspace that provides:
– A Python CLI & Jupyter widget (`packages/backend`)
– A suite of npm/TypeScript front-end libraries & apps (`packages/component`, `packages/table`, `packages/viewer`, `packages/embedding-atlas`)
– High-performance algorithms compiled to WebAssembly (`packages/density-clustering`, `packages/umap-wasm`)
– Documentation site (`packages/docs`)

##### File structure to infer architectural zones
- Top-level: CI config, linting, root package.json (workspaces)
- packages/backend: Python package with
– CLI (`cli.py`)
– Server component for Jupyter/Streamlit integration (`server.py`, `widget.py`, `streamlit.py`)
– Bundled front-end assets built by Vite (`src/anywidget`, `src/streamlit`)
- packages/component & packages/table: Standalone Svelte/TS libraries for EmbeddingView & Table components
- packages/viewer: A standalone web app (Svelte + Vite + DuckDB-WASM) that consumes the component/table libraries
- packages/embedding-atlas: A JS/TS umbrella package that re-exports component, table, UMAP-WASM & density-clustering-WASM APIs for React/Svelte
- packages/density-clustering: Rust library + CLI + WASM build for density clustering
- packages/umap-wasm: C++ UMAP implementation compiled to WASM + JS glue
- packages/docs: VitePress documentation site with demo and how-to guides
- scripts/ & build.sh: Cross-workspace build orchestration

##### README for extra context
- Confirms multi-package structure: Python vs. npm vs. Rust vs. C++
- Points out key use cases: CLI, Jupyter, web embeddable component, demonstration examples
- Mentions demo site and published packages

##### Main components & interactions
a. Python Backend Zone
– CLI tool (invokes Python library)
– Jupyter widget & Streamlit integration
– Embedded HTTP server / anywidget bundler serving front-end widgets
b. Frontend Library Zone (npm)
– EmbeddingView & EmbeddingViewMosaic (`packages/component`)
– Table component (`packages/table`)
– EmbeddingAtlas umbrella package (`packages/embedding-atlas`)
c. Web App Zone
– Viewer app (`packages/viewer`): Svelte front end + DuckDB-WASM for on-the-fly querying
– Example apps (`packages/examples`) demonstrating React & Svelte usage
d. Algorithm Engine Zone
– Density clustering (Rust → WASM)
– UMAP (C++ → WASM)
e. Documentation Zone
– VitePress site (`packages/docs`)
f. Build & CI Zone
– Yarn/npm workspaces, Vite builds, Cargo & Make for WASM, GitHub Actions CI

##### Relationships & data flows
- Python backend calls out to the local Vite-built widget bundle and serves it to Jupyter/Streamlit
- Frontend libraries import the WASM modules for clustering (Rust-WASM) and UMAP (C++-WASM)
- Viewer app consumes component + table libraries, plus DuckDB-WASM for data ingestion
- Examples & docs consume the published npm packages and embed the components
- CI pipeline triggers build across Python, TS, Rust, C++ workspaces and publishes to PyPI and npm

##### Architectural patterns & principles
- Monorepo with polyglot workspaces
- Library-centric design: small, single-responsibility packages (component, table, viewer, density-clustering, umap-wasm)
- Language-specific build pipelines (Cargo for Rust, Make/emscripten for C++, Vite for TS, standard Python packaging)
- WebAssembly “engine” layer that the JS frontends call into (density-clustering WASM, umap-wasm)
- Clear separation of concerns between data-processing (CLI/Jupyter/Python), rendering (WebGPU/WebGL2 via TS/Svelte), and storage/querying (DuckDB-WASM)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Python Backend Zone
    subgraph "Python Backend Zone"
        direction TB
        CLI["Python CLI Tool"]:::pyback
        WidgetBack["Jupyter Widget Bundler (Backend)"]:::pyback
        AnyWidgetFE["Jupyter Widget Bundler (Frontend)"]:::pyback
        StreamlitBack["Streamlit Integration (Backend)"]:::pyback
        StreamlitFE["Streamlit Integration (Frontend)"]:::pyback
        Server["Python HTTP Server"]:::pyback
    end

    %% Frontend Libraries Zone
    subgraph "Frontend Libraries Zone"
        direction TB
        EmbeddingView["EmbeddingView Library"]:::frontend
        TableLib["Table Library"]:::frontend
        AtlasUmbrella["EmbeddingAtlas Umbrella Package"]:::frontend
    end

    %% Web App & Examples Zone
    subgraph "Web App & Examples Zone"
        direction TB
        ViewerApp["Viewer Web Application"]:::webapp
        Examples["Example Applications"]:::webapp
    end

    %% Algorithm Engine Zone
    subgraph "Algorithm Engines Zone"
        direction TB
        DensityWASM["Density-Clustering WASM"]:::engine
        UMAPWASM["UMAP WASM"]:::engine
    end

    %% Documentation Zone
    subgraph "Documentation Zone"
        direction TB
        DocsSite["Documentation Site"]:::docs
    end

    %% CI / Build Zone
    subgraph "CI / Build Zone"
        direction TB
        CI["GitHub Actions CI"]:::ci
        RootBuild["Root Build Script"]:::ci
        BackendBuild["Backend Build Helper"]:::ci
    end

    %% Relationships
    CI -->|triggers build| BackendBuild
    CI -->|triggers build| RootBuild
    CI -->|triggers build| DensityWASM
    CI -->|triggers build| UMAPWASM
    CI -->|triggers build| AtlasUmbrella
    RootBuild -->|npm workspaces| EmbeddingView
    RootBuild -->|npm workspaces| TableLib
    RootBuild -->|npm workspaces| AtlasUmbrella
    BackendBuild -->|pip build| CLI
    BackendBuild -->|pip build| Server

    Server -->|serves HTTP widget| AnyWidgetFE
    WidgetBack -->|bundles| AnyWidgetFE
    StreamlitBack -->|serves| StreamlitFE

    EmbeddingView -->|JS import| AtlasUmbrella
    TableLib -->|JS import| AtlasUmbrella
    DensityWASM -->|WASM module| AtlasUmbrella
    UMAPWASM -->|WASM module| AtlasUmbrella

    AtlasUmbrella -->|JS import| ViewerApp
    EmbeddingView -->|component| ViewerApp
    TableLib -->|component| ViewerApp

    ViewerApp -->|DuckDB-WASM| ViewerApp

    Examples -->|npm install| AtlasUmbrella

    DocsSite -->|uses| Examples
    DocsSite -->|uses| ViewerApp
    DocsSite -->|uses| AtlasUmbrella

    %% Click Events
    click CLI "https://github.com/apple/embedding-atlas/blob/main/packages/backend/embedding_atlas/cli.py"
    click WidgetBack "https://github.com/apple/embedding-atlas/blob/main/packages/backend/embedding_atlas/widget.py"
    click AnyWidgetFE "https://github.com/apple/embedding-atlas/blob/main/packages/backend/src/anywidget/index.ts"
    click StreamlitBack "https://github.com/apple/embedding-atlas/blob/main/packages/backend/embedding_atlas/streamlit.py"
    click StreamlitFE "https://github.com/apple/embedding-atlas/blob/main/packages/backend/src/streamlit/streamlit.ts"
    click Server "https://github.com/apple/embedding-atlas/blob/main/packages/backend/embedding_atlas/server.py"
    click EmbeddingView "https://github.com/apple/embedding-atlas/blob/main/packages/component/src/lib/embedding_view/EmbeddingView.svelte"
    click TableLib "https://github.com/apple/embedding-atlas/blob/main/packages/table/src/lib/Table.svelte"
    click AtlasUmbrella "https://github.com/apple/embedding-atlas/blob/main/packages/embedding-atlas/src/index.ts"
    click ViewerApp "https://github.com/apple/embedding-atlas/blob/main/packages/viewer/src/Entrypoint.svelte"
    click Examples "https://github.com/apple/embedding-atlas/blob/main/packages/examples/src/App.svelte"
    click DensityWASM "https://github.com/apple/embedding-atlas/blob/main/packages/density-clustering/density_clustering/src/lib.rs"
    click UMAPWASM "https://github.com/apple/embedding-atlas/blob/main/packages/umap-wasm/src/index.js"
    click DocsSite "https://github.com/apple/embedding-atlas/blob/main/packages/docs/.vitepress/config.mts"
    click CI "https://github.com/apple/embedding-atlas/blob/main/.github/workflows/ci.yml"
    click RootBuild "https://github.com/apple/embedding-atlas/blob/main/scripts/build.sh"
    click BackendBuild "https://github.com/apple/embedding-atlas/blob/main/packages/backend/build.sh"

    %% Styles
    classDef pyback fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef frontend fill:#d4edda,stroke:#333,stroke-width:1px
    classDef webapp fill:#fff3cd,stroke:#333,stroke-width:1px
    classDef engine fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef docs fill:#e2e3e5,stroke:#333,stroke-width:1px
    classDef ci fill:#d6d1f2,stroke:#333,stroke-width:1px
```
