How apple/embedding-atlas works
Project type and purpose
• This is a polyglot monorepo/workspace that provides:
– A Python CLI & Jupyter widget (packages/backend)
– A suite of npm/TypeScript front-end libraries & apps (packages/component, packages/table, packages/viewer, packages/embedding-atlas)
– High-performance algorithms compiled to WebAssembly (packages/density-clustering, packages/umap-wasm)
– Documentation site (packages/docs)
File structure to infer architectural zones
• Top-level: CI config, linting, root package.json (workspaces)
• packages/backend: Python package with
– CLI (cli.py)
– Server component for Jupyter/Streamlit integration (server.py, widget.py, streamlit.py)
– Bundled front-end assets built by Vite (src/anywidget, src/streamlit)
• packages/component & packages/table: Standalone Svelte/TS libraries for EmbeddingView & Table components
• packages/viewer: A standalone web app (Svelte + Vite + DuckDB-WASM) that consumes the component/table libraries
• packages/embedding-atlas: A JS/TS umbrella package that re-exports component, table, UMAP-WASM & density-clustering-WASM APIs for React/Svelte
• packages/density-clustering: Rust library + CLI + WASM build for density clustering
• packages/umap-wasm: C++ UMAP implementation compiled to WASM + JS glue
• packages/docs: VitePress documentation site with demo and how-to guides
• scripts/ & build.sh: Cross-workspace build orchestration
README for extra context
• Confirms multi-package structure: Python vs. npm vs. Rust vs. C++
• Points out key use cases: CLI, Jupyter, web embeddable component, demonstration examples
• Mentions demo site and published packages
Main components & interactions
a. Python Backend Zone
– CLI tool (invokes Python library)
– Jupyter widget & Streamlit integration
– Embedded HTTP server / anywidget bundler serving front-end widgets
b. Frontend Library Zone (npm)
– EmbeddingView & EmbeddingViewMosaic (packages/component)
– Table component (packages/table)
– EmbeddingAtlas umbrella package (packages/embedding-atlas)
c. Web App Zone
– Viewer app (packages/viewer): Svelte front end + DuckDB-WASM for on-the-fly querying
– Example apps (packages/examples) demonstrating React & Svelte usage
d. Algorithm Engine Zone
– Density clustering (Rust → WASM)
– UMAP (C++ → WASM)
e. Documentation Zone
– VitePress site (packages/docs)
f. Build & CI Zone
– Yarn/npm workspaces, Vite builds, Cargo & Make for WASM, GitHub Actions CI
Relationships & data flows
• Python backend calls out to the local Vite-built widget bundle and serves it to Jupyter/Streamlit
• Frontend libraries import the WASM modules for clustering (Rust-WASM) and UMAP (C++-WASM)
• Viewer app consumes component + table libraries, plus DuckDB-WASM for data ingestion
• Examples & docs consume the published npm packages and embed the components
• CI pipeline triggers build across Python, TS, Rust, C++ workspaces and publishes to PyPI and npm
Architectural patterns & principles
• Monorepo with polyglot workspaces
• Library-centric design: small, single-responsibility packages (component, table, viewer, density-clustering, umap-wasm)
• Language-specific build pipelines (Cargo for Rust, Make/emscripten for C++, Vite for TS, standard Python packaging)
• WebAssembly “engine” layer that the JS frontends call into (density-clustering WASM, umap-wasm)
• Clear separation of concerns between data-processing (CLI/Jupyter/Python), rendering (WebGPU/WebGL2 via TS/Svelte), and storage/querying (DuckDB-WASM)