How michelle123lam/lloom works
Project Type and Purpose
• LLooM is a hybrid full-stack research prototype comprising:
• A Python package (“text_lloom”) implementing the core concept-induction algorithm that orchestrates calls to OpenAI GPT-4o/mini and processes unstructured text.
• A static web application (“LLooM Workbench”) built with Vite, Svelte components, and VitePress for documentation and interactive visualization.
• Purpose: enable analysts to run concept induction on text datasets (via the Python library) and explore the results in an interactive browser UI.
Top-Level File Structure and Architectural Indicators
• “text_lloom/” directory: the algorithmic backend (Python library modules for concept induction, LLM prompts, openai integration, workbench API).
• “src/” directory: the front-end application code in Svelte (components for tables, charts, matrix view, overview histogram).
• “docs/.vitepress/”: documentation site configuration and theming.
• “docs/public/data/”: precomputed JSON/CSV datasets for demos.
• “.github/workflows/deploy.yml”: CI/CD pipeline deploying the static site to GitHub Pages.
• Config files: “requirements.txt” (Python deps), “package.json” (JS deps), “vite.config.js” / “vite.config-select.js” (build targets for different UI modes).
Key Components
A. Algorithm Backend (Python “text_lloom” package)
– Modules: concept induction, concept application, LLM wrapper, prompt definitions, workbench API.
– External Service: OpenAI API (GPT-4o/GPT-4o-mini).
– Data Inputs: raw unstructured text (CSV/JSON), user-authored custom concept definitions.
– Data Outputs: structured concept definitions + match results (JSON).
B. Workbench Front-end (Static Web App)
– Build System: Vite + Svelte, served as static assets.
– Components: ConceptOverview, MatrixView, DetailView, Tables, Charts.
– Data Fetch: loads JSON/CSV from “docs/public/data/” or user-provided API endpoint.
– User Interactions: filter by slices, annotate custom concepts, inspect examples.
C. Documentation & Demo Site
– VitePress site assembling markdown pages, custom Vue components for demos.
– Static assets: demo datasets, media, notebooks.
– CI/CD: GitHub Actions builds & deploys to GitHub Pages.
D. External Integrations
– GitHub Pages: hosts static site and front-end.
– PyPI Registry: hosts the “text_lloom” package for pip install.
– Colab Notebooks: for end-user walkthroughs (links in docs).
Relationships and Data Flows
1. User supplies dataset → calls Python API (locally or in Colab).
2. Python package samples text spans → sends prompts to OpenAI → receives concept candidates.
3. Python refines, scores, and applies concepts back to the dataset → exports JSON results.
4. Front-end loads JSON results via HTTP fetch (static or custom endpoint).
5. Svelte components render visualizations (overview, matrix, detail).
6. User may author new concepts in UI → optional round-trip back to Python API for re-induction (if enabled).
7. CI/CD pipeline triggers on Git push → builds docs & front-end → deploys to GitHub Pages.
Architectural Patterns and Principles
• Separation of concerns: algorithmic layer decoupled from visualization layer.
• Client-server decoupling: front-end is a static asset consumer, backend is a library or microservice.
• Static site generation for docs and demos; dynamic algorithm hosted in user environment.
• Data-driven visualizations: JSON schemas as the contract between backend and front-end.