How rishirelan/data-formulator works
Project Type and Purpose
• This is a full-stack web application named Data Formulator.
• Purpose: Provide a React‐based UI for analysts to iteratively transform data and author rich visualizations by delegating data manipulation tasks to AI agents running in a Python backend.
File-Structure Analysis
• Top-level:
• src/: React+TypeScript front end (Vite) with App.tsx, Redux slice (dfSlice.tsx), components, views, SCSS.
• public/ & embed/: static assets (HTML, media, embed snippets).
• py-src/data_formulator/: Python backend package; app.py, __main__.py (Flask/FastAPI server), agent modules, sandbox.
• Build & config: pyproject.toml, requirements.txt, package.json, vite.config.ts, devcontainer files, GitHub Actions CI.
README Insights
• The Python package exposes a CLI (data_formulator or python -m data_formulator) that starts a local server on port 5000.
• Front end is served either by the Python server in production or by Vite in development.
• Backend calls out to multiple model providers (OpenAI, Azure, Anthropic, LiteLLM, etc.) using API keys from api-keys.env.
• The UI lets users drag-and-drop fields to chart properties, type natural language prompts for non-existent fields, click “Formulate” to trigger data transformation.
Components & Relationships
a. Main Components:
1. Browser/UI
• React App (src/app, src/views, src/components)
• Redux store (dfSlice.tsx)
• Chart rendering (ChartTemplates.tsx)
• Embed loader (embed/dynamic.html + postMessage bridge)
2. Backend API Server
• Flask/FastAPI app (py-src/data_formulator/app.py)
• Data Formulator CLI (__main__.py)
• Agent Orchestrator modules: agent_data_load, agent_data_clean, agent_data_transform, etc.
• Sandbox executor (py_sandbox.py) for running generated code safely
3. External Services
• LLM Providers (OpenAI, Azure OpenAI, Anthropic, Ollama)
• LiteLLM abstraction
4. Static Assets & Dev Tools
• Vite dev server / production static file serving
• GitHub Actions CI/CD
• Devcontainer / Codespaces setup
b. Data & Control Flows:
1. User interacts with React UI →
2. UI dispatches Redux actions →
3. When “Formulate” is clicked, UI issues HTTP POST to /api/formulate on Python server →
4. Flask/FastAPI handler loads requested agent(s) →
5. Agent modules call out to LLM Providers with prompts & existing dataset →
6. LLM returns transformation code or structured results →
7. Backend executes code in sandbox to produce transformed data →
8. Server responds with JSON payload (new dataset + Vega spec + code snippet) →
9. React UI receives response, updates Redux state →
10. ChartTemplates renders updated visualization
c. Architectural Patterns & Principles
• Client-Server Separation: React front end vs. Python backend API
• Agent-Based Orchestration: Modular agents handle specialized tasks (clean, load, transform)
• Sandbox Isolation: Generated code runs in a restricted environment (py_sandbox.py)
• Extensibility: New agents can be plugged in; multiple LLM providers abstracted via client_utils
d. Key Technologies & Libraries
• Front end: React, Redux Toolkit, TypeScript, Vite, SCSS
• Backend: Python 3, Flask or FastAPI, pip package data_formulator
• LLM integration: OpenAI SDK, Anthropic SDK, LiteLLM
• CI/CD & DevOps: GitHub Actions, Devcontainer/Codespaces, Docker