# rishirelan/data-formulator architecture

> Architecture diagram of the GitHub repository rishirelan/data-formulator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rishirelan/data-formulator
- Repository: https://github.com/rishirelan/data-formulator

## Overview

##### Project Type and Purpose
- This is a full-stack web application named **Data Formulator**.
- Purpose: Provide a React‐based UI for analysts to iteratively transform data and author rich visualizations by delegating data manipulation tasks to AI agents running in a Python backend.

##### File-Structure Analysis
- Top-level:
- **src/**: React+TypeScript front end (Vite) with App.tsx, Redux slice (dfSlice.tsx), components, views, SCSS.
- **public/** & **embed/**: static assets (HTML, media, embed snippets).
- **py-src/data_formulator/**: Python backend package; app.py, __main__.py (Flask/FastAPI server), agent modules, sandbox.
- Build & config: pyproject.toml, requirements.txt, package.json, vite.config.ts, devcontainer files, GitHub Actions CI.

##### README Insights
- The Python package exposes a CLI (`data_formulator` or `python -m data_formulator`) that starts a local server on port 5000.
- Front end is served either by the Python server in production or by Vite in development.
- Backend calls out to multiple model providers (OpenAI, Azure, Anthropic, LiteLLM, etc.) using API keys from `api-keys.env`.
- The UI lets users drag-and-drop fields to chart properties, type natural language prompts for non-existent fields, click “Formulate” to trigger data transformation.

##### Components & Relationships
a. Main Components:
1. **Browser/UI**
- React App (src/app, src/views, src/components)
- Redux store (dfSlice.tsx)
- Chart rendering (ChartTemplates.tsx)
- Embed loader (embed/dynamic.html + postMessage bridge)
2. **Backend API Server**
- Flask/FastAPI app (py-src/data_formulator/app.py)
- Data Formulator CLI (`__main__.py`)
- Agent Orchestrator modules: agent_data_load, agent_data_clean, agent_data_transform, etc.
- Sandbox executor (py_sandbox.py) for running generated code safely
3. **External Services**
- LLM Providers (OpenAI, Azure OpenAI, Anthropic, Ollama)
- LiteLLM abstraction
4. **Static Assets & Dev Tools**
- Vite dev server / production static file serving
- GitHub Actions CI/CD
- Devcontainer / Codespaces setup

b. Data & Control Flows:
1. User interacts with React UI →
2. UI dispatches Redux actions →
3. When “Formulate” is clicked, UI issues HTTP POST to `/api/formulate` on Python server →
4. Flask/FastAPI handler loads requested agent(s) →
5. Agent modules call out to LLM Providers with prompts & existing dataset →
6. LLM returns transformation code or structured results →
7. Backend executes code in sandbox to produce transformed data →
8. Server responds with JSON payload (new dataset + Vega spec + code snippet) →
9. React UI receives response, updates Redux state →
10. ChartTemplates renders updated visualization

c. Architectural Patterns & Principles
- **Client-Server Separation**: React front end vs. Python backend API
- **Agent-Based Orchestration**: Modular agents handle specialized tasks (clean, load, transform)
- **Sandbox Isolation**: Generated code runs in a restricted environment (py_sandbox.py)
- **Extensibility**: New agents can be plugged in; multiple LLM providers abstracted via client_utils

d. Key Technologies & Libraries
- Front end: React, Redux Toolkit, TypeScript, Vite, SCSS
- Backend: Python 3, Flask or FastAPI, pip package `data_formulator`
- LLM integration: OpenAI SDK, Anthropic SDK, LiteLLM
- CI/CD & DevOps: GitHub Actions, Devcontainer/Codespaces, Docker

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Browser / Frontend"
        direction TB
        U["User"]:::external
        RA["React App"]:::frontend
        RS["Redux Slice (dfSlice.tsx)"]:::frontend
        SC["Store Config (store.ts)"]:::frontend
        UC["UI Components"]:::frontend
        VC["View Components"]:::frontend
        CT["Chart Templates"]:::frontend
        EL["Embed Loader"]:::frontend
        VC2["Vite Config"]:::frontend
        SA["Static Assets (public/)"]:::frontend
    end

    %% Backend Layer
    subgraph "Python Backend"
        direction TB
        API["Flask/FastAPI API Server"]:::backend
        CLI["Data Formulator CLI"]:::backend
        subgraph "Agent Orchestrator"
            direction TB
            ADL["Data Load Agent"]:::backend
            ADC["Data Clean Agent"]:::backend
            ADTv2["Data Transform Agent v2"]:::backend
            ADT["Data Transform Agent"]:::backend
            ADF["Data Filter Agent"]:::backend
            ASD["Sort Data Agent"]:::backend
            ACD["Concept Derivation Agent"]:::backend
            APCD["Py Concept Derive Agent"]:::backend
            AGPC["Generic Concept Agent"]:::backend
            ACE["Code Explanation Agent"]:::backend
            ADR["Recommendation Agent"]:::backend
            AU["Agent Utils"]:::backend
            CU["Client Utils"]:::backend
        end
        SB["Sandbox Executor"]:::backend
    end

    %% External and Tools
    subgraph "External Services & Tools"
        direction TB
        LLM["LLM Providers (OpenAI, Azure, Anthropic, Ollama)"]:::external
        LLM2["LiteLLM Abstraction"]:::external
        CI["GitHub Actions CI/CD"]:::tools
        DC["Devcontainer Setup"]:::tools
    end

    %% Data Flows
    U -->|"interacts with"| RA
    RA -->|"dispatches actions"| RS
    RS --> SC
    RA --> UC
    RA --> VC
    RA --> CT
    RA --> EL
    RA --> VC2
    RA --> SA

    RA -->|"POST /api/formulate"| API
    API -->|"invokes agents"| ADL
    API --> ADC
    API --> ADTv2
    API --> ADT
    API --> ADF
    API --> ASD
    API --> ACD
    API --> APCD
    API --> AGPC
    API --> ACE
    API --> ADR
    API --> AU
    API --> CU

    ADL -->|"calls LLM"| LLM
    ADC --> LLM
    ADTv2 --> LLM
    ADT --> LLM
    ADF --> LLM
    ASD --> LLM
    ACD --> LLM
    APCD --> LLM
    AGPC --> LLM
    ACE --> LLM2
    ADR --> LLM

    ADL --> SB
    ADC --> SB
    ADTv2 --> SB
    ADT --> SB
    ADF --> SB
    ASD --> SB
    ACD --> SB
    APCD --> SB
    AGPC --> SB
    ACE --> SB
    ADR --> SB

    SB --> API
    API -->|"returns JSON payload"| RA
    RA -->|"updates state and renders"| CT

    API --> CLI
    CLI --> API

    CI --> API
    DC --> API
    CI --> RA
    DC --> RA

    %% Click Events
    click RA "https://github.com/rishirelan/data-formulator/blob/main/src/app/App.tsx"
    click RS "https://github.com/rishirelan/data-formulator/blob/main/src/app/dfSlice.tsx"
    click SC "https://github.com/rishirelan/data-formulator/blob/main/src/app/store.ts"
    click EL "https://github.com/rishirelan/data-formulator/blob/main/embed/dynamic.html"
    click EL "https://github.com/rishirelan/data-formulator/blob/main/src/app/embed.ts"
    click CT "https://github.com/rishirelan/data-formulator/blob/main/src/components/ChartTemplates.tsx"
    click UC "https://github.com/rishirelan/data-formulator/tree/main/src/components/"
    click VC "https://github.com/rishirelan/data-formulator/tree/main/src/views/"
    click SA "https://github.com/rishirelan/data-formulator/tree/main/public/"
    click VC2 "https://github.com/rishirelan/data-formulator/blob/main/vite.config.ts"
    click API "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/app.py"
    click CLI "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/__main__.py"
    click ADL "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_data_load.py"
    click ADC "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_data_clean.py"
    click ADTv2 "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_data_transform_v2.py"
    click ADT "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_data_transformation.py"
    click ADF "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_data_filter.py"
    click ASD "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_sort_data.py"
    click ACD "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_concept_derive.py"
    click APCD "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_py_concept_derive.py"
    click AGPC "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_generic_py_concept.py"
    click ACE "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_code_explanation.py"
    click ADR "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_data_rec.py"
    click AU "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/agent_utils.py"
    click CU "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/agents/client_utils.py"
    click SB "https://github.com/rishirelan/data-formulator/blob/main/py-src/data_formulator/py_sandbox.py"
    click CI "https://github.com/rishirelan/data-formulator/blob/main/.github/workflows/python-build.yml"
    click DC "https://github.com/rishirelan/data-formulator/blob/main/.devcontainer/devcontainer.json"

    %% Styles
    classDef frontend fill:#D6EAF8,stroke:#1B4F72,stroke-width:1px
    classDef backend fill:#D5F5E3,stroke:#145A32,stroke-width:1px
    classDef external fill:#FCF3CF,stroke:#7D6608,stroke-width:1px,stroke-dasharray: 5 5
    classDef tools fill:#FADBD8,stroke:#922B21,stroke-width:1px,stroke-dasharray: 2 2
```
