# microsoft/data-formulator architecture

> Architecture diagram of the GitHub repository microsoft/data-formulator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/data-formulator
- Repository: https://github.com/microsoft/data-formulator

## Overview

#### Purpose and Entry Points
Data Formulator is a local-first AI-assisted data exploration system: users load/connect data, conduct branching analyses in a Data Thread, generate/refine charts, and compose reports. Python serves the API and packaged frontend; React/Vite provides the browser UI. Main runtime entry points are `py-src/data_formulator/__main__.py`, `py-src/data_formulator/app.py`, and `src/index.tsx`. Desktop packaging adds `py-src/data_formulator/desktop.py` and `packaging/data_formulator_desktop.py`.

#### Client Workspace and Interaction Model
The React application owns interactive workspace state, persisted sessions, API calls, auth token handling, and chart/table caches. `src/app/App.tsx` establishes the application shell; `src/app/store.ts` and `src/app/dfSlice.tsx` manage client data-thread/workspace state; `src/app/workspaceService.ts` bridges session persistence. The primary analyst experience is rendered through `src/views/DataFormulator.tsx`, `src/views/DataThread.tsx`, and `src/views/DataLoadingChat.tsx`.

#### HTTP API, Sessions, and Contracts
The Python application exposes routes separating agent execution, tables/loading, session lifecycle, credentials, models, knowledge, and logs. Route modules are the boundary between frontend request contracts and backend services: `py-src/data_formulator/routes/agents.py`, `py-src/data_formulator/routes/tables.py`, and `py-src/data_formulator/routes/sessions.py`. Structured error handling is centralized in `py-src/data_formulator/error_handler.py` and `py-src/data_formulator/errors.py`, rather than leaking connector or agent failures directly to clients.

#### Data Ingestion and Connector Boundary
A common connector/data-loader layer abstracts local files, folders, samples, databases, warehouses, cloud storage, and Superset. Core integration points are `py-src/data_formulator/data_connector.py`, `py-src/data_formulator/data_loader/external_data_loader.py`, and `py-src/data_formulator/data_loader/probe_utils.py`. Concrete loaders include PostgreSQL, BigQuery, Databricks, Kusto, S3, Azure Blob, MongoDB, and Superset. The UI discovers/configures these sources through `src/views/DataSourceSidebar.tsx` and `src/components/ConnectorFormCard.tsx`.

#### Workspace, Catalog, and Data Execution
Loaded source data is organized in managed workspaces with metadata/catalog caching and Parquet-oriented utilities. `py-src/data_formulator/datalake/workspace.py`, `py-src/data_formulator/datalake/workspace_manager.py`, and `py-src/data_formulator/datalake/catalog_refresh.py` define this storage boundary; it can use ephemeral local workspaces or Azure Blob-backed workspaces. Declarative data operations are modeled, persisted, and executed by `py-src/data_formulator/data_operations/models.py`, `py-src/data_formulator/data_operations/repository.py`, and `py-src/data_formulator/data_operations/executor.py`.

#### Agent and Analysis Pipeline
Agent requests receive workspace/context and use specialized capabilities for data loading, diagnostics, chart restyling, sorting, code explanation, and workflow distillation. Primary orchestration artifacts are `py-src/data_formulator/analyst/agent.py`, `py-src/data_formulator/agents/context.py`, and `py-src/data_formulator/agents/reasoning_log.py`; skill packages expose tool-oriented behavior in `py-src/data_formulator/analyst/skills/`. Model selection/configuration is separated in `py-src/data_formulator/model_registry.py` and `py-src/data_formulator/agent_config.py`. Supported external model providers include OpenAI, Azure, Ollama, and Anthropic through LiteLLM.

#### Visualization, Reports, and Safety Boundaries
Chart semantics and Flint/Vega-Lite plot generation are backend workflow concerns in `py-src/data_formulator/workflows/chart_semantics.py` and `py-src/data_formulator/workflows/create_vl_plots.py`; interactive rendering and editing occur in `src/views/VisualizationView.tsx` and `src/views/ChartRenderService.tsx`. Reports are assembled in `src/views/ReportView.tsx`. Generated/executed work is constrained by sandbox implementations in `py-src/data_formulator/sandbox/` and security controls for paths, URL allowlisting, sanitization, code signing, and log redaction in `py-src/data_formulator/security/`.

#### Deployment and Identity
The system runs via Python package, Docker, or desktop distribution; deployment definitions are `Dockerfile`, `docker-compose.yml`, and `pyproject.toml`. Authentication supports OIDC, GitHub OAuth, Azure Easy Auth, Azure CLI, and Kusto OAuth, with token storage and local credential vault abstractions under `py-src/data_formulator/auth/`. The frontend auth lifecycle is represented by `src/app/AuthButton.tsx` and `src/app/OidcCallback.tsx`.

Main entry points: [`py-src/data_formulator/routes/agents.py`](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/routes/agents.py) (API route boundary), [`py-src/data_formulator/analyst/agent.py`](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/analyst/agent.py) (Analyst agent), [`src/app/App.tsx`](https://github.com/microsoft/data-formulator/blob/main/src/app/App.tsx) (Application shell), [`src/app/store.ts`](https://github.com/microsoft/data-formulator/blob/main/src/app/store.ts) (Workspace state)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Browser client"]
  node_browser_entry["React/Vite entry<br/>frontend entry<br/>[index.tsx]"]
  node_app_shell["Application shell<br/>React app<br/>[App.tsx]"]
  node_client_store["Workspace state<br/>client store<br/>[store.ts]"]
  node_session_bridge["Session persistence<br/>workspace service"]
  node_analyst_ui["Data Thread workspace<br/>analyst view<br/>[DataFormulator.tsx]"]
  node_source_ui["Source configuration<br/>connector UI"]
  node_chart_ui["Chart rendering<br/>visualization UI"]
  node_auth_ui["OIDC callback<br/>auth UI<br/>[OidcCallback.tsx]"]
end

subgraph group_api["Python API"]
  node_python_entry["Python runtime<br/>package entry<br/>[__main__.py]"]
  node_api_app["API application<br/>Python app<br/>[app.py]"]
  node_api_routes["API route boundary<br/>route modules<br/>[agents.py]"]
  node_error_boundary["Structured errors<br/>error handler<br/>[error_handler.py]"]
  node_auth_backend["Identity and vault<br/>auth subsystem"]
end

subgraph group_data["Data workspace"]
  node_connectors["Connector boundary<br/>data connector<br/>[data_connector.py]"]
  node_external_loader["External loaders<br/>loader implementations"]
  node_workspace[("Managed workspace<br/>data lake workspace<br/>[workspace.py]")]
  node_operations["Data operations<br/>operation executor<br/>[executor.py]"]
end

subgraph group_analysis["AI analysis"]
  node_agent{{"Analyst agent<br/>agent orchestrator<br/>[agent.py]"}}
  node_model_config["Model selection<br/>model registry<br/>[model_registry.py]"]
  node_chart_workflow["Chart generation<br/>visualization workflow<br/>[create_vl_plots.py]"]
  node_sandbox["Execution safety<br/>sandbox and security"]
end

subgraph group_external["External services"]
  node_data_sources[("Data sources<br/>files, databases, cloud storage")]
  node_model_providers(("LLM providers<br/>OpenAI, Azure, Ollama, Anthropic"))
end

node_browser_entry -->|"boots"| node_app_shell
node_app_shell -->|"uses"| node_client_store
node_client_store -->|"persists sessions"| node_session_bridge
node_app_shell -->|"renders"| node_analyst_ui
node_analyst_ui -->|"configures data"| node_source_ui
node_analyst_ui -->|"displays charts"| node_chart_ui
node_auth_ui -->|"completes login"| node_auth_backend
node_client_store -->|"API requests"| node_api_routes
node_python_entry -->|"starts"| node_api_app
node_api_app -->|"mounts"| node_api_routes
node_api_routes -->|"maps failures"| node_error_boundary
node_api_routes -->|"authenticates"| node_auth_backend
node_api_routes -->|"loads sources"| node_connectors
node_connectors -->|"delegates remote loads"| node_external_loader
node_external_loader -->|"connects to"| node_data_sources
node_connectors -->|"stores loaded data"| node_workspace
node_api_routes -->|"executes transforms"| node_operations
node_operations -->|"reads and writes"| node_workspace
node_api_routes -->|"runs analysis"| node_agent
node_agent -->|"uses context and data"| node_workspace
node_agent -->|"selects model"| node_model_config
node_model_config -->|"invokes via LiteLLM"| node_model_providers
node_agent -->|"generates plots"| node_chart_workflow
node_chart_workflow -->|"returns specifications"| node_chart_ui
node_agent -->|"executes safely"| node_sandbox

click node_browser_entry "https://github.com/microsoft/data-formulator/blob/main/src/index.tsx"
click node_app_shell "https://github.com/microsoft/data-formulator/blob/main/src/app/App.tsx"
click node_client_store "https://github.com/microsoft/data-formulator/blob/main/src/app/store.ts"
click node_session_bridge "https://github.com/microsoft/data-formulator/blob/main/src/app/workspaceService.ts"
click node_analyst_ui "https://github.com/microsoft/data-formulator/blob/main/src/views/DataFormulator.tsx"
click node_source_ui "https://github.com/microsoft/data-formulator/blob/main/src/views/DataSourceSidebar.tsx"
click node_chart_ui "https://github.com/microsoft/data-formulator/blob/main/src/views/VisualizationView.tsx"
click node_auth_ui "https://github.com/microsoft/data-formulator/blob/main/src/app/OidcCallback.tsx"
click node_python_entry "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/__main__.py"
click node_api_app "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/app.py"
click node_api_routes "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/routes/agents.py"
click node_error_boundary "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/error_handler.py"
click node_connectors "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/data_connector.py"
click node_external_loader "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/data_loader/external_data_loader.py"
click node_workspace "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/datalake/workspace.py"
click node_operations "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/data_operations/executor.py"
click node_agent "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/analyst/agent.py"
click node_model_config "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/model_registry.py"
click node_chart_workflow "https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/workflows/create_vl_plots.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser_entry,node_app_shell,node_client_store,node_session_bridge,node_analyst_ui,node_source_ui,node_chart_ui,node_auth_ui toneBlue
class node_python_entry,node_api_app,node_api_routes,node_error_boundary,node_auth_backend toneAmber
class node_connectors,node_external_loader,node_workspace,node_operations toneMint
class node_agent,node_model_config,node_chart_workflow,node_sandbox toneRose
class node_data_sources,node_model_providers toneIndigo
```

## Components

### Browser client

- [React/Vite entry](https://github.com/microsoft/data-formulator/blob/main/src/index.tsx): frontend entry · `src/index.tsx`
- [Application shell](https://github.com/microsoft/data-formulator/blob/main/src/app/App.tsx): React app · `src/app/App.tsx`
- [Workspace state](https://github.com/microsoft/data-formulator/blob/main/src/app/store.ts): client store · Owns data-thread state, API state, and caches. · `src/app/store.ts`
- [Session persistence](https://github.com/microsoft/data-formulator/blob/main/src/app/workspaceService.ts): workspace service · `src/app/workspaceService.ts`
- [Data Thread workspace](https://github.com/microsoft/data-formulator/blob/main/src/views/DataFormulator.tsx): analyst view · `src/views/DataFormulator.tsx`
- [Source configuration](https://github.com/microsoft/data-formulator/blob/main/src/views/DataSourceSidebar.tsx): connector UI · `src/views/DataSourceSidebar.tsx`
- [Chart rendering](https://github.com/microsoft/data-formulator/blob/main/src/views/VisualizationView.tsx): visualization UI · `src/views/VisualizationView.tsx`
- [OIDC callback](https://github.com/microsoft/data-formulator/blob/main/src/app/OidcCallback.tsx): auth UI · `src/app/OidcCallback.tsx`

### Python API

- [Python runtime](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/__main__.py): package entry · `py-src/data_formulator/__main__.py`
- [API application](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/app.py): Python app · Serves the API and packaged frontend. · `py-src/data_formulator/app.py`
- [API route boundary](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/routes/agents.py): route modules · Separates agent, table, and session request contracts. · `py-src/data_formulator/routes/agents.py`
- [Structured errors](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/error_handler.py): error handler · `py-src/data_formulator/error_handler.py`
- **Identity and vault**: auth subsystem · Manages OAuth flows, tokens, and local credentials.

### Data workspace

- [Connector boundary](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/data_connector.py): data connector · Abstracts files, databases, cloud stores, and Superset. · `py-src/data_formulator/data_connector.py`
- [External loaders](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/data_loader/external_data_loader.py): loader implementations · Probes and loads supported remote data sources. · `py-src/data_formulator/data_loader/external_data_loader.py`
- [Managed workspace](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/datalake/workspace.py): data lake workspace · Provides local or Azure Blob-backed storage and catalog metadata. · `py-src/data_formulator/datalake/workspace.py`
- [Data operations](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/data_operations/executor.py): operation executor · Persists and executes declarative transformations. · `py-src/data_formulator/data_operations/executor.py`

### AI analysis

- [Analyst agent](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/analyst/agent.py): agent orchestrator · Runs specialized analysis skills with workspace context. · `py-src/data_formulator/analyst/agent.py`
- [Model selection](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/model_registry.py): model registry · Configures LiteLLM-backed provider access. · `py-src/data_formulator/model_registry.py`
- [Chart generation](https://github.com/microsoft/data-formulator/blob/main/py-src/data_formulator/workflows/create_vl_plots.py): visualization workflow · Builds chart semantics and Flint/Vega-Lite plots. · `py-src/data_formulator/workflows/create_vl_plots.py`
- **Execution safety**: sandbox and security · Constrains generated work and protects paths, URLs, and logs.

### External services

- **Data sources**: files, databases, cloud storage
- **LLM providers**: OpenAI, Azure, Ollama, Anthropic

## Connections

- React/Vite entry → Application shell: boots
- Application shell → Workspace state: uses
- Workspace state → Session persistence: persists sessions
- Application shell → Data Thread workspace: renders
- Data Thread workspace → Source configuration: configures data
- Data Thread workspace → Chart rendering: displays charts
- OIDC callback → Identity and vault: completes login
- Workspace state → API route boundary: API requests
- Python runtime → API application: starts
- API application → API route boundary: mounts
- API route boundary → Structured errors: maps failures
- API route boundary → Identity and vault: authenticates
- API route boundary → Connector boundary: loads sources
- Connector boundary → External loaders: delegates remote loads
- External loaders → Data sources: connects to
- Connector boundary → Managed workspace: stores loaded data
- API route boundary → Data operations: executes transforms
- Data operations → Managed workspace: reads and writes
- API route boundary → Analyst agent: runs analysis
- Analyst agent → Managed workspace: uses context and data
- Analyst agent → Model selection: selects model
- Model selection → LLM providers: invokes via LiteLLM
- Analyst agent → Chart generation: generates plots
- Chart generation → Chart rendering: returns specifications
- Analyst agent → Execution safety: executes safely
