How microsoft/data-formulator works
Purpose and Entry Points
Data Formulator is a local-first AI-assisted data exploration system: users load/connect data, conduct branching analyses in a Data Thread, generate/refine charts, and compose reports. Python serves the API and packaged frontend; React/Vite provides the browser UI. Main runtime entry points are py-src/data_formulator/__main__.py, py-src/data_formulator/app.py, and src/index.tsx. Desktop packaging adds py-src/data_formulator/desktop.py and packaging/data_formulator_desktop.py.
Client Workspace and Interaction Model
The React application owns interactive workspace state, persisted sessions, API calls, auth token handling, and chart/table caches. src/app/App.tsx establishes the application shell; src/app/store.ts and src/app/dfSlice.tsx manage client data-thread/workspace state; src/app/workspaceService.ts bridges session persistence. The primary analyst experience is rendered through src/views/DataFormulator.tsx, src/views/DataThread.tsx, and src/views/DataLoadingChat.tsx.
HTTP API, Sessions, and Contracts
The Python application exposes routes separating agent execution, tables/loading, session lifecycle, credentials, models, knowledge, and logs. Route modules are the boundary between frontend request contracts and backend services: py-src/data_formulator/routes/agents.py, py-src/data_formulator/routes/tables.py, and py-src/data_formulator/routes/sessions.py. Structured error handling is centralized in py-src/data_formulator/error_handler.py and py-src/data_formulator/errors.py, rather than leaking connector or agent failures directly to clients.
Data Ingestion and Connector Boundary
A common connector/data-loader layer abstracts local files, folders, samples, databases, warehouses, cloud storage, and Superset. Core integration points are py-src/data_formulator/data_connector.py, py-src/data_formulator/data_loader/external_data_loader.py, and py-src/data_formulator/data_loader/probe_utils.py. Concrete loaders include PostgreSQL, BigQuery, Databricks, Kusto, S3, Azure Blob, MongoDB, and Superset. The UI discovers/configures these sources through src/views/DataSourceSidebar.tsx and src/components/ConnectorFormCard.tsx.
Workspace, Catalog, and Data Execution
Loaded source data is organized in managed workspaces with metadata/catalog caching and Parquet-oriented utilities. py-src/data_formulator/datalake/workspace.py, py-src/data_formulator/datalake/workspace_manager.py, and py-src/data_formulator/datalake/catalog_refresh.py define this storage boundary; it can use ephemeral local workspaces or Azure Blob-backed workspaces. Declarative data operations are modeled, persisted, and executed by py-src/data_formulator/data_operations/models.py, py-src/data_formulator/data_operations/repository.py, and py-src/data_formulator/data_operations/executor.py.
Agent and Analysis Pipeline
Agent requests receive workspace/context and use specialized capabilities for data loading, diagnostics, chart restyling, sorting, code explanation, and workflow distillation. Primary orchestration artifacts are py-src/data_formulator/analyst/agent.py, py-src/data_formulator/agents/context.py, and py-src/data_formulator/agents/reasoning_log.py; skill packages expose tool-oriented behavior in py-src/data_formulator/analyst/skills/. Model selection/configuration is separated in py-src/data_formulator/model_registry.py and py-src/data_formulator/agent_config.py. Supported external model providers include OpenAI, Azure, Ollama, and Anthropic through LiteLLM.
Visualization, Reports, and Safety Boundaries
Chart semantics and Flint/Vega-Lite plot generation are backend workflow concerns in py-src/data_formulator/workflows/chart_semantics.py and py-src/data_formulator/workflows/create_vl_plots.py; interactive rendering and editing occur in src/views/VisualizationView.tsx and src/views/ChartRenderService.tsx. Reports are assembled in src/views/ReportView.tsx. Generated/executed work is constrained by sandbox implementations in py-src/data_formulator/sandbox/ and security controls for paths, URL allowlisting, sanitization, code signing, and log redaction in py-src/data_formulator/security/.
Deployment and Identity
The system runs via Python package, Docker, or desktop distribution; deployment definitions are Dockerfile, docker-compose.yml, and pyproject.toml. Authentication supports OIDC, GitHub OAuth, Azure Easy Auth, Azure CLI, and Kusto OAuth, with token storage and local credential vault abstractions under py-src/data_formulator/auth/. The frontend auth lifecycle is represented by src/app/AuthButton.tsx and src/app/OidcCallback.tsx.
Main entry points: agents.py API route boundary · agent.py Analyst agent · App.tsx Application shell · store.ts Workspace state
Components 23
Browser client
- React/Vite entry
src/index.tsx - Application shell
src/app/App.tsx - Workspace state · Owns data-thread state, API state, and caches.
src/app/store.ts - Session persistence
src/app/workspaceService.ts - Data Thread workspace
src/views/DataFormulator.tsx - Source configuration
src/views/DataSourceSidebar.tsx - Chart rendering
src/views/VisualizationView.tsx - OIDC callback
src/app/OidcCallback.tsx
Python API
- Python runtime
py-src/data_formulator/__main__.py - API application · Serves the API and packaged frontend.
py-src/data_formulator/app.py - API route boundary · Separates agent, table, and session request contracts.
py-src/data_formulator/routes/agents.py - Structured errors
py-src/data_formulator/error_handler.py - Identity and vault · Manages OAuth flows, tokens, and local credentials.
Data workspace
- Connector boundary · Abstracts files, databases, cloud stores, and Superset.
py-src/data_formulator/data_connector.py - External loaders · Probes and loads supported remote data sources.
py-src/data_formulator/data_loader/external_data_loader.py - Managed workspace · Provides local or Azure Blob-backed storage and catalog metadata.
py-src/data_formulator/datalake/workspace.py - Data operations · Persists and executes declarative transformations.
py-src/data_formulator/data_operations/executor.py
AI analysis
- Analyst agent · Runs specialized analysis skills with workspace context.
py-src/data_formulator/analyst/agent.py - Model selection · Configures LiteLLM-backed provider access.
py-src/data_formulator/model_registry.py - Chart generation · Builds chart semantics and Flint/Vega-Lite plots.
py-src/data_formulator/workflows/create_vl_plots.py - Execution safety · Constrains generated work and protects paths, URLs, and logs.
External services
- Data sources
- LLM providers
Connections 25
- React/Vite entry → Application shell: boots
- Application shell → Workspace state: uses
- Workspace state → Session persistence: persists sessions
- Application shell → Data Thread workspace: renders
- Data Thread workspace → Source configuration: configures data
- Data Thread workspace → Chart rendering: displays charts
- OIDC callback → Identity and vault: completes login
- Workspace state → API route boundary: API requests
- Python runtime → API application: starts
- API application → API route boundary: mounts
- API route boundary → Structured errors: maps failures
- API route boundary → Identity and vault: authenticates
- API route boundary → Connector boundary: loads sources
- Connector boundary → External loaders: delegates remote loads
- External loaders → Data sources: connects to
- Connector boundary → Managed workspace: stores loaded data
- API route boundary → Data operations: executes transforms
- Data operations → Managed workspace: reads and writes
- API route boundary → Analyst agent: runs analysis
- Analyst agent → Managed workspace: uses context and data
- Analyst agent → Model selection: selects model
- Model selection → LLM providers: invokes via LiteLLM
- Analyst agent → Chart generation: generates plots
- Chart generation → Chart rendering: returns specifications
- Analyst agent → Execution safety: executes safely