# eventual-inc/daft architecture

> Architecture diagram of the GitHub repository eventual-inc/daft, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eventual-inc/daft
- Repository: https://github.com/eventual-inc/daft

## Overview

Daft is a Python-native, Rust-powered dataframe engine for structured and multimodal AI workloads. The principal workflow is: a user builds or submits a dataframe query; Daft reads connected data, constructs and optimizes a logical plan, converts it into a local or Ray-backed execution pipeline, batches and processes micropartitions, then returns dataframe results or writes output. AI providers, catalogs, multimodal files, distributed execution, and the dashboard are major supported branches. Some relationships are inferred from documented module boundaries because only sampled excerpts were inspected.

Main entry points: [`src/daft-local-execution/src/pipeline.rs`](https://github.com/eventual-inc/daft/blob/main/src/daft-local-execution/src/pipeline.rs) (Physical Pipeline), [`daft/io/__init__.py`](https://github.com/eventual-inc/daft/blob/main/daft/io/__init__.py) (IO Layer), [`daft/dataframe/dataframe.py`](https://github.com/eventual-inc/daft/blob/main/daft/dataframe/dataframe.py) (Dataframe API), [`daft/logical/builder.py`](https://github.com/eventual-inc/daft/blob/main/daft/logical/builder.py) (Logical Builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api_planning["API And Planning"]
  node_python_api["Python API<br/>[__init__.py]"]
  node_dataframe_api["Dataframe API<br/>[dataframe.py]"]
  node_logical_builder["Logical Builder<br/>[builder.py]"]
  node_sql_api["SQL Interface<br/>[sql.py]"]
  node_optimizer["Plan Optimizer<br/>[unnest_subquery.rs]"]
  node_result_view["Result Display<br/>[display.py]"]
end

subgraph group_data_access["Data Access"]
  node_io_layer["IO Layer<br/>[__init__.py]"]
  node_storage_config["Storage Config<br/>[storage_config.rs]"]
  node_catalogs["Catalog Connectors<br/>[__init__.py]"]
  node_multimodal_files["Multimodal Files<br/>[file.py]"]
  node_writers["Data Writers<br/>[storage_backend.rs]"]
end

subgraph group_execution["Query Execution"]
  node_pipeline["Physical Pipeline<br/>[pipeline.rs]"]
  node_batch_manager["Batch Manager<br/>[batch_manager.rs]"]
  node_core_engine["Columnar Core<br/>[lib.rs]"]
  node_local_runner["Local Runner<br/>[native_runner.py]"]
  node_ray_execution["Ray Execution<br/>[worker_manager.rs]"]
end

subgraph group_integrations["AI And Integrations"]
  node_ai_api["AI Providers<br/>[provider.py]"]
  node_ai_expressions["AI Expressions<br/>[_expressions.py]"]
end

subgraph group_observability["Observability"]
  node_dashboard_engine["Dashboard Engine<br/>[engine.rs]"]
  node_dashboard_ui["Dashboard UI<br/>[page.tsx]"]
end

node_user(("Data Engineer"))
node_object_storage[("Object Storage")]
node_ray_cluster{{"Ray Cluster"}}
node_ai_services{{"AI Services"}}
node_catalog_service[("Catalog Service")]

node_user -->|"uses"| node_python_api
node_python_api -->|"creates"| node_dataframe_api
node_dataframe_api -->|"builds plan"| node_logical_builder
node_sql_api -->|"builds plan"| node_logical_builder
node_logical_builder -->|"optimizes"| node_optimizer
node_optimizer -->|"converts plan"| node_pipeline
node_dataframe_api -->|"reads data"| node_io_layer
node_storage_config -->|"creates client"| node_io_layer
node_catalogs -.->|"queries catalog"| node_catalog_service
node_catalogs -->|"resolves tables"| node_io_layer
node_multimodal_files -->|"loads files"| node_io_layer
node_io_layer -.->|"reads data"| node_object_storage
node_local_runner -->|"runs pipeline"| node_pipeline
node_pipeline -->|"batches input"| node_batch_manager
node_pipeline -->|"executes operators"| node_core_engine
node_ray_execution -->|"distributes pipeline"| node_pipeline
node_ray_execution -->|"manages workers"| node_ray_cluster
node_pipeline -->|"writes results"| node_writers
node_writers -.->|"uploads output"| node_object_storage
node_dataframe_api -->|"adds AI ops"| node_ai_expressions
node_ai_expressions -->|"dispatches inference"| node_ai_api
node_ai_api -.->|"calls models"| node_ai_services
node_pipeline -.->|"reports progress"| node_dashboard_engine
node_dashboard_engine -->|"serves status"| node_dashboard_ui
node_local_runner -->|"returns results"| node_result_view

click node_python_api "https://github.com/eventual-inc/daft/blob/main/daft/__init__.py"
click node_dataframe_api "https://github.com/eventual-inc/daft/blob/main/daft/dataframe/dataframe.py"
click node_logical_builder "https://github.com/eventual-inc/daft/blob/main/daft/logical/builder.py"
click node_sql_api "https://github.com/eventual-inc/daft/blob/main/daft/sql/sql.py"
click node_optimizer "https://github.com/eventual-inc/daft/blob/main/src/daft-logical-plan/src/optimization/rules/unnest_subquery.rs"
click node_io_layer "https://github.com/eventual-inc/daft/blob/main/daft/io/__init__.py"
click node_storage_config "https://github.com/eventual-inc/daft/blob/main/src/daft-scan/src/storage_config.rs"
click node_catalogs "https://github.com/eventual-inc/daft/blob/main/daft/catalog/__init__.py"
click node_multimodal_files "https://github.com/eventual-inc/daft/blob/main/daft/file/file.py"
click node_pipeline "https://github.com/eventual-inc/daft/blob/main/src/daft-local-execution/src/pipeline.rs"
click node_batch_manager "https://github.com/eventual-inc/daft/blob/main/src/daft-local-execution/src/batch_manager.rs"
click node_core_engine "https://github.com/eventual-inc/daft/blob/main/src/daft-core/src/lib.rs"
click node_local_runner "https://github.com/eventual-inc/daft/blob/main/daft/runners/native_runner.py"
click node_ray_execution "https://github.com/eventual-inc/daft/blob/main/src/daft-distributed/src/python/ray/worker_manager.rs"
click node_writers "https://github.com/eventual-inc/daft/blob/main/src/daft-writers/src/storage_backend.rs"
click node_ai_api "https://github.com/eventual-inc/daft/blob/main/daft/ai/provider.py"
click node_ai_expressions "https://github.com/eventual-inc/daft/blob/main/daft/ai/_expressions.py"
click node_dashboard_engine "https://github.com/eventual-inc/daft/blob/main/src/daft-dashboard/src/engine.rs"
click node_dashboard_ui "https://github.com/eventual-inc/daft/blob/main/src/daft-dashboard/frontend/src/app/page.tsx"
click node_result_view "https://github.com/eventual-inc/daft/blob/main/daft/dataframe/display.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_python_api,node_dataframe_api,node_logical_builder,node_sql_api,node_optimizer,node_result_view toneBlue
class node_io_layer,node_storage_config,node_catalogs,node_multimodal_files,node_writers,node_object_storage,node_catalog_service toneAmber
class node_pipeline,node_batch_manager,node_core_engine,node_local_runner,node_ray_execution toneMint
class node_ai_api,node_ai_expressions toneRose
class node_dashboard_engine,node_dashboard_ui,node_user,node_ray_cluster,node_ai_services toneIndigo
```

## Components

### API And Planning

- [Python API](https://github.com/eventual-inc/daft/blob/main/daft/__init__.py): `daft/__init__.py`
- [Dataframe API](https://github.com/eventual-inc/daft/blob/main/daft/dataframe/dataframe.py): `daft/dataframe/dataframe.py`
- [Logical Builder](https://github.com/eventual-inc/daft/blob/main/daft/logical/builder.py): `daft/logical/builder.py`
- [SQL Interface](https://github.com/eventual-inc/daft/blob/main/daft/sql/sql.py): `daft/sql/sql.py`
- [Plan Optimizer](https://github.com/eventual-inc/daft/blob/main/src/daft-logical-plan/src/optimization/rules/unnest_subquery.rs): `src/daft-logical-plan/src/optimization/rules/unnest_subquery.rs`
- [Result Display](https://github.com/eventual-inc/daft/blob/main/daft/dataframe/display.py): `daft/dataframe/display.py`

### Data Access

- [IO Layer](https://github.com/eventual-inc/daft/blob/main/daft/io/__init__.py): `daft/io/__init__.py`
- [Storage Config](https://github.com/eventual-inc/daft/blob/main/src/daft-scan/src/storage_config.rs): `src/daft-scan/src/storage_config.rs`
- [Catalog Connectors](https://github.com/eventual-inc/daft/blob/main/daft/catalog/__init__.py): `daft/catalog/__init__.py`
- [Multimodal Files](https://github.com/eventual-inc/daft/blob/main/daft/file/file.py): `daft/file/file.py`
- [Data Writers](https://github.com/eventual-inc/daft/blob/main/src/daft-writers/src/storage_backend.rs): `src/daft-writers/src/storage_backend.rs`

### Query Execution

- [Physical Pipeline](https://github.com/eventual-inc/daft/blob/main/src/daft-local-execution/src/pipeline.rs): `src/daft-local-execution/src/pipeline.rs`
- [Batch Manager](https://github.com/eventual-inc/daft/blob/main/src/daft-local-execution/src/batch_manager.rs): `src/daft-local-execution/src/batch_manager.rs`
- [Columnar Core](https://github.com/eventual-inc/daft/blob/main/src/daft-core/src/lib.rs): `src/daft-core/src/lib.rs`
- [Local Runner](https://github.com/eventual-inc/daft/blob/main/daft/runners/native_runner.py): `daft/runners/native_runner.py`
- [Ray Execution](https://github.com/eventual-inc/daft/blob/main/src/daft-distributed/src/python/ray/worker_manager.rs): `src/daft-distributed/src/python/ray/worker_manager.rs`

### AI And Integrations

- [AI Providers](https://github.com/eventual-inc/daft/blob/main/daft/ai/provider.py): `daft/ai/provider.py`
- [AI Expressions](https://github.com/eventual-inc/daft/blob/main/daft/ai/_expressions.py): `daft/ai/_expressions.py`

### Observability

- [Dashboard Engine](https://github.com/eventual-inc/daft/blob/main/src/daft-dashboard/src/engine.rs): `src/daft-dashboard/src/engine.rs`
- [Dashboard UI](https://github.com/eventual-inc/daft/blob/main/src/daft-dashboard/frontend/src/app/page.tsx): `src/daft-dashboard/frontend/src/app/page.tsx`

### Other components

- **Data Engineer**
- **Object Storage**
- **Ray Cluster**
- **AI Services**
- **Catalog Service**

## Connections

- Data Engineer → Python API: uses
- Python API → Dataframe API: creates
- Dataframe API → Logical Builder: builds plan
- SQL Interface → Logical Builder: builds plan
- Logical Builder → Plan Optimizer: optimizes
- Plan Optimizer → Physical Pipeline: converts plan
- Dataframe API → IO Layer: reads data
- Storage Config → IO Layer: creates client
- Catalog Connectors → Catalog Service: queries catalog
- Catalog Connectors → IO Layer: resolves tables
- Multimodal Files → IO Layer: loads files
- IO Layer → Object Storage: reads data
- Local Runner → Physical Pipeline: runs pipeline
- Physical Pipeline → Batch Manager: batches input
- Physical Pipeline → Columnar Core: executes operators
- Ray Execution → Physical Pipeline: distributes pipeline
- Ray Execution → Ray Cluster: manages workers
- Physical Pipeline → Data Writers: writes results
- Data Writers → Object Storage: uploads output
- Dataframe API → AI Expressions: adds AI ops
- AI Expressions → AI Providers: dispatches inference
- AI Providers → AI Services: calls models
- Physical Pipeline → Dashboard Engine: reports progress
- Dashboard Engine → Dashboard UI: serves status
- Local Runner → Result Display: returns results
