How eventual-inc/daft works
Daft is a Python-native, Rust-powered dataframe engine for structured and multimodal AI workloads. The principal workflow is: a user builds or submits a dataframe query; Daft reads connected data, constructs and optimizes a logical plan, converts it into a local or Ray-backed execution pipeline, batches and processes micropartitions, then returns dataframe results or writes output. AI providers, catalogs, multimodal files, distributed execution, and the dashboard are major supported branches. Some relationships are inferred from documented module boundaries because only sampled excerpts were inspected.
Main entry points: pipeline.rs Physical Pipeline · __init__.py IO Layer · dataframe.py Dataframe API · builder.py Logical Builder
Components 25
API And Planning
- Python API
daft/__init__.py - Dataframe API
daft/dataframe/dataframe.py - Logical Builder
daft/logical/builder.py - SQL Interface
daft/sql/sql.py - Plan Optimizer
src/daft-logical-plan/src/optimization/rules/unnest_subquery.rs - Result Display
daft/dataframe/display.py
Data Access
- IO Layer
daft/io/__init__.py - Storage Config
src/daft-scan/src/storage_config.rs - Catalog Connectors
daft/catalog/__init__.py - Multimodal Files
daft/file/file.py - Data Writers
src/daft-writers/src/storage_backend.rs
Query Execution
- Physical Pipeline
src/daft-local-execution/src/pipeline.rs - Batch Manager
src/daft-local-execution/src/batch_manager.rs - Columnar Core
src/daft-core/src/lib.rs - Local Runner
daft/runners/native_runner.py - Ray Execution
src/daft-distributed/src/python/ray/worker_manager.rs
AI And Integrations
- AI Providers
daft/ai/provider.py - AI Expressions
daft/ai/_expressions.py
Observability
- Dashboard Engine
src/daft-dashboard/src/engine.rs - Dashboard UI
src/daft-dashboard/frontend/src/app/page.tsx
Other components
- Data Engineer
- Object Storage
- Ray Cluster
- AI Services
- Catalog Service
Connections 25
- Data Engineer → Python API: uses
- Python API → Dataframe API: creates
- Dataframe API → Logical Builder: builds plan
- SQL Interface → Logical Builder: builds plan
- Logical Builder → Plan Optimizer: optimizes
- Plan Optimizer → Physical Pipeline: converts plan
- Dataframe API → IO Layer: reads data
- Storage Config → IO Layer: creates client
- Catalog Connectors → Catalog Service: queries catalog
- Catalog Connectors → IO Layer: resolves tables
- Multimodal Files → IO Layer: loads files
- IO Layer → Object Storage: reads data
- Local Runner → Physical Pipeline: runs pipeline
- Physical Pipeline → Batch Manager: batches input
- Physical Pipeline → Columnar Core: executes operators
- Ray Execution → Physical Pipeline: distributes pipeline
- Ray Execution → Ray Cluster: manages workers
- Physical Pipeline → Data Writers: writes results
- Data Writers → Object Storage: uploads output
- Dataframe API → AI Expressions: adds AI ops
- AI Expressions → AI Providers: dispatches inference
- AI Providers → AI Services: calls models
- Physical Pipeline → Dashboard Engine: reports progress
- Dashboard Engine → Dashboard UI: serves status
- Local Runner → Result Display: returns results