How databendlabs/databend works
Databend is a Rust enterprise data warehouse. Its principal workflow is a client submitting SQL through the HTTP query API, followed by query planning and pipeline execution; results are returned through the query interface. The README also describes analytics, vector and full-text search, schema evolution, transactions, branching, and sandboxed Python UDFs. The graph preserves these product areas, but several are represented by their existing source directories because their implementations were not sampled. Links without direct evidence are omitted or marked unverified.
Main entry points: http_query_handlers.rs HTTP query API · src Query planning · query_pipeline_executor.rs Pipeline executor · api_impl Metadata API
Components 20
SQL query engine
- HTTP query API
src/query/service/src/servers/http/v1/http_query_handlers.rs - Query planning
src/query/service/src - Pipeline construction
src/query/service/src/pipelines/pipeline_builder.rs - Pipeline graph
src/query/pipeline/src/core/pipeline.rs - Pipeline executor
src/query/service/src/pipelines/executor/query_pipeline_executor.rs - Running graph
src/query/service/src/pipelines/executor/executor_graph.rs - Worker task queue
src/query/service/src/pipelines/executor/query_executor_tasks.rs - Query context
src/query/service/src/sessions
Data and search
- Table storage
src/query/storages - Vector and text search
src/query
Metadata services
- Transactions
src/meta/api/src/txn - Data branching
src/query/ee/src/data_sharing/storage.rs - Metadata API
src/meta/api/src/api_impl - Metadata storage
src/meta/app-storage/src/storage_params.rs - Tenant keys
src/meta/app/src/tenant_key/ident.rs
Agent execution
- Sandbox UDFs
src/query/script - Sandbox workers
Runtime support
- Memory management
src/common/base/src/runtime/memory/memory_manager.rs - Stack symbolization
src/common/exception/src/elf/library_manager.rs
Other components
- SQL client
Connections 15
- SQL client → HTTP query API: submits SQL
README.md - HTTP query API → Query planning: starts query
src/query/service/src/servers/http/v1/http_query_handlers.rs - Query planning → Pipeline construction: builds pipeline
src/query/service/src/pipelines/pipeline_builder.rs - Pipeline construction → Pipeline graph: constructs graph
src/query/service/src/pipelines/pipeline_builder.rs - Pipeline graph → Pipeline executor: executes pipeline
src/query/service/src/pipelines/executor/query_pipeline_executor.rs - Pipeline executor → Running graph: runs graph
src/query/service/src/pipelines/executor/query_pipeline_executor.rs - Running graph → Worker task queue: schedules tasks
src/query/service/src/pipelines/executor/executor_graph.rs - HTTP query API → Query context: uses sessions
src/query/service/src/servers/http/v1/http_query_handlers.rs - Query planning → Table storage: reads data
- Vector and text search → Table storage: searches data
src/query/storages/orc/src/read_pipeline.rs - Metadata API → Metadata storage: persists metadata
- Metadata API → Tenant keys: uses tenant keys
- Sandbox UDFs → Sandbox workers: dispatches logic
- HTTP query API → Sandbox UDFs: orchestrates with SQL
- Memory management → Pipeline executor: limits resources