How slothflowlabs/duckle works
Duckle is a self-hosted ETL platform: users author pipelines visually or in SQL/Python, run them locally, then deploy the same pipeline file for scheduled headless execution. The main path runs from workspace editing and persistence through the runner into DuckDB-backed execution and pipeline outputs. Separate capabilities include server operation and scheduling, dbt, AI assistance, Git workflows, and MCP access. README documentation supports these product branches; sampled source confirms selected engine provisioning and runner behavior, but does not establish every internal call path.
Main entry points: src/main.rs Headless runner · App.tsx Workflow editor · lib.rs DuckDB engine · src/main.rs Desktop app
Components 21
Pipeline authoring
- Desktop app
apps/desktop/src/main.rs - Workflow editor
frontend/src/App.tsx - Pipeline canvas
frontend/src/canvas/Canvas.tsx - Pipeline model
frontend/src/pipeline-types.ts - Workspace files
frontend/src/workspace.ts
Pipeline execution
- Headless runner
crates/duckle-runner/src/main.rs - Workflow engine
crates/workflow-engine/src/lib.rs - DuckDB engine
crates/duckdb-engine/src/lib.rs - Data connectors
crates/connectors/src/lib.rs
Deployment and operations
- Scheduling
crates/scheduler/src/lib.rs - Server console
crates/duckle-runner/src/serve.rs - Audit and roles
crates/duckle-runner/src/audit.rs - Pipeline bundles
frontend/src/workflow-ui/BuildPipelineModal.tsx
Optional integrations
- dbt integration
apps/desktop/src/dbt_engine.rs - Duckie assistant
apps/desktop/src/llama_chat.rs - Git workflow
apps/desktop/src/workspace_git.rs - MCP server
crates/duckle-mcp/src/main.rs
Other components
- Pipeline author
- External agent
- Data sources
- Pipeline outputs
Connections 21
- Pipeline author → Desktop app: authors in
- Desktop app → Workflow editor: opens editor
- Workflow editor → Pipeline canvas: edits graph
- Pipeline canvas → Pipeline model: defines nodes
- Workflow editor → Workspace files: persists pipelines
- Workspace files → Headless runner: supplies pipeline
- Headless runner → Workflow engine: runs workflow
- Workflow engine → DuckDB engine: executes plan
- DuckDB engine → Data connectors: uses connectors
- Data connectors → Data sources: reads and writes
- DuckDB engine → Pipeline outputs: produces results
- Pipeline author → Pipeline bundles: builds artifact
- Pipeline bundles → Headless runner: embeds runner
- Scheduling → Headless runner: triggers runs
- Pipeline author → Server console: operates pipelines
- Server console → Audit and roles: records activity
- Headless runner → dbt integration: runs dbt nodes
- Pipeline author → Duckie assistant: requests assistance
- Pipeline author → Git workflow: manages workspace
- External agent → MCP server: requests tools
- MCP server → Headless runner: accesses pipelines