How zerodha/dungbeetle works
DungBeetle accepts SQL-job requests over HTTP, queues work for asynchronous workers, executes registered SQL tasks against source databases, and writes each job’s results to a result database. Its API also exposes job and group status, cancellation, pending-job listings, and metrics; the Go client wraps several API operations. The graph follows the sampled startup, routing, task-loading, execution, and result-backend code. Broker choices and some execution wiring are documented but not fully sampled, so those relationships are cited to the README or left unverified.
Main entry points: core.go Job Core · init.go Runtime Wiring · http.go HTTP API · sqldb.go SQL Result Backend
Components 15
API and Client
- Go API Client
client/client.go - HTTP API
cmd/http.go - API Models
models/models.go
Job Runtime
- Server Startup
cmd/main.go - Runtime Wiring
cmd/init.go - SQL Task Loading
internal/core/tasks.go - Job Core
internal/core/core.go - Job Queue and State
SQL Data
- Source DB Pools
internal/dbpool/dbpool.go - Source SQL Databases
- Result Backend Selection
internal/core/results.go - SQL Result Backend
internal/resultbackends/sqldb/sqldb.go - Results Databases
Operations
- Metrics
internal/core/metrics.go
Other components
- Calling Application
Connections 22
- Calling Application → Go API Client: uses
client/client.go - Go API Client → HTTP API: sends HTTP
client/client.go - Server Startup → Runtime Wiring: initializes
cmd/main.go - Server Startup → HTTP API: starts
cmd/main.go - Runtime Wiring → Job Core: constructs
cmd/init.go - Runtime Wiring → Source DB Pools: opens pools
cmd/init.go - Runtime Wiring → SQL Result Backend: constructs
cmd/init.go - Runtime Wiring → SQL Task Loading: loads tasks
cmd/init.go - Runtime Wiring → Job Queue and State: configures
cmd/init.go - HTTP API → Job Core: dispatches jobs
cmd/http.go - HTTP API → API Models: uses schemas
cmd/http.go - Job Core → Job Queue and State: schedules work
README.md - Job Core → SQL Task Loading: executes tasks
internal/core/core.go - SQL Task Loading → Source DB Pools: selects databases
internal/core/tasks.go - Source DB Pools → Source SQL Databases: connects
internal/dbpool/dbpool.go - Job Core → Source SQL Databases: runs SQL
internal/core/core.go - Job Core → Result Backend Selection: selects backend
- Result Backend Selection → SQL Result Backend: chooses backend
- Job Core → SQL Result Backend: writes results
README.md - SQL Result Backend → Results Databases: stores tables
internal/resultbackends/sqldb/sqldb.go - HTTP API → Metrics: exposes metrics
cmd/http.go - Job Core → Metrics: records metrics