# zerodha/dungbeetle architecture

> Architecture diagram of the GitHub repository zerodha/dungbeetle, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zerodha/dungbeetle
- Repository: https://github.com/zerodha/dungbeetle

## Overview

DungBeetle accepts SQL-job requests over HTTP, queues work for asynchronous workers, executes registered SQL tasks against source databases, and writes each job’s results to a result database. Its API also exposes job and group status, cancellation, pending-job listings, and metrics; the Go client wraps several API operations. The graph follows the sampled startup, routing, task-loading, execution, and result-backend code. Broker choices and some execution wiring are documented but not fully sampled, so those relationships are cited to the README or left unverified.

Main entry points: [`internal/core/core.go`](https://github.com/zerodha/dungbeetle/blob/master/internal/core/core.go) (Job Core), [`cmd/init.go`](https://github.com/zerodha/dungbeetle/blob/master/cmd/init.go) (Runtime Wiring), [`cmd/http.go`](https://github.com/zerodha/dungbeetle/blob/master/cmd/http.go) (HTTP API), [`internal/resultbackends/sqldb/sqldb.go`](https://github.com/zerodha/dungbeetle/blob/master/internal/resultbackends/sqldb/sqldb.go) (SQL Result Backend)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API and Client"]
  node_client["Go API Client<br/>[client.go]"]
  node_http["HTTP API<br/>[http.go]"]
  node_models["API Models<br/>[models.go]"]
end

subgraph group_runtime["Job Runtime"]
  node_startup["Server Startup<br/>[main.go]"]
  node_init["Runtime Wiring<br/>[init.go]"]
  node_tasks["SQL Task Loading<br/>[tasks.go]"]
  node_core["Job Core<br/>[core.go]"]
  node_queue["Job Queue and State"]
end

subgraph group_data["SQL Data"]
  node_sourcepool["Source DB Pools<br/>[dbpool.go]"]
  node_source[("Source SQL Databases")]
  node_resultselect["Result Backend Selection<br/>[results.go]"]
  node_resultbackend["SQL Result Backend<br/>[sqldb.go]"]
  node_results[("Results Databases")]
end

subgraph group_ops["Operations"]
  node_metrics["Metrics<br/>[metrics.go]"]
end

node_app(("Calling Application"))

node_app -->|"uses"| node_client
node_client -->|"sends HTTP"| node_http
node_startup -->|"initializes"| node_init
node_startup -->|"starts"| node_http
node_init -->|"constructs"| node_core
node_init -->|"opens pools"| node_sourcepool
node_init -->|"constructs"| node_resultbackend
node_init -->|"loads tasks"| node_tasks
node_init -->|"configures"| node_queue
node_http -->|"dispatches jobs"| node_core
node_http -->|"uses schemas"| node_models
node_core -.->|"schedules work"| node_queue
node_core -->|"executes tasks"| node_tasks
node_tasks -->|"selects databases"| node_sourcepool
node_sourcepool -->|"connects"| node_source
node_core -->|"runs SQL"| node_source
node_core -.->|"selects backend"| node_resultselect
node_resultselect -.->|"chooses backend"| node_resultbackend
node_core -.->|"writes results"| node_resultbackend
node_resultbackend -->|"stores tables"| node_results
node_http -->|"exposes metrics"| node_metrics
node_core -.->|"records metrics"| node_metrics

click node_client "https://github.com/zerodha/dungbeetle/blob/master/client/client.go"
click node_http "https://github.com/zerodha/dungbeetle/blob/master/cmd/http.go"
click node_models "https://github.com/zerodha/dungbeetle/blob/master/models/models.go"
click node_startup "https://github.com/zerodha/dungbeetle/blob/master/cmd/main.go"
click node_init "https://github.com/zerodha/dungbeetle/blob/master/cmd/init.go"
click node_tasks "https://github.com/zerodha/dungbeetle/blob/master/internal/core/tasks.go"
click node_core "https://github.com/zerodha/dungbeetle/blob/master/internal/core/core.go"
click node_sourcepool "https://github.com/zerodha/dungbeetle/blob/master/internal/dbpool/dbpool.go"
click node_resultselect "https://github.com/zerodha/dungbeetle/blob/master/internal/core/results.go"
click node_resultbackend "https://github.com/zerodha/dungbeetle/blob/master/internal/resultbackends/sqldb/sqldb.go"
click node_metrics "https://github.com/zerodha/dungbeetle/blob/master/internal/core/metrics.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client,node_http,node_models toneBlue
class node_startup,node_init,node_tasks,node_core,node_queue toneAmber
class node_sourcepool,node_source,node_resultselect,node_resultbackend,node_results toneMint
class node_metrics toneRose
class node_app toneIndigo
```

## Components

### API and Client

- [Go API Client](https://github.com/zerodha/dungbeetle/blob/master/client/client.go): `client/client.go`
- [HTTP API](https://github.com/zerodha/dungbeetle/blob/master/cmd/http.go): `cmd/http.go`
- [API Models](https://github.com/zerodha/dungbeetle/blob/master/models/models.go): `models/models.go`

### Job Runtime

- [Server Startup](https://github.com/zerodha/dungbeetle/blob/master/cmd/main.go): `cmd/main.go`
- [Runtime Wiring](https://github.com/zerodha/dungbeetle/blob/master/cmd/init.go): `cmd/init.go`
- [SQL Task Loading](https://github.com/zerodha/dungbeetle/blob/master/internal/core/tasks.go): `internal/core/tasks.go`
- [Job Core](https://github.com/zerodha/dungbeetle/blob/master/internal/core/core.go): `internal/core/core.go`
- **Job Queue and State**

### SQL Data

- [Source DB Pools](https://github.com/zerodha/dungbeetle/blob/master/internal/dbpool/dbpool.go): `internal/dbpool/dbpool.go`
- **Source SQL Databases**
- [Result Backend Selection](https://github.com/zerodha/dungbeetle/blob/master/internal/core/results.go): `internal/core/results.go`
- [SQL Result Backend](https://github.com/zerodha/dungbeetle/blob/master/internal/resultbackends/sqldb/sqldb.go): `internal/resultbackends/sqldb/sqldb.go`
- **Results Databases**

### Operations

- [Metrics](https://github.com/zerodha/dungbeetle/blob/master/internal/core/metrics.go): `internal/core/metrics.go`

### Other components

- **Calling Application**

## Connections

- Calling Application → Go API Client: uses (evidence: `client/client.go`)
- Go API Client → HTTP API: sends HTTP (evidence: `client/client.go`)
- Server Startup → Runtime Wiring: initializes (evidence: `cmd/main.go`)
- Server Startup → HTTP API: starts (evidence: `cmd/main.go`)
- Runtime Wiring → Job Core: constructs (evidence: `cmd/init.go`)
- Runtime Wiring → Source DB Pools: opens pools (evidence: `cmd/init.go`)
- Runtime Wiring → SQL Result Backend: constructs (evidence: `cmd/init.go`)
- Runtime Wiring → SQL Task Loading: loads tasks (evidence: `cmd/init.go`)
- Runtime Wiring → Job Queue and State: configures (evidence: `cmd/init.go`)
- HTTP API → Job Core: dispatches jobs (evidence: `cmd/http.go`)
- HTTP API → API Models: uses schemas (evidence: `cmd/http.go`)
- Job Core → Job Queue and State: schedules work (evidence: `README.md`)
- Job Core → SQL Task Loading: executes tasks (evidence: `internal/core/core.go`)
- SQL Task Loading → Source DB Pools: selects databases (evidence: `internal/core/tasks.go`)
- Source DB Pools → Source SQL Databases: connects (evidence: `internal/dbpool/dbpool.go`)
- Job Core → Source SQL Databases: runs SQL (evidence: `internal/core/core.go`)
- Job Core → Result Backend Selection: selects backend
- Result Backend Selection → SQL Result Backend: chooses backend
- Job Core → SQL Result Backend: writes results (evidence: `README.md`)
- SQL Result Backend → Results Databases: stores tables (evidence: `internal/resultbackends/sqldb/sqldb.go`)
- HTTP API → Metrics: exposes metrics (evidence: `cmd/http.go`)
- Job Core → Metrics: records metrics
