# vivekagent47/dstream architecture

> Architecture diagram of the GitHub repository vivekagent47/dstream, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vivekagent47/dstream
- Repository: https://github.com/vivekagent47/dstream

## Overview

dstream is a webhook gateway and developer console: senders post to an ingest URL, events are durably recorded and queued, and workers apply connection policy before delivering with retries and attempt visibility. The API and dashboard also manage sources, destinations, connections, outbound applications, endpoint portals, identity, and operational views; the CLI supports local forwarding. Edges cite excerpts or README statements; several dashboard and API capabilities are represented at their known paths, but unsampled internals and links are omitted where invocation evidence is unavailable.

Main entry points: [`internal/api/router.go`](https://github.com/vivekagent47/dstream/blob/main/internal/api/router.go) (HTTP API router), [`internal/dqueue/dqueue.go`](https://github.com/vivekagent47/dstream/blob/main/internal/dqueue/dqueue.go) (Redis queues), [`internal/ingest/handler.go`](https://github.com/vivekagent47/dstream/blob/main/internal/ingest/handler.go) (Webhook ingest), [`internal/deliver/handler.go`](https://github.com/vivekagent47/dstream/blob/main/internal/deliver/handler.go) (Connection delivery)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Inbound pipeline"]
  node_ingress["Webhook ingest<br/>[handler.go]"]
  node_durable[("Event store<br/>[events.sql.go]")]
end

subgraph group_delivery["Delivery runtime"]
  node_queue["Redis queues<br/>[dqueue.go]"]
  node_worker["Delivery worker<br/>[worker.go]"]
  node_delivery_handler["Connection delivery<br/>[handler.go]"]
  node_filter["Filter engine<br/>[filter.go]"]
  node_transform["Transform engine<br/>[transform.go]"]
end

subgraph group_outbound["Outbound webhooks"]
  node_outbound_api["Publish and portal API"]
  node_outbound_handler["Message fan-out<br/>[handler.go]"]
  node_app_store[("Application records")]
end

subgraph group_control["API and identity"]
  node_api["HTTP API router<br/>[router.go]"]
  node_pipeline_api["Pipeline management"]
  node_identity_api["Identity and orgs"]
  node_auth["Authentication"]
  node_admin["Operations console<br/>[router.go]"]
  node_metrics["Metrics collector<br/>[collector.go]"]
end

subgraph group_dashboard["Dashboard"]
  node_web["Dashboard routes<br/>[routeTree.gen.ts]"]
  node_api_client["Dashboard API client<br/>[api.ts]"]
  node_portal["Customer portal<br/>[route.tsx]"]
end

node_sender(("Webhook sender"))
node_endpoint(("Webhook endpoint"))
node_database[("PostgreSQL")]
node_cli["dstream CLI<br/>[cli.go]"]
node_local_app(("Local app"))

node_sender -->|"posts webhook"| node_ingress
node_ingress -->|"records request/events"| node_durable
node_ingress -.->|"enqueues delivery"| node_queue
node_worker -->|"consumes jobs"| node_queue
node_worker -->|"processes deliveries"| node_delivery_handler
node_delivery_handler -->|"filters payload"| node_filter
node_delivery_handler -->|"transforms payload"| node_transform
node_delivery_handler -->|"sends webhook"| node_endpoint
node_api -->|"routes pipeline"| node_pipeline_api
node_api -->|"routes identity"| node_identity_api
node_api -->|"routes outbound"| node_outbound_api
node_api -->|"applies auth"| node_auth
node_identity_api -->|"uses auth services"| node_auth
node_pipeline_api -->|"queues tests/replay"| node_queue
node_pipeline_api -->|"manages pipeline data"| node_durable
node_outbound_api -->|"stores applications"| node_app_store
node_outbound_handler -->|"schedules deliveries"| node_queue
node_outbound_handler -->|"filters messages"| node_filter
node_outbound_handler -->|"transforms messages"| node_transform
node_web -.->|"requests API"| node_api_client
node_api_client -.->|"calls endpoints"| node_api
node_portal -.->|"uses portal API"| node_api
node_admin -->|"inspects queues"| node_queue
node_metrics -->|"queries metrics"| node_database
node_worker -.->|"loads delivery state"| node_database
node_ingress -.->|"persists events"| node_database
node_cli -.->|"opens tunnel"| node_api
node_cli -.->|"forwards requests"| node_local_app

click node_ingress "https://github.com/vivekagent47/dstream/blob/main/internal/ingest/handler.go"
click node_durable "https://github.com/vivekagent47/dstream/blob/main/internal/store/events.sql.go"
click node_queue "https://github.com/vivekagent47/dstream/blob/main/internal/dqueue/dqueue.go"
click node_worker "https://github.com/vivekagent47/dstream/blob/main/cmd/dstream/worker.go"
click node_delivery_handler "https://github.com/vivekagent47/dstream/blob/main/internal/deliver/handler.go"
click node_filter "https://github.com/vivekagent47/dstream/blob/main/internal/filter/filter.go"
click node_transform "https://github.com/vivekagent47/dstream/blob/main/internal/transform/transform.go"
click node_api "https://github.com/vivekagent47/dstream/blob/main/internal/api/router.go"
click node_pipeline_api "https://github.com/vivekagent47/dstream/tree/main/internal/api/pipeline"
click node_identity_api "https://github.com/vivekagent47/dstream/tree/main/internal/api/identity"
click node_auth "https://github.com/vivekagent47/dstream/tree/main/internal/auth"
click node_outbound_api "https://github.com/vivekagent47/dstream/tree/main/internal/api/outbound"
click node_outbound_handler "https://github.com/vivekagent47/dstream/blob/main/internal/webhook/handler.go"
click node_app_store "https://github.com/vivekagent47/dstream/blob/main/internal/store/applications.sql.go"
click node_web "https://github.com/vivekagent47/dstream/blob/main/web/src/routeTree.gen.ts"
click node_api_client "https://github.com/vivekagent47/dstream/blob/main/web/src/lib/api.ts"
click node_portal "https://github.com/vivekagent47/dstream/blob/main/web/src/routes/portal/route.tsx"
click node_admin "https://github.com/vivekagent47/dstream/blob/main/internal/admin/router.go"
click node_metrics "https://github.com/vivekagent47/dstream/blob/main/internal/metrics/collector.go"
click node_cli "https://github.com/vivekagent47/dstream/blob/main/cmd/dstream/cli.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingress,node_durable toneBlue
class node_queue,node_worker,node_delivery_handler,node_filter,node_transform,node_database toneAmber
class node_outbound_api,node_outbound_handler,node_app_store,node_sender,node_endpoint toneMint
class node_api,node_pipeline_api,node_identity_api,node_auth,node_admin,node_metrics toneRose
class node_web,node_api_client,node_portal,node_local_app toneIndigo
class node_cli toneTeal
```

## Components

### Inbound pipeline

- [Webhook ingest](https://github.com/vivekagent47/dstream/blob/main/internal/ingest/handler.go): `internal/ingest/handler.go`
- [Event store](https://github.com/vivekagent47/dstream/blob/main/internal/store/events.sql.go): `internal/store/events.sql.go`

### Delivery runtime

- [Redis queues](https://github.com/vivekagent47/dstream/blob/main/internal/dqueue/dqueue.go): `internal/dqueue/dqueue.go`
- [Delivery worker](https://github.com/vivekagent47/dstream/blob/main/cmd/dstream/worker.go): `cmd/dstream/worker.go`
- [Connection delivery](https://github.com/vivekagent47/dstream/blob/main/internal/deliver/handler.go): `internal/deliver/handler.go`
- [Filter engine](https://github.com/vivekagent47/dstream/blob/main/internal/filter/filter.go): `internal/filter/filter.go`
- [Transform engine](https://github.com/vivekagent47/dstream/blob/main/internal/transform/transform.go): `internal/transform/transform.go`

### Outbound webhooks

- [Publish and portal API](https://github.com/vivekagent47/dstream/tree/main/internal/api/outbound): `internal/api/outbound`
- [Message fan-out](https://github.com/vivekagent47/dstream/blob/main/internal/webhook/handler.go): `internal/webhook/handler.go`
- [Application records](https://github.com/vivekagent47/dstream/blob/main/internal/store/applications.sql.go): `internal/store/applications.sql.go`

### API and identity

- [HTTP API router](https://github.com/vivekagent47/dstream/blob/main/internal/api/router.go): `internal/api/router.go`
- [Pipeline management](https://github.com/vivekagent47/dstream/tree/main/internal/api/pipeline): `internal/api/pipeline`
- [Identity and orgs](https://github.com/vivekagent47/dstream/tree/main/internal/api/identity): `internal/api/identity`
- [Authentication](https://github.com/vivekagent47/dstream/tree/main/internal/auth): `internal/auth`
- [Operations console](https://github.com/vivekagent47/dstream/blob/main/internal/admin/router.go): `internal/admin/router.go`
- [Metrics collector](https://github.com/vivekagent47/dstream/blob/main/internal/metrics/collector.go): `internal/metrics/collector.go`

### Dashboard

- [Dashboard routes](https://github.com/vivekagent47/dstream/blob/main/web/src/routeTree.gen.ts): `web/src/routeTree.gen.ts`
- [Dashboard API client](https://github.com/vivekagent47/dstream/blob/main/web/src/lib/api.ts): `web/src/lib/api.ts`
- [Customer portal](https://github.com/vivekagent47/dstream/blob/main/web/src/routes/portal/route.tsx): `web/src/routes/portal/route.tsx`

### Other components

- **Webhook sender**
- **Webhook endpoint**
- **PostgreSQL**
- [dstream CLI](https://github.com/vivekagent47/dstream/blob/main/cmd/dstream/cli.go): `cmd/dstream/cli.go`
- **Local app**

## Connections

- Webhook sender → Webhook ingest: posts webhook
- Webhook ingest → Event store: records request/events (evidence: `internal/ingest/handler.go`)
- Webhook ingest → Redis queues: enqueues delivery
- Delivery worker → Redis queues: consumes jobs (evidence: `cmd/dstream/worker.go`)
- Delivery worker → Connection delivery: processes deliveries (evidence: `cmd/dstream/worker.go`)
- Connection delivery → Filter engine: filters payload (evidence: `internal/deliver/handler.go`)
- Connection delivery → Transform engine: transforms payload (evidence: `internal/deliver/handler.go`)
- Connection delivery → Webhook endpoint: sends webhook
- HTTP API router → Pipeline management: routes pipeline (evidence: `internal/api/router.go`)
- HTTP API router → Identity and orgs: routes identity (evidence: `internal/api/router.go`)
- HTTP API router → Publish and portal API: routes outbound (evidence: `internal/api/router.go`)
- HTTP API router → Authentication: applies auth (evidence: `internal/api/router.go`)
- Identity and orgs → Authentication: uses auth services (evidence: `internal/api/identity/handlers.go`)
- Pipeline management → Redis queues: queues tests/replay (evidence: `internal/api/pipeline/handlers.go`)
- Pipeline management → Event store: manages pipeline data (evidence: `internal/api/pipeline/handlers.go`)
- Publish and portal API → Application records: stores applications (evidence: `internal/api/outbound/applications.go`)
- Message fan-out → Redis queues: schedules deliveries (evidence: `internal/webhook/handler.go`)
- Message fan-out → Filter engine: filters messages (evidence: `internal/webhook/handler.go`)
- Message fan-out → Transform engine: transforms messages (evidence: `internal/webhook/handler.go`)
- Dashboard routes → Dashboard API client: requests API
- Dashboard API client → HTTP API router: calls endpoints
- Customer portal → HTTP API router: uses portal API
- Operations console → Redis queues: inspects queues (evidence: `internal/admin/router.go`)
- Metrics collector → PostgreSQL: queries metrics (evidence: `internal/metrics/collector.go`)
- Delivery worker → PostgreSQL: loads delivery state
- Webhook ingest → PostgreSQL: persists events
- dstream CLI → HTTP API router: opens tunnel
- dstream CLI → Local app: forwards requests
