# pixelzzz99/vantapipe architecture

> Architecture diagram of the GitHub repository pixelzzz99/vantapipe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pixelzzz99/vantapipe
- Repository: https://github.com/pixelzzz99/vantapipe

## Overview

dataflow-rs is a configurable ETL service: a Node.js gateway exposes the dashboard and API, while a Rust engine loads pipeline definitions, schedules runs, extracts rows, transforms them, and loads results. The graph follows the documented request and ETL paths, with state, history, reload, and operational controls represented as supporting subsystems. Some implementation details are unsampled; edges involving those areas rely only on README statements or the source index, and uncertain relationships are omitted.

Main entry points: [`src/runtime.rs`](https://github.com/pixelzzz99/vantapipe/blob/main/src/runtime.rs) (Pipeline builder), [`src/scheduler.rs`](https://github.com/pixelzzz99/vantapipe/blob/main/src/scheduler.rs) (Pipeline scheduler), [`src/pipeline.rs`](https://github.com/pixelzzz99/vantapipe/blob/main/src/pipeline.rs) (ETL pipeline), [`src/main.rs`](https://github.com/pixelzzz99/vantapipe/blob/main/src/main.rs) (Engine startup)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_gateway["Public gateway"]
  node_gateway_app["Gateway server<br/>[index.js]"]
  node_auth["Basic auth<br/>[auth.js]"]
  node_proxy["Engine proxy<br/>[proxy.js]"]
  node_dashboard["Dashboard UI<br/>[dashboard.html]"]
end

subgraph group_engine["Engine control"]
  node_engine_api["Engine HTTP / WS API<br/>[mod.rs]"]
  node_api_handlers["API handlers<br/>[handlers.rs]"]
  node_live_logs["Live log stream<br/>[ws.rs]"]
  node_startup["Engine startup<br/>[main.rs]"]
  node_config["Config validation<br/>[config.rs]"]
  node_registry["Pipeline registry<br/>[registry.rs]"]
  node_watcher["Config file watcher<br/>[watcher.rs]"]
  node_scheduler["Pipeline scheduler<br/>[scheduler.rs]"]
  node_runtime["Pipeline builder<br/>[runtime.rs]"]
end

subgraph group_etl["ETL execution"]
  node_pipeline["ETL pipeline<br/>[pipeline.rs]"]
  node_extractor["Extractor interface<br/>[mod.rs]"]
  node_transforms["Transform implementations<br/>[aggregator.rs]"]
end

subgraph group_connectors["Data connectors"]
  node_postgres_source["Postgres extractor<br/>[postgres.rs]"]
  node_clickhouse_source["ClickHouse extractor<br/>[clickhouse.rs]"]
  node_csv_source["CSV watcher extractor<br/>[csv.rs]"]
  node_postgres_loader["Postgres destination<br/>[postgres.rs]"]
end

subgraph group_ops["State and observability"]
  node_persistent_state[("Incremental state<br/>[state.rs]")]
  node_history[("Run history<br/>[history.rs]")]
  node_retry["Retry policy<br/>[retry.rs]"]
end

node_browser(("Browser / user"))

node_browser -->|"uses"| node_gateway_app
node_gateway_app -->|"authenticates"| node_auth
node_gateway_app -->|"proxies requests"| node_proxy
node_gateway_app -->|"serves dashboard"| node_dashboard
node_proxy -->|"forwards requests"| node_engine_api
node_engine_api -->|"routes API calls"| node_api_handlers
node_engine_api -->|"routes log stream"| node_live_logs
node_startup -->|"loads configs"| node_config
node_startup -->|"registers pipelines"| node_registry
node_startup -->|"starts workers"| node_scheduler
node_startup -->|"starts API"| node_engine_api
node_api_handlers -->|"sends controls"| node_scheduler
node_api_handlers -->|"registers pipelines"| node_registry
node_watcher -->|"registers new configs"| node_registry
node_watcher -->|"requests reload"| node_scheduler
node_scheduler -->|"rebuilds pipeline"| node_runtime
node_scheduler -->|"runs pipeline"| node_pipeline
node_scheduler -->|"retries failures"| node_retry
node_scheduler -->|"records runs"| node_history
node_runtime -->|"loads state"| node_persistent_state
node_runtime -->|"builds pipeline"| node_pipeline
node_runtime -->|"configures source"| node_postgres_source
node_runtime -->|"configures source"| node_clickhouse_source
node_runtime -->|"configures source"| node_csv_source
node_runtime -->|"configures destination"| node_postgres_loader
node_pipeline -->|"extracts rows"| node_extractor
node_pipeline -->|"transforms rows"| node_transforms
node_pipeline -->|"loads rows"| node_postgres_loader
node_csv_source -->|"tracks files"| node_persistent_state

click node_gateway_app "https://github.com/pixelzzz99/vantapipe/blob/main/server/src/index.js"
click node_auth "https://github.com/pixelzzz99/vantapipe/blob/main/server/src/auth.js"
click node_proxy "https://github.com/pixelzzz99/vantapipe/blob/main/server/src/proxy.js"
click node_dashboard "https://github.com/pixelzzz99/vantapipe/blob/main/server/public/dashboard.html"
click node_engine_api "https://github.com/pixelzzz99/vantapipe/blob/main/src/web/mod.rs"
click node_api_handlers "https://github.com/pixelzzz99/vantapipe/blob/main/src/web/handlers.rs"
click node_live_logs "https://github.com/pixelzzz99/vantapipe/blob/main/src/web/ws.rs"
click node_startup "https://github.com/pixelzzz99/vantapipe/blob/main/src/main.rs"
click node_config "https://github.com/pixelzzz99/vantapipe/blob/main/src/config.rs"
click node_registry "https://github.com/pixelzzz99/vantapipe/blob/main/src/registry.rs"
click node_watcher "https://github.com/pixelzzz99/vantapipe/blob/main/src/watcher.rs"
click node_scheduler "https://github.com/pixelzzz99/vantapipe/blob/main/src/scheduler.rs"
click node_runtime "https://github.com/pixelzzz99/vantapipe/blob/main/src/runtime.rs"
click node_pipeline "https://github.com/pixelzzz99/vantapipe/blob/main/src/pipeline.rs"
click node_extractor "https://github.com/pixelzzz99/vantapipe/blob/main/src/extractor/mod.rs"
click node_transforms "https://github.com/pixelzzz99/vantapipe/blob/main/src/transformer/aggregator.rs"
click node_postgres_source "https://github.com/pixelzzz99/vantapipe/blob/main/src/extractor/postgres.rs"
click node_clickhouse_source "https://github.com/pixelzzz99/vantapipe/blob/main/src/extractor/clickhouse.rs"
click node_csv_source "https://github.com/pixelzzz99/vantapipe/blob/main/src/extractor/csv.rs"
click node_postgres_loader "https://github.com/pixelzzz99/vantapipe/blob/main/src/loader/postgres.rs"
click node_persistent_state "https://github.com/pixelzzz99/vantapipe/blob/main/src/state.rs"
click node_history "https://github.com/pixelzzz99/vantapipe/blob/main/src/history.rs"
click node_retry "https://github.com/pixelzzz99/vantapipe/blob/main/src/retry.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gateway_app,node_auth,node_proxy,node_dashboard,node_browser toneBlue
class node_engine_api,node_api_handlers,node_live_logs,node_startup,node_config,node_registry,node_watcher,node_scheduler,node_runtime toneAmber
class node_pipeline,node_extractor,node_transforms toneMint
class node_postgres_source,node_clickhouse_source,node_csv_source,node_postgres_loader toneRose
class node_persistent_state,node_history,node_retry toneIndigo
```

## Components

### Public gateway

- [Gateway server](https://github.com/pixelzzz99/vantapipe/blob/main/server/src/index.js): `server/src/index.js`
- [Basic auth](https://github.com/pixelzzz99/vantapipe/blob/main/server/src/auth.js): `server/src/auth.js`
- [Engine proxy](https://github.com/pixelzzz99/vantapipe/blob/main/server/src/proxy.js): `server/src/proxy.js`
- [Dashboard UI](https://github.com/pixelzzz99/vantapipe/blob/main/server/public/dashboard.html): `server/public/dashboard.html`

### Engine control

- [Engine HTTP / WS API](https://github.com/pixelzzz99/vantapipe/blob/main/src/web/mod.rs): `src/web/mod.rs`
- [API handlers](https://github.com/pixelzzz99/vantapipe/blob/main/src/web/handlers.rs): `src/web/handlers.rs`
- [Live log stream](https://github.com/pixelzzz99/vantapipe/blob/main/src/web/ws.rs): `src/web/ws.rs`
- [Engine startup](https://github.com/pixelzzz99/vantapipe/blob/main/src/main.rs): `src/main.rs`
- [Config validation](https://github.com/pixelzzz99/vantapipe/blob/main/src/config.rs): `src/config.rs`
- [Pipeline registry](https://github.com/pixelzzz99/vantapipe/blob/main/src/registry.rs): `src/registry.rs`
- [Config file watcher](https://github.com/pixelzzz99/vantapipe/blob/main/src/watcher.rs): `src/watcher.rs`
- [Pipeline scheduler](https://github.com/pixelzzz99/vantapipe/blob/main/src/scheduler.rs): `src/scheduler.rs`
- [Pipeline builder](https://github.com/pixelzzz99/vantapipe/blob/main/src/runtime.rs): `src/runtime.rs`

### ETL execution

- [ETL pipeline](https://github.com/pixelzzz99/vantapipe/blob/main/src/pipeline.rs): `src/pipeline.rs`
- [Extractor interface](https://github.com/pixelzzz99/vantapipe/blob/main/src/extractor/mod.rs): `src/extractor/mod.rs`
- [Transform implementations](https://github.com/pixelzzz99/vantapipe/blob/main/src/transformer/aggregator.rs): `src/transformer/aggregator.rs`

### Data connectors

- [Postgres extractor](https://github.com/pixelzzz99/vantapipe/blob/main/src/extractor/postgres.rs): `src/extractor/postgres.rs`
- [ClickHouse extractor](https://github.com/pixelzzz99/vantapipe/blob/main/src/extractor/clickhouse.rs): `src/extractor/clickhouse.rs`
- [CSV watcher extractor](https://github.com/pixelzzz99/vantapipe/blob/main/src/extractor/csv.rs): `src/extractor/csv.rs`
- [Postgres destination](https://github.com/pixelzzz99/vantapipe/blob/main/src/loader/postgres.rs): `src/loader/postgres.rs`

### State and observability

- [Incremental state](https://github.com/pixelzzz99/vantapipe/blob/main/src/state.rs): `src/state.rs`
- [Run history](https://github.com/pixelzzz99/vantapipe/blob/main/src/history.rs): `src/history.rs`
- [Retry policy](https://github.com/pixelzzz99/vantapipe/blob/main/src/retry.rs): `src/retry.rs`

### Other components

- **Browser / user**

## Connections

- Browser / user → Gateway server: uses (evidence: `README.md`)
- Gateway server → Basic auth: authenticates (evidence: `server/src/index.js`)
- Gateway server → Engine proxy: proxies requests (evidence: `server/src/index.js`)
- Gateway server → Dashboard UI: serves dashboard (evidence: `README.md`)
- Engine proxy → Engine HTTP / WS API: forwards requests (evidence: `README.md`)
- Engine HTTP / WS API → API handlers: routes API calls (evidence: `src/web/mod.rs`)
- Engine HTTP / WS API → Live log stream: routes log stream (evidence: `src/web/mod.rs`)
- Engine startup → Config validation: loads configs (evidence: `src/main.rs`)
- Engine startup → Pipeline registry: registers pipelines (evidence: `src/main.rs`)
- Engine startup → Pipeline scheduler: starts workers (evidence: `src/main.rs`)
- Engine startup → Engine HTTP / WS API: starts API (evidence: `src/main.rs`)
- API handlers → Pipeline scheduler: sends controls (evidence: `src/web/handlers.rs`)
- API handlers → Pipeline registry: registers pipelines (evidence: `src/registry.rs`)
- Config file watcher → Pipeline registry: registers new configs (evidence: `src/watcher.rs`)
- Config file watcher → Pipeline scheduler: requests reload (evidence: `src/watcher.rs`)
- Pipeline scheduler → Pipeline builder: rebuilds pipeline (evidence: `src/scheduler.rs`)
- Pipeline scheduler → ETL pipeline: runs pipeline (evidence: `src/scheduler.rs`)
- Pipeline scheduler → Retry policy: retries failures (evidence: `src/scheduler.rs`)
- Pipeline scheduler → Run history: records runs (evidence: `src/scheduler.rs`)
- Pipeline builder → Incremental state: loads state (evidence: `src/runtime.rs`)
- Pipeline builder → ETL pipeline: builds pipeline (evidence: `src/runtime.rs`)
- Pipeline builder → Postgres extractor: configures source (evidence: `src/runtime.rs`)
- Pipeline builder → ClickHouse extractor: configures source (evidence: `src/runtime.rs`)
- Pipeline builder → CSV watcher extractor: configures source (evidence: `src/runtime.rs`)
- Pipeline builder → Postgres destination: configures destination (evidence: `src/runtime.rs`)
- ETL pipeline → Extractor interface: extracts rows (evidence: `src/pipeline.rs`)
- ETL pipeline → Transform implementations: transforms rows (evidence: `src/pipeline.rs`)
- ETL pipeline → Postgres destination: loads rows (evidence: `src/pipeline.rs`)
- CSV watcher extractor → Incremental state: tracks files (evidence: `src/extractor/csv.rs`)
