How pixelzzz99/vantapipe works
dataflow-rs is a configurable ETL service: a Node.js gateway exposes the dashboard and API, while a Rust engine loads pipeline definitions, schedules runs, extracts rows, transforms them, and loads results. The graph follows the documented request and ETL paths, with state, history, reload, and operational controls represented as supporting subsystems. Some implementation details are unsampled; edges involving those areas rely only on README statements or the source index, and uncertain relationships are omitted.
Main entry points: runtime.rs Pipeline builder · scheduler.rs Pipeline scheduler · pipeline.rs ETL pipeline · main.rs Engine startup
Components 24
Public gateway
- Gateway server
server/src/index.js - Basic auth
server/src/auth.js - Engine proxy
server/src/proxy.js - Dashboard UI
server/public/dashboard.html
Engine control
- Engine HTTP / WS API
src/web/mod.rs - API handlers
src/web/handlers.rs - Live log stream
src/web/ws.rs - Engine startup
src/main.rs - Config validation
src/config.rs - Pipeline registry
src/registry.rs - Config file watcher
src/watcher.rs - Pipeline scheduler
src/scheduler.rs - Pipeline builder
src/runtime.rs
ETL execution
- ETL pipeline
src/pipeline.rs - Extractor interface
src/extractor/mod.rs - Transform implementations
src/transformer/aggregator.rs
Data connectors
- Postgres extractor
src/extractor/postgres.rs - ClickHouse extractor
src/extractor/clickhouse.rs - CSV watcher extractor
src/extractor/csv.rs - Postgres destination
src/loader/postgres.rs
State and observability
- Incremental state
src/state.rs - Run history
src/history.rs - Retry policy
src/retry.rs
Other components
- Browser / user
Connections 29
- Browser / user → Gateway server: uses
README.md - Gateway server → Basic auth: authenticates
server/src/index.js - Gateway server → Engine proxy: proxies requests
server/src/index.js - Gateway server → Dashboard UI: serves dashboard
README.md - Engine proxy → Engine HTTP / WS API: forwards requests
README.md - Engine HTTP / WS API → API handlers: routes API calls
src/web/mod.rs - Engine HTTP / WS API → Live log stream: routes log stream
src/web/mod.rs - Engine startup → Config validation: loads configs
src/main.rs - Engine startup → Pipeline registry: registers pipelines
src/main.rs - Engine startup → Pipeline scheduler: starts workers
src/main.rs - Engine startup → Engine HTTP / WS API: starts API
src/main.rs - API handlers → Pipeline scheduler: sends controls
src/web/handlers.rs - API handlers → Pipeline registry: registers pipelines
src/registry.rs - Config file watcher → Pipeline registry: registers new configs
src/watcher.rs - Config file watcher → Pipeline scheduler: requests reload
src/watcher.rs - Pipeline scheduler → Pipeline builder: rebuilds pipeline
src/scheduler.rs - Pipeline scheduler → ETL pipeline: runs pipeline
src/scheduler.rs - Pipeline scheduler → Retry policy: retries failures
src/scheduler.rs - Pipeline scheduler → Run history: records runs
src/scheduler.rs - Pipeline builder → Incremental state: loads state
src/runtime.rs - Pipeline builder → ETL pipeline: builds pipeline
src/runtime.rs - Pipeline builder → Postgres extractor: configures source
src/runtime.rs - Pipeline builder → ClickHouse extractor: configures source
src/runtime.rs - Pipeline builder → CSV watcher extractor: configures source
src/runtime.rs - Pipeline builder → Postgres destination: configures destination
src/runtime.rs - ETL pipeline → Extractor interface: extracts rows
src/pipeline.rs - ETL pipeline → Transform implementations: transforms rows
src/pipeline.rs - ETL pipeline → Postgres destination: loads rows
src/pipeline.rs - CSV watcher extractor → Incremental state: tracks files
src/extractor/csv.rs