# rstar-910/postgres-workflow-engine architecture

> Architecture diagram of the GitHub repository rstar-910/postgres-workflow-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rstar-910/postgres-workflow-engine
- Repository: https://github.com/rstar-910/postgres-workflow-engine

## Overview

FlowForge is a durable PostgreSQL-backed workflow engine. A client defines a versioned task DAG and submits a run through the REST API; workers claim and execute eligible tasks, then commit outcomes that advance dependencies and finalize the run. The scheduler handles expired leases, due retries, deadlines, and retention. PostgreSQL is the shared source of truth. The README documents dashboard and observability capabilities; some links below rely on that documentation or the SOURCE INDEX rather than sampled implementation excerpts.

Main entry points: [`src/flowforge/api/routes.py`](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/api/routes.py) (REST API), [`src/flowforge/worker/runtime.py`](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/worker/runtime.py) (Worker runtime), [`src/flowforge/scheduler/service.py`](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/scheduler/service.py) (Scheduler service), [`src/flowforge/db/session.py`](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/db/session.py) (Database sessions)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API and dashboard"]
  node_dashboard["Dashboard UI<br/>[app.js]"]
  node_api["REST API<br/>[routes.py]"]
  node_auth["API authentication<br/>[auth.py]"]
  node_schemas["API schemas<br/>[schemas.py]"]
end

subgraph group_workflow["Workflow lifecycle"]
  node_definitions["Definition validation<br/>[definition.py]"]
  node_retry["Retry policy<br/>[retry.py]"]
  node_states["State machines<br/>[states.py]"]
  node_submission["Run submission<br/>[submission.py]"]
  node_queries["Run queries<br/>[queries.py]"]
  node_control["Run control<br/>[control.py]"]
  node_progression["DAG progression<br/>[progression.py]"]
end

subgraph group_execution["Workers and scheduling"]
  node_execution["Task execution engine<br/>[execution.py]"]
  node_worker["Worker runtime<br/>[runtime.py]"]
  node_executor["Handler executor<br/>[executor.py]"]
  node_registry["Handler registry<br/>[registry.py]"]
  node_context["Task context<br/>[context.py]"]
  node_scheduler["Scheduler service<br/>[service.py]"]
  node_recovery["Lease recovery<br/>[recovery.py]"]
  node_timeouts["Deadline timeouts<br/>[timeouts.py]"]
  node_retention["Data retention<br/>[retention.py]"]
end

subgraph group_storage["Persistence"]
  node_postgres[("PostgreSQL<br/>[models.py]")]
  node_sql["SQL helpers<br/>[sql.py]"]
  node_dbsession["Database sessions<br/>[session.py]"]
end

subgraph group_ops["Configuration and observability"]
  node_config["Process configuration<br/>[config.py]"]
  node_logging["Structured logging<br/>[logging.py]"]
  node_metrics["Prometheus metrics<br/>[metrics.py]"]
  node_tracing["OpenTelemetry tracing<br/>[tracing.py]"]
end

node_client(("API client"))

node_client -->|"uses REST"| node_api
node_dashboard -->|"uses REST"| node_api
node_api -->|"authenticates"| node_auth
node_api -->|"validates requests"| node_schemas
node_api -->|"validates definitions"| node_definitions
node_api -->|"submits runs"| node_submission
node_api -->|"queries runs"| node_queries
node_api -->|"controls runs"| node_control
node_submission -->|"validates DAG"| node_definitions
node_submission -->|"uses SQL helpers"| node_sql
node_queries -->|"uses SQL helpers"| node_sql
node_control -->|"advances runs"| node_progression
node_control -->|"retries tasks"| node_submission
node_worker -->|"claims and commits"| node_execution
node_worker -->|"runs handlers"| node_executor
node_executor -->|"resolves handlers"| node_registry
node_executor -->|"builds task context"| node_context
node_execution -->|"advances dependencies"| node_progression
node_execution -->|"uses SQL helpers"| node_sql
node_scheduler -->|"recovers leases"| node_recovery
node_scheduler -->|"checks deadlines"| node_timeouts
node_scheduler -->|"prunes history"| node_retention
node_scheduler -->|"opens sessions"| node_dbsession
node_api -->|"opens sessions"| node_dbsession
node_worker -->|"opens sessions"| node_dbsession
node_api -->|"reads and writes"| node_postgres
node_worker -->|"claims and commits"| node_postgres
node_scheduler -->|"recovers and prunes"| node_postgres
node_sql -->|"uses table models"| node_postgres
node_dbsession -->|"reads settings"| node_config
node_api -->|"loads settings"| node_config
node_worker -->|"loads settings"| node_config
node_scheduler -->|"loads settings"| node_config
node_dbsession -->|"records query timing"| node_metrics
node_api -->|"emits logs"| node_logging
node_api -->|"records HTTP metrics"| node_metrics
node_api -->|"propagates trace context"| node_tracing
node_worker -->|"emits correlated logs"| node_logging
node_worker -->|"records execution metrics"| node_metrics
node_worker -->|"propagates trace context"| node_tracing

click node_dashboard "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/api/static/app.js"
click node_api "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/api/routes.py"
click node_auth "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/api/auth.py"
click node_schemas "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/api/schemas.py"
click node_definitions "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/domain/definition.py"
click node_retry "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/domain/retry.py"
click node_states "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/domain/states.py"
click node_submission "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/submission.py"
click node_queries "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/queries.py"
click node_control "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/control.py"
click node_progression "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/progression.py"
click node_execution "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/execution.py"
click node_worker "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/worker/runtime.py"
click node_executor "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/worker/executor.py"
click node_registry "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/worker/registry.py"
click node_context "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/worker/context.py"
click node_scheduler "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/scheduler/service.py"
click node_recovery "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/recovery.py"
click node_timeouts "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/timeouts.py"
click node_retention "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/retention.py"
click node_postgres "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/db/models.py"
click node_sql "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/sql.py"
click node_dbsession "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/db/session.py"
click node_config "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/config.py"
click node_logging "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/observability/logging.py"
click node_metrics "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/observability/metrics.py"
click node_tracing "https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/observability/tracing.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard,node_api,node_auth,node_schemas,node_client toneBlue
class node_definitions,node_retry,node_states,node_submission,node_queries,node_control,node_progression toneAmber
class node_execution,node_worker,node_executor,node_registry,node_context,node_scheduler,node_recovery,node_timeouts,node_retention toneMint
class node_postgres,node_sql,node_dbsession toneRose
class node_config,node_logging,node_metrics,node_tracing toneIndigo
```

## Components

### API and dashboard

- [Dashboard UI](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/api/static/app.js): `src/flowforge/api/static/app.js`
- [REST API](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/api/routes.py): `src/flowforge/api/routes.py`
- [API authentication](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/api/auth.py): `src/flowforge/api/auth.py`
- [API schemas](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/api/schemas.py): `src/flowforge/api/schemas.py`

### Workflow lifecycle

- [Definition validation](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/domain/definition.py): `src/flowforge/domain/definition.py`
- [Retry policy](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/domain/retry.py): `src/flowforge/domain/retry.py`
- [State machines](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/domain/states.py): `src/flowforge/domain/states.py`
- [Run submission](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/submission.py): `src/flowforge/engine/submission.py`
- [Run queries](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/queries.py): `src/flowforge/engine/queries.py`
- [Run control](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/control.py): `src/flowforge/engine/control.py`
- [DAG progression](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/progression.py): `src/flowforge/engine/progression.py`

### Workers and scheduling

- [Task execution engine](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/execution.py): `src/flowforge/engine/execution.py`
- [Worker runtime](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/worker/runtime.py): `src/flowforge/worker/runtime.py`
- [Handler executor](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/worker/executor.py): `src/flowforge/worker/executor.py`
- [Handler registry](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/worker/registry.py): `src/flowforge/worker/registry.py`
- [Task context](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/worker/context.py): `src/flowforge/worker/context.py`
- [Scheduler service](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/scheduler/service.py): `src/flowforge/scheduler/service.py`
- [Lease recovery](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/recovery.py): `src/flowforge/engine/recovery.py`
- [Deadline timeouts](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/timeouts.py): `src/flowforge/engine/timeouts.py`
- [Data retention](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/retention.py): `src/flowforge/engine/retention.py`

### Persistence

- [PostgreSQL](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/db/models.py): `src/flowforge/db/models.py`
- [SQL helpers](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/engine/sql.py): `src/flowforge/engine/sql.py`
- [Database sessions](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/db/session.py): `src/flowforge/db/session.py`

### Configuration and observability

- [Process configuration](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/config.py): `src/flowforge/config.py`
- [Structured logging](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/observability/logging.py): `src/flowforge/observability/logging.py`
- [Prometheus metrics](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/observability/metrics.py): `src/flowforge/observability/metrics.py`
- [OpenTelemetry tracing](https://github.com/rstar-910/postgres-workflow-engine/blob/main/src/flowforge/observability/tracing.py): `src/flowforge/observability/tracing.py`

### Other components

- **API client**

## Connections

- API client → REST API: uses REST
- Dashboard UI → REST API: uses REST
- REST API → API authentication: authenticates (evidence: `src/flowforge/api/routes.py`)
- REST API → API schemas: validates requests (evidence: `src/flowforge/api/routes.py`)
- REST API → Definition validation: validates definitions (evidence: `src/flowforge/api/routes.py`)
- REST API → Run submission: submits runs (evidence: `src/flowforge/api/routes.py`)
- REST API → Run queries: queries runs (evidence: `src/flowforge/api/routes.py`)
- REST API → Run control: controls runs (evidence: `src/flowforge/api/routes.py`)
- Run submission → Definition validation: validates DAG (evidence: `src/flowforge/engine/submission.py`)
- Run submission → SQL helpers: uses SQL helpers (evidence: `src/flowforge/engine/submission.py`)
- Run queries → SQL helpers: uses SQL helpers (evidence: `src/flowforge/engine/queries.py`)
- Run control → DAG progression: advances runs (evidence: `src/flowforge/engine/control.py`)
- Run control → Run submission: retries tasks (evidence: `src/flowforge/engine/control.py`)
- Worker runtime → Task execution engine: claims and commits (evidence: `src/flowforge/worker/runtime.py`)
- Worker runtime → Handler executor: runs handlers (evidence: `src/flowforge/worker/runtime.py`)
- Handler executor → Handler registry: resolves handlers (evidence: `src/flowforge/worker/executor.py`)
- Handler executor → Task context: builds task context (evidence: `src/flowforge/worker/executor.py`)
- Task execution engine → DAG progression: advances dependencies (evidence: `src/flowforge/engine/execution.py`)
- Task execution engine → SQL helpers: uses SQL helpers (evidence: `src/flowforge/engine/execution.py`)
- Scheduler service → Lease recovery: recovers leases (evidence: `src/flowforge/scheduler/service.py`)
- Scheduler service → Deadline timeouts: checks deadlines (evidence: `src/flowforge/scheduler/service.py`)
- Scheduler service → Data retention: prunes history (evidence: `src/flowforge/scheduler/service.py`)
- Scheduler service → Database sessions: opens sessions (evidence: `src/flowforge/scheduler/service.py`)
- REST API → Database sessions: opens sessions (evidence: `src/flowforge/api/app.py`)
- Worker runtime → Database sessions: opens sessions (evidence: `src/flowforge/worker/runtime.py`)
- REST API → PostgreSQL: reads and writes
- Worker runtime → PostgreSQL: claims and commits
- Scheduler service → PostgreSQL: recovers and prunes
- SQL helpers → PostgreSQL: uses table models (evidence: `src/flowforge/engine/sql.py`)
- Database sessions → Process configuration: reads settings (evidence: `src/flowforge/db/session.py`)
- REST API → Process configuration: loads settings (evidence: `src/flowforge/api/app.py`)
- Worker runtime → Process configuration: loads settings (evidence: `src/flowforge/worker/runtime.py`)
- Scheduler service → Process configuration: loads settings (evidence: `src/flowforge/scheduler/service.py`)
- Database sessions → Prometheus metrics: records query timing (evidence: `src/flowforge/db/session.py`)
- REST API → Structured logging: emits logs
- REST API → Prometheus metrics: records HTTP metrics
- REST API → OpenTelemetry tracing: propagates trace context (evidence: `src/flowforge/api/routes.py`)
- Worker runtime → Structured logging: emits correlated logs (evidence: `src/flowforge/worker/runtime.py`)
- Worker runtime → Prometheus metrics: records execution metrics (evidence: `src/flowforge/worker/runtime.py`)
- Worker runtime → OpenTelemetry tracing: propagates trace context (evidence: `src/flowforge/worker/runtime.py`)
