How rstar-910/postgres-workflow-engine works
FlowForge is a durable PostgreSQL-backed workflow engine. A client defines a versioned task DAG and submits a run through the REST API; workers claim and execute eligible tasks, then commit outcomes that advance dependencies and finalize the run. The scheduler handles expired leases, due retries, deadlines, and retention. PostgreSQL is the shared source of truth. The README documents dashboard and observability capabilities; some links below rely on that documentation or the SOURCE INDEX rather than sampled implementation excerpts.
Main entry points: routes.py REST API · runtime.py Worker runtime · service.py Scheduler service · session.py Database sessions
Components 28
API and dashboard
- Dashboard UI
src/flowforge/api/static/app.js - REST API
src/flowforge/api/routes.py - API authentication
src/flowforge/api/auth.py - API schemas
src/flowforge/api/schemas.py
Workflow lifecycle
- Definition validation
src/flowforge/domain/definition.py - Retry policy
src/flowforge/domain/retry.py - State machines
src/flowforge/domain/states.py - Run submission
src/flowforge/engine/submission.py - Run queries
src/flowforge/engine/queries.py - Run control
src/flowforge/engine/control.py - DAG progression
src/flowforge/engine/progression.py
Workers and scheduling
- Task execution engine
src/flowforge/engine/execution.py - Worker runtime
src/flowforge/worker/runtime.py - Handler executor
src/flowforge/worker/executor.py - Handler registry
src/flowforge/worker/registry.py - Task context
src/flowforge/worker/context.py - Scheduler service
src/flowforge/scheduler/service.py - Lease recovery
src/flowforge/engine/recovery.py - Deadline timeouts
src/flowforge/engine/timeouts.py - Data retention
src/flowforge/engine/retention.py
Persistence
- PostgreSQL
src/flowforge/db/models.py - SQL helpers
src/flowforge/engine/sql.py - Database sessions
src/flowforge/db/session.py
Configuration and observability
- Process configuration
src/flowforge/config.py - Structured logging
src/flowforge/observability/logging.py - Prometheus metrics
src/flowforge/observability/metrics.py - OpenTelemetry tracing
src/flowforge/observability/tracing.py
Other components
- API client
Connections 40
- API client → REST API: uses REST
- Dashboard UI → REST API: uses REST
- REST API → API authentication: authenticates
src/flowforge/api/routes.py - REST API → API schemas: validates requests
src/flowforge/api/routes.py - REST API → Definition validation: validates definitions
src/flowforge/api/routes.py - REST API → Run submission: submits runs
src/flowforge/api/routes.py - REST API → Run queries: queries runs
src/flowforge/api/routes.py - REST API → Run control: controls runs
src/flowforge/api/routes.py - Run submission → Definition validation: validates DAG
src/flowforge/engine/submission.py - Run submission → SQL helpers: uses SQL helpers
src/flowforge/engine/submission.py - Run queries → SQL helpers: uses SQL helpers
src/flowforge/engine/queries.py - Run control → DAG progression: advances runs
src/flowforge/engine/control.py - Run control → Run submission: retries tasks
src/flowforge/engine/control.py - Worker runtime → Task execution engine: claims and commits
src/flowforge/worker/runtime.py - Worker runtime → Handler executor: runs handlers
src/flowforge/worker/runtime.py - Handler executor → Handler registry: resolves handlers
src/flowforge/worker/executor.py - Handler executor → Task context: builds task context
src/flowforge/worker/executor.py - Task execution engine → DAG progression: advances dependencies
src/flowforge/engine/execution.py - Task execution engine → SQL helpers: uses SQL helpers
src/flowforge/engine/execution.py - Scheduler service → Lease recovery: recovers leases
src/flowforge/scheduler/service.py - Scheduler service → Deadline timeouts: checks deadlines
src/flowforge/scheduler/service.py - Scheduler service → Data retention: prunes history
src/flowforge/scheduler/service.py - Scheduler service → Database sessions: opens sessions
src/flowforge/scheduler/service.py - REST API → Database sessions: opens sessions
src/flowforge/api/app.py - Worker runtime → Database sessions: opens sessions
src/flowforge/worker/runtime.py - REST API → PostgreSQL: reads and writes
- Worker runtime → PostgreSQL: claims and commits
- Scheduler service → PostgreSQL: recovers and prunes
- SQL helpers → PostgreSQL: uses table models
src/flowforge/engine/sql.py - Database sessions → Process configuration: reads settings
src/flowforge/db/session.py - REST API → Process configuration: loads settings
src/flowforge/api/app.py - Worker runtime → Process configuration: loads settings
src/flowforge/worker/runtime.py - Scheduler service → Process configuration: loads settings
src/flowforge/scheduler/service.py - Database sessions → Prometheus metrics: records query timing
src/flowforge/db/session.py - REST API → Structured logging: emits logs
- REST API → Prometheus metrics: records HTTP metrics
- REST API → OpenTelemetry tracing: propagates trace context
src/flowforge/api/routes.py - Worker runtime → Structured logging: emits correlated logs
src/flowforge/worker/runtime.py - Worker runtime → Prometheus metrics: records execution metrics
src/flowforge/worker/runtime.py - Worker runtime → OpenTelemetry tracing: propagates trace context
src/flowforge/worker/runtime.py