# runabol/tork architecture

> Architecture diagram of the GitHub repository runabol/tork, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/runabol/tork
- Repository: https://github.com/runabol/tork

## Overview

Based on our analysis of the file tree and the README, “Tork” is a fully featured, highly scalable workflow engine. It’s designed to manage and execute workflows (jobs and tasks) in both stand-alone and distributed setups. The project is written in Go and is composed of several well-structured packages that together implement a rich set of features including a REST API, CLI, containerized task isolation (via Docker, Podman, and Shell runtimes), integrated brokers (supporting in-memory and RabbitMQ), middleware extensions, and datastore support (using PostgreSQL).

- Frontend/Client Layer: Although the Web UI is hosted elsewhere, note that Tork exposes a REST API that the Web UI (or any client) can use.
- CLI Interface: Represent the “cli” package as a user-facing component that initiates various commands (run, health, migrate, etc.).
- API Gateway/REST Layer: Mark the REST API endpoint which sits along with the CLI commands to expose functionality.
- Core Workflow Engine:
- “engine” package – the central orchestration layer handling job coordination, scheduling, and execution.
- “internal/coordinator” – where the workflow coordination and API handlers live.
- “internal/worker” – which manages task execution.
- “internal/eval”, “internal/host”, and similar packages that support evaluation of expressions, host management etc.
- Broker System:
- “broker” package – supports messaging through both in-memory and RabbitMQ implementations.
- Datastore:
- “datastore” and “db/postgres” directories – for persistence, particularly storing job/task state within a PostgreSQL database.
- Middleware:
- “middleware” with distinct subdirectories (job, task, node, web) – illustrates the extensible aspects used for adding functionalities like logging, redaction, and webhook handling.
- Runtime Components:
- “runtime” package – handles execution environments. This includes Docker, Podman, and Shell-based runtimes. These modules are responsible for providing isolation and executing tasks in containers.
- Utilities and Internal Libraries:
- “internal” directories for caching, synchronization, UUID generation, and error management that support the core engine.
- Configuration and Infrastructure:
- “conf” and “configs” – provide configuration management and sample setup files.
- Docker Compose and Makefiles – indicate CI/CD integrations and deployment as well as orchestration details.

##### Relationships and Data Flow
- Internal data flows within the workflow engine such as:
- API → Coordinator → Scheduler and Worker.
- Workers, via middleware and runtime layers, execute tasks and then report back status to the engine.
- The Broker communicates between various modules to handle messaging and buffering of events.
- Persistence flows: tasks and job statuses are stored into and read from the Datastore (Postgres).
- External dependencies (such as RabbitMQ) are specifically noted connecting to the Broker.

##### Architectural Patterns
- Modular and layered structure of the application which encourages clean separation of concerns.
- Extensibility provided by the middleware architecture.
- Support for different runtime environments which provides flexibility and isolation for task execution.
- Coordination patterns between schedulers and workers for achieving horizontal scalability and fault tolerance.

##### Technologies and Frameworks
- Go as the programming language.
- PostgreSQL for data persistence.
- RabbitMQ (along with an in-memory alternative) as the messaging broker.
- Docker, Podman, and shell execution environments.
- GitHub Actions workflows (as seen in the .github directory) to illustrate CI/CD integration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        RESTAPI["REST API Endpoint"]:::client
        CLI["CLI Interface"]:::client
    end

    %% Orchestration/Engine Layer
    subgraph "Orchestration Engine Layer"
        ENGINE["Engine Orchestration"]:::engine
        COORD["Workflow Coordinator"]:::engine
        WORKER["Task Worker"]:::engine
        EVALHOST["Expression & Host Managers"]:::engine
        UTIL["Internal Utilities & Libraries"]:::utility
    end

    %% Integration / Middleware Layer
    subgraph "Integration / Middleware Layer"
        BROKER["Message Broker"]:::integration
        M_JOB["Middleware - Job"]:::integration
        M_TASK["Middleware - Task"]:::integration
        M_NODE["Middleware - Node"]:::integration
        M_WEB["Middleware - Web"]:::integration
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        DB["PostgreSQL Datastore"]:::infra
        RUNTIME["Runtime Environments"]:::infra
        DOCKER["Docker Runtime"]:::infra
        PODMAN["Podman Runtime"]:::infra
        SHELL["Shell Runtime"]:::infra
        CONFIG["Configuration Files"]:::infra
        CI["CI/CD & Deployment"]:::infra
    end

    %% Data Flow Connections
    RESTAPI -->|"calls"| COORD
    CLI -->|"issues_command"| ENGINE

    COORD -->|"schedules"| ENGINE
    COORD -->|"delegates_task"| WORKER
    ENGINE -->|"manages"| WORKER
    ENGINE -->|"utilizes"| UTIL
    WORKER -->|"reports_status"| ENGINE

    WORKER -->|"executes_in"| RUNTIME
    RUNTIME -->|"via_docker"| DOCKER
    RUNTIME -->|"via_podman"| PODMAN
    RUNTIME -->|"via_shell"| SHELL

    ENGINE -->|"publishes_events"| BROKER
    BROKER -->|"activates"| M_JOB
    BROKER --> M_TASK
    BROKER --> M_NODE
    BROKER --> M_WEB

    ENGINE -->|"persists_state"| DB
    COORD -->|"updates_state"| DB

    %% Click Events for Client Layer
    click RESTAPI "https://github.com/runabol/tork/tree/main/internal/coordinator/api"
    click RESTAPI "https://github.com/runabol/tork/blob/main/internal/httpx/httpx.go"
    click CLI "https://github.com/runabol/tork/tree/main/cli"

    %% Click Events for Orchestration/Engine Layer
    click ENGINE "https://github.com/runabol/tork/tree/main/engine"
    click COORD "https://github.com/runabol/tork/tree/main/internal/coordinator"
    click WORKER "https://github.com/runabol/tork/tree/main/internal/worker"
    click EVALHOST "https://github.com/runabol/tork/tree/main/internal/eval"
    click EVALHOST "https://github.com/runabol/tork/tree/main/internal/host"
    click UTIL "https://github.com/runabol/tork/tree/main/internal"

    %% Click Events for Integration / Middleware Layer
    click BROKER "https://github.com/runabol/tork/tree/main/broker"
    click M_JOB "https://github.com/runabol/tork/tree/main/middleware/job"
    click M_TASK "https://github.com/runabol/tork/tree/main/middleware/task"
    click M_NODE "https://github.com/runabol/tork/tree/main/middleware/node"
    click M_WEB "https://github.com/runabol/tork/tree/main/middleware/web"

    %% Click Events for Infrastructure Layer
    click DB "https://github.com/runabol/tork/tree/main/datastore"
    click DB "https://github.com/runabol/tork/tree/main/datastore/postgres"
    click DB "https://github.com/runabol/tork/tree/main/db/postgres"
    click RUNTIME "https://github.com/runabol/tork/tree/main/runtime"
    click DOCKER "https://github.com/runabol/tork/tree/main/runtime/docker"
    click PODMAN "https://github.com/runabol/tork/tree/main/runtime/podman"
    click SHELL "https://github.com/runabol/tork/tree/main/runtime/shell"
    click CONFIG "https://github.com/runabol/tork/tree/main/conf"
    click CONFIG "https://github.com/runabol/tork/tree/main/configs"
    click CI "https://github.com/runabol/tork/blob/main/.github"
    click CI "https://github.com/runabol/tork/blob/main/docker-compose.yaml"
    click CI "https://github.com/runabol/tork/tree/main/Makefile"
    click CI "https://github.com/runabol/tork/blob/main/Makefile.release"

    %% Class Styles
    classDef client fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef engine fill:#ABEBC6,stroke:#1E8449,stroke-width:2px;
    classDef integration fill:#F5CBA7,stroke:#9C640C,stroke-width:2px;
    classDef infra fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef utility fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
```
