# hatchet-dev/hatchet architecture

> Architecture diagram of the GitHub repository hatchet-dev/hatchet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hatchet-dev/hatchet
- Repository: https://github.com/hatchet-dev/hatchet

## Overview

##### Main Components and Layers
- API / Control Plane:
– The “api”, “cmd/hatchet-api”, and “internal/services/grpc” folders manage the core API, request handling, authentication (authn/authz middleware), and GRPC endpoints.
– The API controller layer handles incoming requests from client applications.
- Worker Engine & Task Orchestration:
– The “cmd/hatchet-engine”, “internal/services/dispatcher”, “internal/services/scheduler”, “internal/services/ingestor”, and “pkg/worker” directories indicate that Hatchet has a dedicated worker engine.
– Modules for task routing, workflow orchestration (including DAG and durable executions), and flow control (e.g., concurrency and rate limits) reside here.
– The orchestration logic is built on top of the task factories that allow developers to register tasks and workflows.
- Persistence Layer:
– Postgres (with SQL schema migrations in the “sql/schema” folder and integration modules in “pkg/repository/postgres”) is the underpinning storage engine.
– Durable task queues and persistent storage of task histories are implemented in the repository and msgqueue layers.
- Integration and External Services:
– The “internal/integrations”, “pkg/integrations”, and service folders (for alerting, email, Slack, etc.) contain modules that integrate with third-party services.
– Message queue implementations for RabbitMQ and other protocols also exist under “internal/msgqueue”.
- SDKs and External Client Interfaces:
– There is comprehensive support for clients written in Python (under “sdks/python”), Typescript (under “sdks/typescript”) and references in Go.
– These SDKs encapsulate client calls to the API and include features such as task registration, event triggers, and orchestration.
- Web UI and Frontend Documentation:
– A separate “frontend” directory provides the user dashboard and can be integrated with the core system to display real-time metrics, logs, and workflow status.
– The real-time Web UI is supported by built-in alerting and logging components mentioned in the README.
- Supporting Infrastructure:
– The GitHub workflows, Dockerfiles (in “build/package”), and configuration (like Caddyfile, Taskfile.yaml, etc.) show the deployment and CI/CD aspects.
– There is also tooling for schema migration and developer utilities provided under “hack” and “internal/testutils”.

##### Relationships and Data/Task Flow
- The API server (or gateway) receives calls and forwards them to appropriate internal controllers.
- The API interacts with the Worker Engine, which is responsible for task queueing, orchestration and message distribution to workers.
- Worker processes communicate with the persistent storage (Postgres) using modules in “pkg/repository” to record states, logs, and workflow details.
- Integration modules (for alerting, integration with external systems, and messaging queues) feed into or out of the orchestration layer.
- The Web UI sits on top of the API and visualizes data from the persistent storage and orchestration outcomes.

##### Relevant Technologies and Principles
- Technologies: Go, Postgres, GRPC, Docker (build/deployment scripts), and multi-language SDK support (Python, Typescript).
- Architectural principles include durability, resiliency (with built-in retry and alerting mechanisms), and scalability/decoupling of API from the worker layer.
- Patterns like message queues (with support for RabbitMQ and PostgreSQL-based queues) and microservice-like separation (API vs. Worker vs. Integrations) are central.

## Diagram (Mermaid)

```mermaid
graph TD
  %% Client/SDK Clients
  subgraph "Client/SDK Clients"
    Python_SDK("Python SDK"):::sdk
    TS_SDK("Typescript SDK"):::sdk
    Go_Client("Go Client"):::sdk
  end

  %% API / Control Plane
  subgraph "API / Control Plane"
    API_Server("Hatchet API Gateway"):::api
    API_Cmd("API Server Entry Point"):::api
    GRPC("gRPC Endpoints"):::api
  end

  %% Worker Engine & Task Orchestration
  subgraph "Worker Engine & Task Orchestration"
    Worker_Engine("Worker Engine"):::worker
    Dispatcher("Task Dispatcher"):::worker
    Scheduler("Task Scheduler"):::worker
    Ingestor("Task Ingestor"):::worker
    Worker_Routines("Worker Routines"):::worker
  end

  %% Persistence Layer
  subgraph "Persistence Layer"
    Postgres_Repo("Persistent Storage (Postgres)"):::persistence
    SQL_Schema("SQL Schema Migrations"):::persistence
  end

  %% Integration & External Services
  subgraph "Integration & External Services"
    Integrations("External Integrations"):::integration
    Shared_Integrations("Shared Integration Modules"):::integration
    Msgqueue("Message Queue"):::integration
  end

  %% Web UI & Documentation
  subgraph "Web UI & Documentation"
    Web_UI("Web UI Dashboard"):::webui
    Frontend_Docs("Frontend Documentation"):::webui
  end

  %% Supporting Infrastructure
  subgraph "Supporting Infrastructure"
    CI_CD("CI/CD Workflows"):::infra
    Docker_Build("Docker & Deployment Scripts"):::infra
    Taskfile("Taskfile Configuration"):::infra
    Hack("Developer Utilities"):::infra
    DockerCompose("Docker Compose Configuration"):::infra
  end

  %% Connections between layers
  %% Client/SDK Clients to API / Control Plane
  Python_SDK -->|"submitTask"| API_Server
  TS_SDK -->|"submitTask"| API_Server
  Go_Client -->|"submitTask"| API_Server

  %% API internal routing
  API_Server -->|"routesTo"| API_Cmd
  API_Server -->|"exposes"| GRPC
  
  %% API triggering Worker Engine
  API_Server -->|"triggers"| Worker_Engine

  %% Worker Engine orchestration flow
  Worker_Engine -->|"dispatches"| Dispatcher
  Worker_Engine -->|"schedules"| Scheduler
  Worker_Engine -->|"ingests"| Ingestor
  Ingestor -->|"processes"| Worker_Routines

  %% Worker updating Persistence and Integrations
  Worker_Routines -->|"updatesState"| Postgres_Repo
  Worker_Routines -->|"sendsAlert"| Integrations
  Dispatcher -->|"queuesTask"| Msgqueue

  %% Data flow to Web UI
  API_Server -->|"providesData"| Web_UI
  Postgres_Repo -->|"feedsData"| Web_UI

  %% Interaction between Integration components
  Integrations -->|"configSync"| Shared_Integrations

  %% Supporting Infrastructure deploying components
  CI_CD -->|"deploys"| API_Server
  Docker_Build -->|"builds"| Worker_Engine
  Taskfile -->|"configures"| API_Server
  Hack -->|"assists"| SQL_Schema
  DockerCompose -->|"orchestrates"| API_Server
  DockerCompose -->|"orchestrates"| Worker_Engine
  DockerCompose -->|"orchestrates"| Web_UI

  %% Click Events for API / Control Plane
  click API_Server "https://github.com/hatchet-dev/hatchet/tree/main/api/v1/server"
  click API_Cmd "https://github.com/hatchet-dev/hatchet/tree/main/cmd/hatchet-api"
  click GRPC "https://github.com/hatchet-dev/hatchet/tree/main/internal/services/grpc"
  
  %% Click Events for Worker Engine & Task Orchestration
  click Worker_Engine "https://github.com/hatchet-dev/hatchet/tree/main/cmd/hatchet-engine"
  click Dispatcher "https://github.com/hatchet-dev/hatchet/tree/main/internal/services/dispatcher"
  click Scheduler "https://github.com/hatchet-dev/hatchet/tree/main/internal/services/scheduler"
  click Ingestor "https://github.com/hatchet-dev/hatchet/tree/main/internal/services/ingestor"
  click Worker_Routines "https://github.com/hatchet-dev/hatchet/tree/main/pkg/worker"
  
  %% Click Events for Persistence Layer
  click Postgres_Repo "https://github.com/hatchet-dev/hatchet/tree/main/pkg/repository/postgres"
  click SQL_Schema "https://github.com/hatchet-dev/hatchet/tree/main/sql/schema"

  %% Click Events for Integration & External Services
  click Integrations "https://github.com/hatchet-dev/hatchet/tree/main/internal/integrations"
  click Shared_Integrations "https://github.com/hatchet-dev/hatchet/tree/main/pkg/integrations"
  click Msgqueue "https://github.com/hatchet-dev/hatchet/tree/main/internal/msgqueue"

  %% Click Events for SDKs and External Client Interfaces
  click Python_SDK "https://github.com/hatchet-dev/hatchet/tree/main/sdks/python"
  click TS_SDK "https://github.com/hatchet-dev/hatchet/tree/main/sdks/typescript"
  click Go_Client "https://github.com/hatchet-dev/hatchet/tree/main/pkg/client"
  
  %% Click Events for Web UI & Documentation
  click Web_UI "https://github.com/hatchet-dev/hatchet/tree/main/frontend"
  click Frontend_Docs "https://github.com/hatchet-dev/hatchet/tree/main/frontend/docs"
  
  %% Click Events for Supporting Infrastructure
  click CI_CD "https://github.com/hatchet-dev/hatchet/tree/main/.github/workflows"
  click Docker_Build "https://github.com/hatchet-dev/hatchet/tree/main/build/package"
  click Taskfile "https://github.com/hatchet-dev/hatchet/blob/main/Taskfile.yaml"
  click Hack "https://github.com/hatchet-dev/hatchet/tree/main/hack"
  click DockerCompose "https://github.com/hatchet-dev/hatchet/blob/main/docker-compose.yml"

  %% Styles for different layers
  classDef sdk fill:#f0f9ff,stroke:#333,stroke-width:2px;
  classDef api fill:#e0ffe0,stroke:#333,stroke-width:2px;
  classDef worker fill:#fff0f0,stroke:#333,stroke-width:2px;
  classDef persistence fill:#f0fff0,stroke:#333,stroke-width:2px;
  classDef integration fill:#fffce0,stroke:#333,stroke-width:2px;
  classDef webui fill:#e0e0ff,stroke:#333,stroke-width:2px;
  classDef infra fill:#f9e0ff,stroke:#333,stroke-width:2px;
```
