How hatchet-dev/hatchet works
Main Components and Layers
• API / Control Plane:
– The “api”, “cmd/hatchet-api”, and “internal/services/grpc” folders manage the core API, request handling, authentication (authn/authz middleware), and GRPC endpoints.
– The API controller layer handles incoming requests from client applications.
• Worker Engine & Task Orchestration:
– The “cmd/hatchet-engine”, “internal/services/dispatcher”, “internal/services/scheduler”, “internal/services/ingestor”, and “pkg/worker” directories indicate that Hatchet has a dedicated worker engine.
– Modules for task routing, workflow orchestration (including DAG and durable executions), and flow control (e.g., concurrency and rate limits) reside here.
– The orchestration logic is built on top of the task factories that allow developers to register tasks and workflows.
• Persistence Layer:
– Postgres (with SQL schema migrations in the “sql/schema” folder and integration modules in “pkg/repository/postgres”) is the underpinning storage engine.
– Durable task queues and persistent storage of task histories are implemented in the repository and msgqueue layers.
• Integration and External Services:
– The “internal/integrations”, “pkg/integrations”, and service folders (for alerting, email, Slack, etc.) contain modules that integrate with third-party services.
– Message queue implementations for RabbitMQ and other protocols also exist under “internal/msgqueue”.
• SDKs and External Client Interfaces:
– There is comprehensive support for clients written in Python (under “sdks/python”), Typescript (under “sdks/typescript”) and references in Go.
– These SDKs encapsulate client calls to the API and include features such as task registration, event triggers, and orchestration.
• Web UI and Frontend Documentation:
– A separate “frontend” directory provides the user dashboard and can be integrated with the core system to display real-time metrics, logs, and workflow status.
– The real-time Web UI is supported by built-in alerting and logging components mentioned in the README.
• Supporting Infrastructure:
– The GitHub workflows, Dockerfiles (in “build/package”), and configuration (like Caddyfile, Taskfile.yaml, etc.) show the deployment and CI/CD aspects.
– There is also tooling for schema migration and developer utilities provided under “hack” and “internal/testutils”.
Relationships and Data/Task Flow
• The API server (or gateway) receives calls and forwards them to appropriate internal controllers.
• The API interacts with the Worker Engine, which is responsible for task queueing, orchestration and message distribution to workers.
• Worker processes communicate with the persistent storage (Postgres) using modules in “pkg/repository” to record states, logs, and workflow details.
• Integration modules (for alerting, integration with external systems, and messaging queues) feed into or out of the orchestration layer.
• The Web UI sits on top of the API and visualizes data from the persistent storage and orchestration outcomes.
Relevant Technologies and Principles
• Technologies: Go, Postgres, GRPC, Docker (build/deployment scripts), and multi-language SDK support (Python, Typescript).
• Architectural principles include durability, resiliency (with built-in retry and alerting mechanisms), and scalability/decoupling of API from the worker layer.
• Patterns like message queues (with support for RabbitMQ and PostgreSQL-based queues) and microservice-like separation (API vs. Worker vs. Integrations) are central.