How aizuda/snail-job works
1. What this repo is
- snail-job is a distributed task platform with two core product lines:
- task retry: resilient, replayable retry workflows for application failures
- task scheduling: distributed cron/triggered jobs and workflow orchestration
- The repository is organized as a multi-module Spring Boot ecosystem rather than a single app.
2. Top-level architecture
- The codebase splits cleanly into:
- client libraries that application services embed
- server-side runtime that schedules, dispatches, and tracks tasks
- interface layer for web/admin/API access
- shared common libraries for models, logging, networking, and utilities
- datasource templates for database-specific persistence SQL/XML
3. Client side boundary
- snail-job-client-* provides the embedding layer for application services.
- snail-job-client-starter is the Spring Boot autoconfiguration entrypoint.
- snail-job-client-common contains shared RPC, endpoint scanning, log reporting, and client lifecycle plumbing.
- snail-job-client-job-core implements job-side execution semantics: job executors, sharding/map-reduce support, endpoint registration, callback handling, timers, and OpenAPI/RPC integration.
- snail-job-client-retry-core implements retry semantics: @Retryable interception, local vs remote retry execution, argument serialization, idempotent ID generation, callback/reporting, and retry-specific context propagation.
4. Server side boundary
- snail-job-server-starter is the runnable server application.
- It wires the server runtime, loads configuration, starts dispatch processes, and includes startup/shutdown listeners.
- This is the operational entrypoint that coordinates task execution, node membership, and offline handling.
5. Server runtime is split by domain
- snail-job-server-dispatcher contains the heavy runtime logic and is split into:
- snail-job-server-job-task: scheduling and execution of jobs/workflows
- snail-job-server-retry-task: retry orchestration and retry lifecycle management
- snail-job-server-common: shared server runtime infrastructure
- These modules use an actor-oriented style in many places (dispatch/*Actor, Pekko support), which suggests asynchronous orchestration rather than direct request/response only.
6. Job scheduling and workflow flow
- Job definitions and workflow definitions are persisted, then prepared into batches/tasks.
- Timer wheels and timeout check tasks drive when work becomes runnable.
- Execution moves through:
- task generation
- prepare/lock/block-strategy selection
- dispatch to client nodes
- client callback/result handling
- status updates and summary aggregation
- Workflow execution has its own graph/cache, node execution, and workflow-specific stop/result handlers.
7. Retry flow
- Retry requests can originate from the business client or be managed manually from the server UI/API.
- The retry pipeline includes:
- trigger/retry request intake
- context and task generation
- block/rate-limit checks
- dispatch to a retry-capable client endpoint
- callback/result processing
- dead-letter handling when retry thresholds are exceeded
- The retry module is parallel to job scheduling but has its own state machine, alarm thresholds, and dead-letter persistence.
8. Control plane APIs
- snail-job-server-interface exposes the external-facing interfaces:
- snail-job-server-web: internal admin web controllers, WebSocket/log streaming, auth, and view-model conversions
- snail-job-server-openapi: programmatic API surface with authentication interceptor and service layer
- snail-job-server-service: business/service layer used by both UI and OpenAPI
- This layer is the boundary for operators and integrations, translating external requests into the server runtime’s domain objects.
9. Persistence and database portability
- snail-job-datasource/snail-job-datasource-template is the core persistence abstraction using MyBatis-style mappers and data objects.
- Database-specific modules exist for MySQL, MariaDB, Oracle, PostgreSQL, SQL Server, Kingbase, and DM8.
- The repo includes SQL bootstrap scripts and schema migration helpers under doc/sql, doc/docker, and doc/tool.
- This indicates the platform is designed to run across multiple enterprise databases with vendor-specific mapper XML overrides.
10. Networking and transport
- The platform uses a mix of:
- HTTP/OpenAPI for admin and public control operations
- gRPC for client/server RPC and low-latency task callbacks
- WebSocket for live log/console-style streaming in the UI
- Shared common modules provide generated protobuf/gRPC types, RPC abstractions, request builders, headers, and response models.
- The separation suggests runtime task execution is RPC-driven, while operator/admin access is HTTP-driven.
11. Cross-cutting concerns
- Logging is a first-class subsystem:
- shared log abstractions in snail-job-common-log
- client log appenders/reporters
- server log storage, merge, and cleanup schedules
- Alarm/notification is built in:
- email, DingTalk, Lark, WeCom/webhook integrations
- server-side alarm events for job, retry, and workflow failures
- Security/authentication appears in the web/openapi layers with interceptors and login-related annotations.
12. Consistency, scheduling, and coordination
- Distributed coordination is handled with:
- locks and lock providers
- load-balancing strategies for clients and servers
- idempotency helpers for timers and task triggers
- caches for endpoint metadata, executor info, and group state
- These subsystems are important in diagrams because they sit between the scheduler and the actual clients, preventing duplicate execution and coordinating cluster behavior.
13. Operational packaging
- The repo includes:
- a runnable server starter
- Docker support
- CI workflows for image build and Maven publish
- docs and demo/migration tooling
- This shows the project is meant to be deployed as a platform service, not just used as a library.
14. How to draw the architecture
- A good diagram should show four layers:
- Application clients embedding snail-job client libraries
- Server control plane exposing OpenAPI/Web/UI
- Runtime dispatch plane handling job/retry/workflow orchestration, timers, locking, and RPC
- Persistence/infra with DB, logs, alarms, and multiple transport protocols
- The key data paths are:
- operator/API request → service layer → dispatcher/runtime → database
- application client registration/reporting → RPC/OpenAPI → dispatcher/runtime
- timer/scan event → task generation → client dispatch → callback/result → status/log persistence
Main entry points: SnailJobServiceStarter.java Service layer · SnailJobServerJobTaskStarter.java Job runtime · SnailJobServerRetryTaskStarter.java Retry runtime · CommonLogConfigure.java Common log
Components 19
Client side
Embedded libraries used by application services
- Client starter · Bootstraps embedded client capabilities and registers client beans
snail-job-client-starter/src/main/java/com/aizuda/snailjob/client/starter/EnableSnailJob.java - Client common · Shared endpoint scanning, RPC transport, lifecycle, and log reporting
snail-job-client/snail-job-client-common/src/main/java/com/aizuda/snailjob/client/common/RpcClient.java - Job client · Job executors, sharding/map-reduce, callbacks, timers, and OpenAPI/RPC integration
snail-job-client/snail-job-client-job-core/src/main/java/com/aizuda/snailjob/client/job/core/executor/AbstractJobExecutor.java - Retry client · @Retryable interception, local/remote retry, serialization, and reporting
snail-job-client/snail-job-client-retry-core/src/main/java/com/aizuda/snailjob/client/core/intercepter/SnailRetryInterceptor.java
Server runtime
Dispatcher and execution engine
- Server app · Runnable server entrypoint and lifecycle coordination
snail-job-server-starter/src/main/java/com/aizuda/snailjob/server/SnailJobServerApplication.java - Dispatch service · Starts dispatch processes, handles membership, startup, shutdown, and offline handling
snail-job-server-starter/src/main/java/com/aizuda/snailjob/server/starter/dispatch/DispatchService.java - Server common · Shared server-side load balancing, locking, RPC, Pekko actors, and request handlers
snail-job-server-dispatcher/snail-job-server-common/src/main/java/com/aizuda/snailjob/server/common/config/SnailJobServerCommonAutoConfiguration.java - Job runtime · Task generation, prepare/execute/result, callbacks, timers, stop handling, and workflow orchestration
snail-job-server-dispatcher/snail-job-server-job-task/src/main/java/com/aizuda/snailjob/server/job/task/SnailJobServerJobTaskStarter.java - Retry runtime · Retry generation, rate limiting, prepare/result, dead-letter handling, and alarm schedules
snail-job-server-dispatcher/snail-job-server-retry-task/src/main/java/com/aizuda/snailjob/server/retry/task/SnailJobServerRetryTaskStarter.java
Control plane
Web and OpenAPI surfaces for operators and integrations
- OpenAPI · Programmatic job, retry, and workflow APIs with authentication
snail-job-server-interface/snail-job-server-openapi/src/main/java/com/aizuda/snailjob/server/openapi/SnailJobOpenApiStarter.java - Admin UI · Browser-facing controllers, auth, WebSocket/log streaming, and view models
snail-job-server-interface/snail-job-server-web/src/main/java/com/aizuda/snailjob/server/web/SnailJobWebStarter.java - Service layer · Shared business services used by OpenAPI and UI
snail-job-server-interface/snail-job-server-service/src/main/java/com/aizuda/snailjob/server/service/SnailJobServiceStarter.java
Shared infra
Common models, logging, persistence, and vendor templates
- Common core · Cross-cutting enums, models, RPC payloads, cron/expression, and gRPC types
snail-job-common/snail-job-common-core/src/main/java/com/aizuda/snailjob/common/core/CommonCoreConfigure.java - Common log · Logging abstraction and local/remote log backends used across client and server
snail-job-common/snail-job-common-log/src/main/java/com/aizuda/snailjob/common/log/CommonLogConfigure.java - Common model · Request and response DTOs shared between clients, server, and control plane
snail-job-common/snail-job-common-model/src/main/java/com/aizuda/snailjob/model/request/JobApiRequest.java - Datasource template · Core persistence abstraction, mappers, data objects, and DB-agnostic SQL templates
snail-job-datasource/snail-job-datasource-template/src/main/java/com/aizuda/snailjob/template/datasource/config/SnailJobTemplateAutoConfiguration.java - DB overrides · Database-specific mapper XMLs for MySQL, PostgreSQL, Oracle, SQL Server, MariaDB, Kingbase, and DM8
snail-job-datasource/snail-job-mysql-datasource/src/main/resources/mysql/mapper/JobMapper.xml - Alarms · Email, DingTalk, Lark, WeCom, and webhook alarm strategies
snail-job-common/snail-job-common-core/src/main/java/com/aizuda/snailjob/common/core/alarm/SnailJobAlarmFactory.java - RPC transport · gRPC and HTTP request/response plumbing used for control and dispatch paths
snail-job-common/snail-job-common-core/src/main/proto/snail_job_grpc_service.proto
Connections 31
- Client starter → Client common: configures · Enables shared client plumbing and bean registration
- Client starter → Job client: enables · Wires job execution client support
- Client starter → Retry client: enables · Wires retry client support
- Job client → RPC transport: calls · Uses RPC and OpenAPI transport to register, report, and trigger jobs
- Retry client → RPC transport: calls · Uses RPC and OpenAPI transport to trigger and report retries
- Job client → Common core: depends on · Shares job context, headers, results, and protocol types
- Retry client → Common core: depends on · Shares retry context, headers, results, and protocol types
- Job client → Common log: reports · Sends job execution logs and metadata
- Retry client → Common log: reports · Sends retry logs and metadata
- Server app → Dispatch service: starts · Bootstraps dispatch processes and cluster lifecycle
- Dispatch service → Server common: uses · Relies on shared server infrastructure and actors
- Dispatch service → Job runtime: starts · Activates job scheduling and execution runtime
- Dispatch service → Retry runtime: starts · Activates retry orchestration runtime
- Job runtime → Datasource template: persists · Reads and updates job, workflow, task, and log state
- Retry runtime → Datasource template: persists · Reads and updates retry, dead-letter, task, and log state
- Job runtime → Common log: stores logs · Merges, clears, and persists job logs
- Retry runtime → Common log: stores logs · Merges, clears, and persists retry logs
- Job runtime → Alarms: emits · Raises job and workflow failure notifications
- Retry runtime → Alarms: emits · Raises retry and dead-letter notifications
- OpenAPI → Service layer: delegates · Routes API requests into shared business services
- Admin UI → Service layer: delegates · Routes admin actions into shared business services
- Service layer → Job runtime: controls · Translates API/UI operations into job runtime actions
- Service layer → Retry runtime: controls · Translates API/UI operations into retry runtime actions
- Service layer → Datasource template: reads/writes · Uses persistence to query and mutate platform state
- Service layer → Common model: maps · Converts between UI/API models and shared DTOs
- OpenAPI → RPC transport: exposes · Publishes HTTP control endpoints over the shared transport model
- Admin UI → RPC transport: streams · Uses WebSocket and HTTP endpoints for console interactions
- Server common → RPC transport: implements · Provides server RPC handlers and client-facing request plumbing
- Server common → Common core: depends on · Shares common runtime types, expressions, and utility models
- Common core → Common log: integrates · Provides shared logging abstractions and runtime helpers
- Datasource template → DB overrides: adapts · Loads vendor-specific mapper XML overrides