# jayanth1802/distributed-job-scheduler architecture

> Architecture diagram of the GitHub repository jayanth1802/distributed-job-scheduler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jayanth1802/distributed-job-scheduler
- Repository: https://github.com/jayanth1802/distributed-job-scheduler

## Overview

1. **Overall shape**
   - This is a monorepo for a distributed job scheduler.
   - The system is split into four runtime units: **Postgres-backed database layer**, **Express API**, **polling worker service**, and **React dashboard**.
   - There is **no separate queue broker**; the database is the coordination point for scheduling, claiming, retries, and status history.

2. **Database as the core system boundary**
   - `packages/db` owns the schema and migration runner (`schema.sql`, `migrate.js`, `index.js`).
   - PostgreSQL is the single source of truth for organizations, projects, queues, workers, jobs, retry state, and likely DLQ-related state.
   - Both API and worker depend on the shared DB package, so most system behavior crosses the DB boundary rather than service-to-service RPC.

3. **API service**
   - `packages/api` is an Express server (`server.js`) with route modules for `auth`, `projects`, `queues`, `jobs`, `workers`, `retryPolicies`, and `dashboard`.
   - It handles request validation, authentication middleware, and CRUD/query endpoints for the scheduler domain.
   - The API is also the system’s **WebSocket relay**, converting database change notifications into live UI updates.

4. **Authentication and multi-tenant access**
   - `packages/api/src/routes/auth.js` and `middleware/auth.js` indicate JWT-based auth at the API boundary.
   - The dashboard and API are organized around **organization/project scoping**, with the UI exposing a `ProjectPicker`.
   - This makes “project” the main tenant-level boundary for operational views and job submission.

5. **Worker service**
   - `packages/worker` is a separate Node.js process that continuously polls for runnable jobs.
   - The worker is designed for horizontal scaling: multiple worker processes can run against the same database without double-claiming jobs.
   - Its core flow is `claim → execute → retry/DLQ`, implemented across `claim.js`, `executor.js`, `retryStrategy.js`, `scheduler.js`, and `index.js`.

6. **Atomic job claiming**
   - The worker uses PostgreSQL row locking with `SELECT ... FOR UPDATE SKIP LOCKED` to claim jobs safely under concurrency.
   - This is the key distributed-systems mechanism in the repo: the database serializes ownership while allowing many workers to compete.
   - The repo’s tests explicitly cover claim exclusivity, so this is a central architectural guarantee.

7. **Scheduling and retries**
   - `packages/shared/cron.js` is used by both API and worker to parse cron expressions consistently.
   - The worker’s `scheduler.js` materializes cron/recurring work on a periodic tick, controlled by `SCHEDULER_TICK_MS`.
   - `retryStrategy.js` handles delay/backoff calculations, and the system transitions to a **Dead Letter Queue** after attempts are exhausted.

8. **Task execution boundary**
   - `packages/worker/src/tasks/index.js` suggests a task registry or dispatcher that maps job types to executors.
   - This is the boundary between generic scheduler mechanics and application-specific work execution.
   - In architecture terms, jobs are generic records in Postgres, while task handlers are the code that turns them into actual work.

9. **Live operational visibility**
   - `packages/dashboard` is a Vite/React 18 app with pages for overview, job explorer, job detail, queues, workers, and login.
   - It talks to the API through `packages/dashboard/src/lib/api.js` and uses native WebSocket support for live updates.
   - The UI also uses Recharts for metrics-style visualization, which means the dashboard is not just CRUD; it is an ops console.

10. **Real-time update path**
   - The live-update path is: **Postgres `LISTEN/NOTIFY` → API WebSocket relay → browser WebSocket client**.
   - This keeps the dashboard reactive without having workers or the UI poll the database directly.
   - The API is therefore both a REST control plane and a real-time event bridge.

11. **Deployment/runtime topology**
   - At runtime, the repo expects three long-lived processes: API on `:4000`, worker(s), and dashboard on `:5173`.
   - The dashboard dev server proxies `/api` and `/ws` to the API, but in architecture terms the browser still only depends on the API service.
   - The only required external infrastructure is PostgreSQL 14+.

12. **Shared code and quality controls**
   - `packages/shared` contains cross-cutting logic, currently cron parsing, to prevent drift between services.
   - Tests focus on critical system invariants: concurrency/claim safety, cron handling, retry math, job validation, and auth behavior.
   - That test scope reinforces the architecture’s main risk areas: distributed coordination, time-based scheduling, and access control.

Main entry points: [`packages/api/src/server.js`](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/server.js) (API server), [`packages/worker/src/index.js`](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/index.js) (Worker), [`packages/dashboard/src/App.jsx`](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/dashboard/src/App.jsx) (Dashboard), [`packages/db/index.js`](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/db/index.js) (DB package)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime units"]
  node_api["API server<br/>Express control plane<br/>[server.js]"]
  node_authmw["Auth middleware<br/>JWT auth<br/>[auth.js]"]
  node_apiauth["Auth routes<br/>login/session<br/>[auth.js]"]
  node_projects["Projects<br/>tenant APIs<br/>[projects.js]"]
  node_queues["Queues<br/>queue APIs<br/>[queues.js]"]
  node_jobs["Jobs<br/>job APIs<br/>[jobs.js]"]
  node_workersapi["Workers API<br/>worker APIs<br/>[workers.js]"]
  node_retryapi["Retry APIs<br/>policy APIs<br/>[retryPolicies.js]"]
  node_dashapi["Dashboard API<br/>live update bridge<br/>[dashboard.js]"]
  node_worker["Worker<br/>polling service<br/>[index.js]"]
  node_claim["Claiming<br/>row-lock claim<br/>[claim.js]"]
  node_exec["Executor<br/>task runner<br/>[executor.js]"]
  node_tasks["Task registry<br/>job handlers<br/>[index.js]"]
  node_retry["Retry strategy<br/>backoff+DLQ<br/>[retryStrategy.js]"]
  node_sched["Scheduler tick<br/>recurring scheduler<br/>[scheduler.js]"]
  node_dashboard["Dashboard<br/>React ops console<br/>[App.jsx]"]
  node_uikit["Project picker<br/>tenant UI<br/>[ProjectPicker.jsx]"]
  node_apiclient["API client<br/>browser client<br/>[api.js]"]
  node_pages["Ops pages<br/>views<br/>[Overview.jsx]"]
end

subgraph group_data["Data plane"]
  node_postgres[("Postgres<br/>database<br/>[schema.sql]")]
  node_dbpkg["DB package<br/>schema+migrations<br/>[index.js]"]
  node_sharedcron["Cron utils<br/>shared cron<br/>[cron.js]"]
end

node_dbpkg -->|"migrates+reads"| node_postgres
node_api -->|"queries"| node_dbpkg
node_worker -->|"polls"| node_dbpkg
node_api -->|"parses cron"| node_sharedcron
node_worker -->|"parses cron"| node_sharedcron
node_api -->|"guards routes"| node_authmw
node_api -->|"serves"| node_apiauth
node_api -->|"serves"| node_projects
node_api -->|"serves"| node_queues
node_api -->|"serves"| node_jobs
node_api -->|"serves"| node_workersapi
node_api -->|"serves"| node_retryapi
node_api -->|"serves"| node_dashapi
node_worker -->|"claims"| node_claim
node_worker -->|"executes"| node_exec
node_exec -->|"dispatches"| node_tasks
node_worker -->|"retries"| node_retry
node_worker -->|"ticks"| node_sched
node_dashboard -->|"uses"| node_apiclient
node_apiclient -->|"calls"| node_api
node_uikit -->|"navigates"| node_dashboard
node_pages -->|"renders"| node_dashboard

click node_postgres "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/db/schema.sql"
click node_dbpkg "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/db/index.js"
click node_sharedcron "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/shared/cron.js"
click node_api "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/server.js"
click node_authmw "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/middleware/auth.js"
click node_apiauth "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/auth.js"
click node_projects "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/projects.js"
click node_queues "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/queues.js"
click node_jobs "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/jobs.js"
click node_workersapi "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/workers.js"
click node_retryapi "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/retryPolicies.js"
click node_dashapi "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/dashboard.js"
click node_worker "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/index.js"
click node_claim "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/claim.js"
click node_exec "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/executor.js"
click node_tasks "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/tasks/index.js"
click node_retry "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/retryStrategy.js"
click node_sched "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/scheduler.js"
click node_dashboard "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/dashboard/src/App.jsx"
click node_uikit "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/dashboard/src/components/ProjectPicker.jsx"
click node_apiclient "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/dashboard/src/lib/api.js"
click node_pages "https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/dashboard/src/pages/Overview.jsx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_authmw,node_apiauth,node_projects,node_queues,node_jobs,node_workersapi,node_retryapi,node_dashapi,node_worker,node_claim,node_exec,node_tasks,node_retry,node_sched,node_dashboard,node_uikit,node_apiclient,node_pages toneBlue
class node_postgres,node_dbpkg,node_sharedcron toneAmber
```

## Components

### Runtime units

Long-lived services and the browser client

- [API server](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/server.js): Express control plane · REST API, auth boundary, and WebSocket relay · `packages/api/src/server.js`
- [Auth middleware](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/middleware/auth.js): JWT auth · Request authentication and tenant access enforcement · `packages/api/src/middleware/auth.js`
- [Auth routes](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/auth.js): login/session · Authentication endpoints for the dashboard and API clients · `packages/api/src/routes/auth.js`
- [Projects](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/projects.js): tenant APIs · Organization and project-scoped control and lookup endpoints · `packages/api/src/routes/projects.js`
- [Queues](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/queues.js): queue APIs · Queue configuration and queue-level job operations · `packages/api/src/routes/queues.js`
- [Jobs](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/jobs.js): job APIs · Job creation, querying, detail views, and lifecycle endpoints · `packages/api/src/routes/jobs.js`
- [Workers API](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/workers.js): worker APIs · Worker registry and operational endpoints · `packages/api/src/routes/workers.js`
- [Retry APIs](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/retryPolicies.js): policy APIs · Retry policy and delay configuration endpoints · `packages/api/src/routes/retryPolicies.js`
- [Dashboard API](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/api/src/routes/dashboard.js): live update bridge · API surface used by the dashboard and WebSocket event relay · `packages/api/src/routes/dashboard.js`
- [Worker](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/index.js): polling service · Continuously claims, executes, retries, and dead-letters jobs · `packages/worker/src/index.js`
- [Claiming](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/claim.js): row-lock claim · Atomic job acquisition with SKIP LOCKED to avoid double-processing · `packages/worker/src/claim.js`
- [Executor](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/executor.js): task runner · Dispatches claimed jobs to task handlers · `packages/worker/src/executor.js`
- [Task registry](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/tasks/index.js): job handlers · Maps job types to application-specific executors · `packages/worker/src/tasks/index.js`
- [Retry strategy](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/retryStrategy.js): backoff+DLQ · Backoff calculations and dead-letter transition logic · `packages/worker/src/retryStrategy.js`
- [Scheduler tick](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/worker/src/scheduler.js): recurring scheduler · Materializes cron-based work on a periodic poll · `packages/worker/src/scheduler.js`
- [Dashboard](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/dashboard/src/App.jsx): React ops console · Browser UI for overview, jobs, queues, workers, and login · `packages/dashboard/src/App.jsx`
- [Project picker](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/dashboard/src/components/ProjectPicker.jsx): tenant UI · Project-scoped navigation and selection in the dashboard · `packages/dashboard/src/components/ProjectPicker.jsx`
- [API client](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/dashboard/src/lib/api.js): browser client · Dashboard HTTP/WebSocket client for API communication · `packages/dashboard/src/lib/api.js`
- [Ops pages](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/dashboard/src/pages/Overview.jsx): views · Overview, job explorer, job detail, queues, workers, and login screens · `packages/dashboard/src/pages/Overview.jsx`

### Data plane

PostgreSQL-backed coordination and shared logic

- [Postgres](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/db/schema.sql): database · Single source of truth for jobs, workers, queues, orgs, projects, retries, and history · `packages/db/schema.sql`
- [DB package](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/db/index.js): schema+migrations · Shared database access, schema, and migration runner · `packages/db/index.js`
- [Cron utils](https://github.com/jayanth1802/distributed-job-scheduler/blob/main/packages/shared/cron.js): shared cron · Common cron parsing used by API and worker · `packages/shared/cron.js`

## Connections

- DB package → Postgres: migrates+reads · Creates and manages the schema in PostgreSQL
- API server → DB package: queries · Reads and writes scheduler state through the shared DB layer
- Worker → DB package: polls · Claims and updates jobs through the shared DB layer
- API server → Cron utils: parses cron · Uses shared cron logic for request validation and scheduling rules
- Worker → Cron utils: parses cron · Uses the same cron semantics for recurring work
- API server → Auth middleware: guards routes · Applies JWT authentication and access checks
- API server → Auth routes: serves · Exposes login and auth-related endpoints
- API server → Projects: serves · Exposes project and tenant-scoped endpoints
- API server → Queues: serves · Exposes queue management endpoints
- API server → Jobs: serves · Exposes job lifecycle endpoints
- API server → Workers API: serves · Exposes worker operational endpoints
- API server → Retry APIs: serves · Exposes retry policy endpoints
- API server → Dashboard API: serves · Feeds dashboard queries and live updates
- Worker → Claiming: claims · Atomically acquires runnable jobs
- Worker → Executor: executes · Runs claimed jobs through the execution pipeline
- Executor → Task registry: dispatches · Selects the handler for each job type
- Worker → Retry strategy: retries · Computes backoff and DLQ transitions after failures
- Worker → Scheduler tick: ticks · Creates runnable jobs from cron or recurring schedules
- Dashboard → API client: uses · Dashboard talks to the API over HTTP and WebSocket
- API client → API server: calls · Browser client communicates with the API control plane
- Project picker → Dashboard: navigates · Project selection scopes dashboard views
- Ops pages → Dashboard: renders · Operational views make up the main user interface
