# ltsopensource/light-task-scheduler architecture

> Architecture diagram of the GitHub repository ltsopensource/light-task-scheduler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ltsopensource/light-task-scheduler
- Repository: https://github.com/ltsopensource/light-task-scheduler

## Overview

##### Project type and purpose
- This is a distributed, full-stack task scheduling framework ("LTS – Light Task Scheduler")
- It comprises several cooperating node types (JobClient, JobTracker, TaskTracker, MonitorAgent, Admin UI) plus supporting modules (registry, fail-store, queue, remoting, SPI extensions)

##### Top-level components (modules)
- lts-core: The heart of the framework (cluster management, remoting, queue abstractions, fail-store, SPI)
- lts-jobclient: Submits jobs and receives callbacks/results
- lts-jobtracker: Central scheduler; accepts jobs, persists them, elects master, assigns to TaskTrackers
- lts-tasktracker: Executes jobs, reports back results and business logs
- lts-monitor: Agent that collects JVM / node metrics and pushes them to storage
- lts-admin: Web‐based management console (Velocity templates, REST APIs, uses Spring+Jetty)
- lts-spring: Spring/Boot integration (auto-configuration, XML/annotation support)
- lts-startup: Launch scripts and container bootstrapping for all node types

##### External services and infrastructure
- Registry: Zookeeper or Redis cluster for service discovery and master election
- Storage: MySQL or MongoDB for job queues, monitoring data, logs
- FailStore: pluggable local persistence (LevelDB, RocksDB, etc.) for resiliency
- Remoting: Netty or Mina transport with SPI-driven serialization (FastJSON, Hessian2, Java)
- Web server: Jetty for serving lts-admin.war

##### Component interactions and data flows
A. Job Submission Flow
- JobClient discovers JobTracker via Registry
- JobClient → JobTracker (RemotingCommand: JobSubmitRequest)
- JobTracker persists to JobQueue (MySQL/Mongo)
- JobTracker schedules job to TaskTracker (RemotingCommand: JobPushRequest)
- TaskTracker executes, returns Result (JobCompletedRequest)
- JobTracker persists status, optionally notifies JobClient Callback (JobFinishedResponse)

B. Monitoring & Admin Flow
- MonitorAgent collects JVM/Node metrics → writes to Monitor DB via HTTP command API
- Admin UI calls JobTracker REST APIs and Monitor APIs → displays queues, logs, charts

##### Architectural patterns & design principles
- SPI / Factory: pluggable queue store, fail-store, logger, serialization, registry
- Master election: each NodeGroup elects one master (Registry + listeners)
- Load balancing: abstract LoadBalance implementations (consistent hash, round-robin)
- Retry & FailStore: local durable queue for handling transient network failures
- MVC for Admin: Velocity templates + Spring MVC controllers + REST API layer
- Decoupling via RemotingCommand and HTTPCommand abstractions

##### Essential elements
- Clear labels for modules and subcomponents (e.g., “JobTracker: Cluster + Queue + Remoting”)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Services
    Registry("Registry\n(Zookeeper/Redis)"):::external
    click Registry "https://github.com/ltsopensource/light-task-scheduler/tree/master/lts-core/registry"

    %% Core Runtime Nodes
    subgraph "Submission Plane"
        direction TB
        JobClient["JobClient"]:::core
    end
    subgraph "Scheduling Plane"
        direction TB
        subgraph "JobTracker Cluster"
            direction TB
            JT["JobTracker"]:::core
            JTsub1["ClusterManager\n(MasterElector, NodeRegistry)"]:::core
            JTsub2["Remoting Server/Client\n(Netty/Mina)"]:::core
            JTsub3["QueueFactory\n(MySQL/Mongo)"]:::core
            JTsub4["FailStoreFactory"]:::core
            JTsub5["CommandProcessors\n(JobSubmit, JobPush, etc.)"]:::core
        end
        subgraph "TaskTracker Cluster"
            direction TB
            TT["TaskTracker"]:::core
            TTsub1["ClusterManager"]:::core
            TTsub2["Remoting Server/Client"]:::core
            TTsub3["TaskProcessor"]:::core
        end
    end

    %% Management Plane
    AdminUI["Admin UI\n(Spring MVC + Velocity)"]:::ui
    Monitor["MonitorAgent"]:::monitor

    %% Backends
    Storage["Storage\n(MySQL/Mongo)"]:::datastore
    click Storage "https://github.com/ltsopensource/light-task-scheduler/tree/master/lts-core/queue"
    FailStore["FailStore\n(LevelDB/RocksDB)"]:::datastore
    click FailStore "https://github.com/ltsopensource/light-task-scheduler/tree/master/lts-core/fail-store"

    %% Interactions
    Registry <--> JobClient
    Registry <--> JT
    Registry <--> TT

    JobClient -->|JobSubmitRequest| JT
    JT -->|persist job| Storage
    JT -->|JobPushRequest| TT
    TT -->|JobCompletedRequest| JT
    JT -->|JobFinishedResponse| JobClient

    Monitor -->|push metrics| Storage
    AdminUI -->|REST API| JT
    AdminUI -->|HTTP API| Monitor

    %% SPI Extension Points (dashed)
    JTsub3 -.->|SPI: QueueStore| Storage
    JTsub4 -.->|SPI: FailStore| FailStore
    JTsub2 -.->|SPI: Serializer| JTsub2

    %% Click Events
    click JobClient "https://github.com/ltsopensource/light-task-scheduler/blob/master/lts-jobclient/src/main/java/com/github/ltsopensource/jobclient/JobClient.java"
    click JT "https://github.com/ltsopensource/light-task-scheduler/blob/master/lts-jobtracker/src/main/java/com/github/ltsopensource/jobtracker/JobTracker.java"
    click TT "https://github.com/ltsopensource/light-task-scheduler/blob/master/lts-tasktracker/src/main/java/com/github/ltsopensource/tasktracker/TaskTracker.java"
    click Monitor "https://github.com/ltsopensource/light-task-scheduler/blob/master/lts-monitor/src/main/java/com/github/ltsopensource/monitor/MonitorAgent.java"
    click AdminUI "https://github.com/ltsopensource/light-task-scheduler/tree/master/lts-admin/src/main/java/com/github/ltsopensource/admin/"
    click AdminUI "https://github.com/ltsopensource/light-task-scheduler/tree/master/lts-admin/src/main/webapp/WEB-INF/views/"
    click Spring "https://github.com/ltsopensource/light-task-scheduler/tree/master/lts-spring/src/main/java/com/github/ltsopensource/spring/boot/"
    click Startup "https://github.com/ltsopensource/light-task-scheduler/tree/master/lts-startup/src/main/java/com/github/ltsopensource/startup/"
    click Startup "https://github.com/ltsopensource/light-task-scheduler/tree/master/lts-startup/src/main/resources/bin/"

    %% Styles
    classDef core fill:#bbdefb,stroke:#1e88e5,color:#0d47a1;
    classDef datastore fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20;
    classDef external fill:#ffe0b2,stroke:#ef6c00,color:#e65100;
    classDef ui fill:#ffcdd2,stroke:#c62828,color:#b71c1c;
    classDef monitor fill:#eeeeee,stroke:#757575,color:#424242;
```
