How ltsopensource/light-task-scheduler works
Project type and purpose
• This is a distributed, full-stack task scheduling framework ("LTS – Light Task Scheduler")
• It comprises several cooperating node types (JobClient, JobTracker, TaskTracker, MonitorAgent, Admin UI) plus supporting modules (registry, fail-store, queue, remoting, SPI extensions)
Top-level components (modules)
• lts-core: The heart of the framework (cluster management, remoting, queue abstractions, fail-store, SPI)
• lts-jobclient: Submits jobs and receives callbacks/results
• lts-jobtracker: Central scheduler; accepts jobs, persists them, elects master, assigns to TaskTrackers
• lts-tasktracker: Executes jobs, reports back results and business logs
• lts-monitor: Agent that collects JVM / node metrics and pushes them to storage
• lts-admin: Web‐based management console (Velocity templates, REST APIs, uses Spring+Jetty)
• lts-spring: Spring/Boot integration (auto-configuration, XML/annotation support)
• lts-startup: Launch scripts and container bootstrapping for all node types
External services and infrastructure
• Registry: Zookeeper or Redis cluster for service discovery and master election
• Storage: MySQL or MongoDB for job queues, monitoring data, logs
• FailStore: pluggable local persistence (LevelDB, RocksDB, etc.) for resiliency
• Remoting: Netty or Mina transport with SPI-driven serialization (FastJSON, Hessian2, Java)
• Web server: Jetty for serving lts-admin.war
Component interactions and data flows
A. Job Submission Flow
• JobClient discovers JobTracker via Registry
• JobClient → JobTracker (RemotingCommand: JobSubmitRequest)
• JobTracker persists to JobQueue (MySQL/Mongo)
• JobTracker schedules job to TaskTracker (RemotingCommand: JobPushRequest)
• TaskTracker executes, returns Result (JobCompletedRequest)
• JobTracker persists status, optionally notifies JobClient Callback (JobFinishedResponse)
B. Monitoring & Admin Flow
• MonitorAgent collects JVM/Node metrics → writes to Monitor DB via HTTP command API
• Admin UI calls JobTracker REST APIs and Monitor APIs → displays queues, logs, charts
Architectural patterns & design principles
• SPI / Factory: pluggable queue store, fail-store, logger, serialization, registry
• Master election: each NodeGroup elects one master (Registry + listeners)
• Load balancing: abstract LoadBalance implementations (consistent hash, round-robin)
• Retry & FailStore: local durable queue for handling transient network failures
• MVC for Admin: Velocity templates + Spring MVC controllers + REST API layer
• Decoupling via RemotingCommand and HTTPCommand abstractions
Essential elements
• Clear labels for modules and subcomponents (e.g., “JobTracker: Cluster + Queue + Remoting”)