How etf1/kafka-message-scheduler works
Project Type and Purpose
• The project is a specialized scheduling service for Kafka messages. Its purpose is to schedule and trigger Kafka messages at specified times based on custom headers and payload information.
• The README explains how scheduled messages are produced on one topic and, at the proper time, are consumed, processed, and then reissued to target topics. It also covers additional aspects like history auditing, failover, and high availability.
File Structure for Architectural Clues
• Top-level directories and files point to multiple components:
• apiserver: Contains the REST API implementation (server.go, routes.go) which is likely used for managing schedules.
• clientlib: Provides a Go client library to wrap Kafka messages, facilitating the creation and management of schedule messages.
• cmd: Contains different entry points (kafka and mini), reflecting different runtime deployments (full scheduler and a “mini” version for integration testing).
• runner, schedule, and scheduler: These directories show encapsulated logic for running the scheduler, handling scheduling details, and triggering messages.
• internal: Contains internal modules such as collectors, stores, and helper functions, indicating components like in-memory storage (hmap) for schedules and possibly a strategy pattern for retries.
• instrument/prometheus: Indicates observability via Prometheus metrics.
• Configuration files (Dockerfile, docker-compose, YAML configs) indicate containerization and Kubernetes deployment support.
Main Components
• Client Applications:
• External producers (using either this client library or other language clients) that produce schedule messages to the scheduler’s designated Kafka topic.
• Kafka Broker (External Dependency):
• The Kafka cluster hosting multiple topics: the “schedules” topic for incoming schedule messages, the target topics (e.g., online-videos as in the readme examples), and the “history” topic for auditing.
• Scheduler Core Service:
• Acts as a proxy consumer reading from the schedules topic.
• Processes scheduled messages: It loads and holds messages in an in-memory store (see internal/store and internal/collector).
• Implements scheduling logic (using components from schedule, runner, and scheduler directories) to trigger messages at the appropriate time.
• Upon triggering, the scheduler produces messages to the target topic and a tombstone message to the schedules topic.
• Manages rescheduling, updates (if messages are modified), and deletion once processed.
• API Server Component:
• Exposed under apiserver; provides REST endpoints for operations such as monitoring, schedule management, or querying status.
• Metrics & Observability:
• The instrument/prometheus module that exposes metrics (at port 8001 or as configured) to Prometheus.
• Supporting Tools & Runtimes:
• The “mini” scheduler (cmd/mini), which runs without dependencies for integration tests.
• Client library (clientlib) aiding integration with Go applications.
• Retry strategies (located in clientlib/retry) which may affect how downstream consumers handle rescheduled (retry) messages.
Relationships and Data Flow
• Data Flow:
• Scheduling Request: A client (or code using the client library) sends a schedule message to the “schedules” topic with required headers (scheduler-epoch, scheduler-target-topic, scheduler-target-key).
• The Scheduler Service reads the topic (from startup and continuously), loads daily schedules into memory, and monitors for changes.
• When the execution time arrives, the scheduler triggers the message:
▪ It produces a message to the designated target Kafka topic.
▪ It also sends a copy to the “history” topic for auditing.
▪ Finally, it produces a tombstone message in the “schedules” topic to delete the original message.
• The API Server and possibly the admin GUI (from the external repository mentioned in the README) provide visibility into this process.
• Interactions:
• Multiple instances of the scheduler (high availability) work concurrently and are load balanced based on Kafka partitions.
• Observability components (Prometheus endpoints) provide operational metrics.
Architectural Patterns and Design Principles
• Statelessness: The scheduler is designed to be stateless – all schedule state is stored in Kafka, making it easy to restart and scale horizontally.
• Memory-Based Caching: It temporarily caches the current day’s schedules in memory (as seen in internal store implementations using hash maps).
• Producer-Consumer Pattern: Using Kafka as the backbone for sending and receiving messages.
• Retry and Failover: The design ensures missed or delayed messages are handled (see runner and retry strategy implementations).
• Separation of Concerns: Distinct components handle REST API, scheduling logic, metrics, and client library functionalities.
Relevant Technologies and Frameworks
• Language: Go (all modules and services are written in Go).
• Messaging: Apache Kafka (using confluent’s rdkafka library).
• Containerization & Deployment: Docker and Kubernetes (as evidenced by Dockerfiles, docker-compose, and k8s YAML files).
• Observability: Prometheus (with specific endpoints for metrics).
• CI/CD: GitHub Actions (.github/workflows directory).
Final Touches
• Incorporate any important technical details from the README such as the scheduling headers, tombstone mechanism for deletion, and discussions regarding partitioning configuration (murmur2_random vs. consistent_random).
• Clearly separate environment-specific components – for example, showing both the full Kafka-based runtime (cmd/kafka) and the “mini” version.