How sky-uk/kafka-message-scheduler works
Main Components Identification
• Scheduler Core: Represented by the scheduler modules (look in directories named “scheduler-3” and “scheduler”). These components include:
• Startup Logic: The component that invokes [kafka-topic-loader] to consume all messages from the schedule topics to rehydrate the servicing actor state.
• Scheduling Actors: Components like Scheduler, ScheduleQueue, SchedulePublisher, EventSubscriber, and auxiliary components (e.g., converters, domain classes/models) that manage schedule processing.
• Kafka Infrastructure:
• Input: “schedule-topics” from which schedule messages (with a key as Schedule ID and Avro encoded message values) are consumed.
• Output: The topic to which the final payload is sent at the scheduled time.
• Avro Schema Generation: The avro sub-project – includes code (e.g., avro/src/main/scala/GenerateSchema.scala) that produces the Avro schema of the Schedule case class.
• Monitoring and Metrics: Integration with Kamon and Prometheus. Kamon is configured in the code (see references in the “scheduler” directory) and Prometheus configurations are available in the docker compose files.
• Deployment/External Tools: Docker configurations (docker/, it/docker/, and associated docker-compose files) that offer a containerized deployment of the system as well as testing/IT setups.
• Build and CI/CD: sbt configuration files (build.sbt, project/Dependencies.scala, etc.) and Travis config showing continuous integration setup.
Relationships and Interactions
• Startup Loader: Upon startup, the system uses the kafka-topic-loader to load existing messages from Kafka, rehydrating the state within the internal scheduling actors.
– Ingestion from Kafka into Scheduler Core.
– Internal scheduling delay (with a timer or scheduling queue).
– Outgoing message publication back to Kafka.
• Avro and Serialization Components: Show that the messages are encoded/decoded using Avro, with separate converters also visible for JSON and Circe (as seen in the scheduler-3 module directories).
• Monitoring Path: Illustrate metrics being gathered via Kamon and exposed on a Prometheus endpoint (port 9095 by default), and the integration with Prometheus which scrapes these metrics.
• Optional Actor Model Subsystem: Optionally, detail that the scheduler uses the Akka actor model (as indicated by files like PublisherActor.scala, SchedulingActor.scala, TerminatorActor.scala) to manage concurrent scheduling functions.
Architectural Patterns or Design Principles
• The system leverages an event-driven architecture – using Kafka to decouple message ingestion and final message dispatch.
• It uses the actor model (via Akka) to manage resilience and scheduling concurrency.
• The architecture cleverly uses Kafka’s log compaction to both delete schedules after processing and to serve as a durable state store during restarts.
• There is evidence of separation of concerns: Avro schema generation is separated into its own module; configuration and dependency management are clearly modularized.
Relevant Technologies, Frameworks, and Libraries
• Scala and sbt for application building.
• Akka actors for concurrency and scheduling.
• Kafka as the messaging backbone.
• Avro for serialization (and corresponding converters for JSON/Circe).
• Kamon for monitoring, with Prometheus for scraping.
• Docker for deployment and integration testing.
Elements
• Kafka (Schedule Topics and Destination Topics)
• Scheduler Core (with internal components such as Startup Loader, Scheduling Actors, Publisher, and Queue)
• Avro Schema Generator (and associated serialization utilities)
• Monitoring (Kamon and Prometheus)
• Deployment (Docker & CI/CD)
• Data ingestion from Kafka into the scheduler.
• Internal scheduling processing and message transformation (serialization/deserialization).
• Outgoing message dispatch to Kafka.
• Periodic metrics reporting towards Prometheus.
• Data stores (Kafka topics)
• Processing components (scheduler sub-systems)
• Third-party services (Prometheus, Docker container boundaries)
• Supporting utilities (Avro generators, test suites)
• All components should have clear labels and annotations, perhaps even including key configuration notes (like scheduled message delivery rules and log compaction details).