# sky-uk/kafka-message-scheduler architecture

> Architecture diagram of the GitHub repository sky-uk/kafka-message-scheduler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sky-uk/kafka-message-scheduler
- Repository: https://github.com/sky-uk/kafka-message-scheduler

## Overview

##### Main Components Identification
- Scheduler Core: Represented by the scheduler modules (look in directories named “scheduler-3” and “scheduler”). These components include:
- Startup Logic: The component that invokes [kafka-topic-loader] to consume all messages from the schedule topics to rehydrate the servicing actor state.
- Scheduling Actors: Components like Scheduler, ScheduleQueue, SchedulePublisher, EventSubscriber, and auxiliary components (e.g., converters, domain classes/models) that manage schedule processing.
- Kafka Infrastructure:
- Input: “schedule-topics” from which schedule messages (with a key as Schedule ID and Avro encoded message values) are consumed.
- Output: The topic to which the final payload is sent at the scheduled time.
- Avro Schema Generation: The avro sub-project – includes code (e.g., avro/src/main/scala/GenerateSchema.scala) that produces the Avro schema of the Schedule case class.
- Monitoring and Metrics: Integration with Kamon and Prometheus. Kamon is configured in the code (see references in the “scheduler” directory) and Prometheus configurations are available in the docker compose files.
- Deployment/External Tools: Docker configurations (docker/, it/docker/, and associated docker-compose files) that offer a containerized deployment of the system as well as testing/IT setups.
- Build and CI/CD: sbt configuration files (build.sbt, project/Dependencies.scala, etc.) and Travis config showing continuous integration setup.

##### Relationships and Interactions
- Startup Loader: Upon startup, the system uses the kafka-topic-loader to load existing messages from Kafka, rehydrating the state within the internal scheduling actors.
– Ingestion from Kafka into Scheduler Core.
– Internal scheduling delay (with a timer or scheduling queue).
– Outgoing message publication back to Kafka.
- Avro and Serialization Components: Show that the messages are encoded/decoded using Avro, with separate converters also visible for JSON and Circe (as seen in the scheduler-3 module directories).
- Monitoring Path: Illustrate metrics being gathered via Kamon and exposed on a Prometheus endpoint (port 9095 by default), and the integration with Prometheus which scrapes these metrics.
- Optional Actor Model Subsystem: Optionally, detail that the scheduler uses the Akka actor model (as indicated by files like PublisherActor.scala, SchedulingActor.scala, TerminatorActor.scala) to manage concurrent scheduling functions.

##### Architectural Patterns or Design Principles
- The system leverages an event-driven architecture – using Kafka to decouple message ingestion and final message dispatch.
- It uses the actor model (via Akka) to manage resilience and scheduling concurrency.
- The architecture cleverly uses Kafka’s log compaction to both delete schedules after processing and to serve as a durable state store during restarts.
- There is evidence of separation of concerns: Avro schema generation is separated into its own module; configuration and dependency management are clearly modularized.

##### Relevant Technologies, Frameworks, and Libraries
- Scala and sbt for application building.
- Akka actors for concurrency and scheduling.
- Kafka as the messaging backbone.
- Avro for serialization (and corresponding converters for JSON/Circe).
- Kamon for monitoring, with Prometheus for scraping.
- Docker for deployment and integration testing.

##### Elements
- Kafka (Schedule Topics and Destination Topics)
- Scheduler Core (with internal components such as Startup Loader, Scheduling Actors, Publisher, and Queue)
- Avro Schema Generator (and associated serialization utilities)
- Monitoring (Kamon and Prometheus)
- Deployment (Docker & CI/CD)
- Data ingestion from Kafka into the scheduler.
- Internal scheduling processing and message transformation (serialization/deserialization).
- Outgoing message dispatch to Kafka.
- Periodic metrics reporting towards Prometheus.
- Data stores (Kafka topics)
- Processing components (scheduler sub-systems)
- Third-party services (Prometheus, Docker container boundaries)
- Supporting utilities (Avro generators, test suites)
- All components should have clear labels and annotations, perhaps even including key configuration notes (like scheduled message delivery rules and log compaction details).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Kafka Infrastructure
    subgraph "Kafka Infrastructure"
        KInput["Kafka Input (schedule-topics)"]:::kafka
        KOutput["Kafka Output (destination topic)"]:::kafka
    end

    %% Scheduler Core Layer
    subgraph "Scheduler Core"
        subgraph "Scheduler-3 Module"
            S3["Core Logic & Scheduling Algorithms"]:::scheduler
            S3Kafka["Kafka Integration (Sch3)"]:::scheduler
        end
        subgraph "Scheduler Module (Actor Model)"
            SActor["Actor Model Components (Scheduler, Queue, Publisher, Subscriber)"]:::scheduler
            SActKafka["Kafka Integration (Sch)"]:::scheduler
        end
        Loader["Startup Loader (kafka-topic-loader)"]:::scheduler
    end

    %% Avro and Serialization
    subgraph "Avro & Serialization"
        Avro["Avro Schema Generator"]:::serialization
        Converters["Converters (JSON, Circe)"]:::serialization
    end

    %% Monitoring and Metrics
    subgraph "Monitoring & Metrics"
        Kamon["Kamon Integration"]:::monitoring
        Prometheus["Prometheus Endpoint (port_9095)"]:::monitoring
    end

    %% Deployment and CI/CD
    subgraph "Deployment & CI/CD"
        Docker["Docker Deployment & IT Testing"]:::deployment
        CI["Build & CI/CD (sbt, Travis, Dependencies)"]:::deployment
    end

    %% Connections and Data Flow
    KInput -->|"ingests"| Loader
    Loader -->|"rehydrates_state"| S3
    Loader -->|"rehydrates_state"| SActor
    S3 -->|"processes_message"| S3Kafka
    SActor -->|"processes_message"| SActKafka
    S3Kafka -->|"publishes"| KOutput
    SActKafka -->|"publishes"| KOutput
    S3 -->|"uses_serialization"| Avro
    SActor -->|"uses_serialization"| Converters
    Kamon -->|"reports_to"| Prometheus
    S3 -->|"monitored_by"| Kamon
    SActor -->|"monitored_by"| Kamon
    Docker -->|"deploys"| S3
    Docker -->|"deploys"| SActor
    CI -->|"builds_tests"| S3
    CI -->|"builds_tests"| SActor

    %% Click Events for components
    click S3 "https://github.com/sky-uk/kafka-message-scheduler/tree/master/scheduler-3/src/main/scala/uk/sky/scheduler"
    click SActor "https://github.com/sky-uk/kafka-message-scheduler/tree/master/scheduler/src/main/scala/com/sky/kms"
    click S3Kafka "https://github.com/sky-uk/kafka-message-scheduler/tree/master/scheduler-3/src/main/scala/uk/sky/scheduler/kafka"
    click SActKafka "https://github.com/sky-uk/kafka-message-scheduler/tree/master/scheduler/src/main/scala/com/sky/kms/kafka"
    click Avro "https://github.com/sky-uk/kafka-message-scheduler/blob/master/avro/src/main/scala/GenerateSchema.scala"
    click Kamon "https://github.com/sky-uk/kafka-message-scheduler/blob/master/scheduler/src/main/scala/com/sky/kms/monitoring/KamonMonitoring.scala"
    click Docker "https://github.com/sky-uk/kafka-message-scheduler/tree/master/docker/"
    click CI "https://github.com/sky-uk/kafka-message-scheduler/blob/master/build.sbt"

    %% Styling definitions
    classDef kafka fill:#f9c74f,stroke:#333,stroke-width:2px;
    classDef scheduler fill:#90be6d,stroke:#333,stroke-width:2px;
    classDef serialization fill:#f9844a,stroke:#333,stroke-width:2px;
    classDef monitoring fill:#577590,stroke:#333,stroke-width:2px;
    classDef deployment fill:#277da1,stroke:#333,stroke-width:2px;
```
