# kaji-sugiura/kafka-favorite-events architecture

> Architecture diagram of the GitHub repository kaji-sugiura/kafka-favorite-events, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaji-sugiura/kafka-favorite-events
- Repository: https://github.com/kaji-sugiura/kafka-favorite-events

## Overview

This project is designed as an asynchronous event-driven system that processes “favorite” events, aggregates rankings, and leverages Kafka for message passing between services. The key architectural goal is to decouple user event submission from asynchronous processing, data persistence, and aggregation – a PoC aimed at demonstrating scalability, non-blocking processing, and DLQ mechanisms.

##### Main Components
- API Server (located under the api-server directory): This service handles incoming HTTP requests, processes “like” or “favorite” events, and publishes these events to Kafka. It includes handlers (e.g., like_handler.go), models (like_event.go), and Kafka producer logic (kafka_producer.go).
- Kafka: Acts as the message broker. It is set up via Docker Compose and is the central component for asynchronous messaging.
- Consumer (found in the consumer directory): This service listens on Kafka topics, retrieves “like” events, and writes the event data into MySQL. It contains consumer logic (like_consumer.go) to process events.
- MySQL Database: Launched via Docker Compose and initialized with the provided schema (schema/init.sql), this service stores event data and aggregated results.
- Batch Job/CLI Application (in the batch directory): This Go-based batch program (aggregate.go) periodically aggregates the event data to update rankings.
- Additional Supporting Configurations and Artifacts:
- Docker Compose: Manages the orchestration and startup of Kafka and MySQL.
- Configuration Files: e.g., config/config.yml, which likely holds environment settings.
- DLQ (Dead Letter Queue) Handling: While not shown as a separate component file, DLQ functionality is mentioned in the README and should be visualized as a dedicated path within Kafka processing flows.

##### Relationships and Interactions
- The API Server is the entry point for “favorite” events. It sends events to Kafka using the Kafka producer.
- Kafka acts as the buffer between the API and downstream processing. It also handles DLQ paths for events that fail processing.
- The Consumer listens to Kafka messages and updates the MySQL database accordingly.
- The Batch job fetches data from MySQL to compute the top rankings and may update MySQL with the aggregation results.
- There is also an implicit relation for system monitoring and error handling, especially with the DLQ mechanism.

##### Architectural Patterns and Design Principles
- Asynchronous Message Processing: Utilizing Kafka decouples components enabling scalability and fault isolation.
- Layered Architecture: API server (presentation layer), Kafka (middleware), Consumer & Batch (processing layer), and MySQL (data layer).
- Event-driven Design: Internal event flows (publishing, processing, persisting) are key to handling dynamic event streams.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "API Server"
        direction TB
        APIMain["main.go (Entry Point)"]:::api
        APIHandler["HTTP Handler (like_handler.go)"]:::api
        APIModel["Model (like_event.go)"]:::api
        APIProducer["Kafka Producer (kafka_producer.go)"]:::api
    end

    Kafka["Kafka Broker<br/>(with DLQ)"]:::kafka
    DLQ["Dead Letter Queue (DLQ)"]:::kafka
    subgraph "Consumer Service"
        direction TB
        ConsumerMain["main.go (Entry)"]:::consumer
        ConsumerLogic["Consumer Logic (like_consumer.go)"]:::consumer
    end
    MySQL["MySQL Database"]:::mysql
    Batch["Batch Job (aggregate.go)"]:::batch
    Config["Configuration<br/>(config/config.yml)"]:::config
    Compose["Docker Compose<br/>(Orchestration)"]:::management

    %% Connections
    APIMain -->|"EventPublish"| Kafka
    APIHandler -->|"EventPublish"| Kafka
    APIModel -->|"EventPublish"| Kafka
    APIProducer -->|"EventPublish"| Kafka

    Kafka -->|"DLQrouting"| DLQ
    Kafka -->|"EventConsume"| ConsumerMain
    ConsumerLogic -->|"DataInsert/Update"| MySQL
    ConsumerMain -->|"DataInsert/Update"| MySQL
    MySQL -->|"DataAggregation"| Batch
    Batch -->|"UpdateRankings"| MySQL

    %% Docker Compose manages Kafka and MySQL
    Compose ---|"Orchestration"| Kafka
    Compose ---|"Orchestration"| MySQL

    %% Config file usage by API Server and Batch Job
    Config ---|"Settings"| APIProducer
    Config ---|"Settings"| Batch

    %% Click Events for API Server Components
    click APIMain "https://github.com/kaji-sugiura/kafka-favorite-events/blob/main/api-server/main.go"
    click APIHandler "https://github.com/kaji-sugiura/kafka-favorite-events/blob/main/api-server/handler/like_handler.go"
    click APIModel "https://github.com/kaji-sugiura/kafka-favorite-events/blob/main/api-server/model/like_event.go"
    click APIProducer "https://github.com/kaji-sugiura/kafka-favorite-events/blob/main/api-server/producer/kafka_producer.go"

    %% Click Events for Kafka & Docker Compose (note: DLQ is conceptual within Kafka)
    click Compose "https://github.com/kaji-sugiura/kafka-favorite-events/blob/main/docker-compose.yml"

    %% Click Events for Consumer Components
    click ConsumerMain "https://github.com/kaji-sugiura/kafka-favorite-events/blob/main/consumer/main.go"
    click ConsumerLogic "https://github.com/kaji-sugiura/kafka-favorite-events/blob/main/consumer/consumer/like_consumer.go"

    %% Click Event for MySQL Database initialization
    click MySQL "https://github.com/kaji-sugiura/kafka-favorite-events/blob/main/schema/init.sql"

    %% Click Event for Batch Job
    click Batch "https://github.com/kaji-sugiura/kafka-favorite-events/blob/main/batch/aggregate.go"

    %% Click Event for Configuration File
    click Config "https://github.com/kaji-sugiura/kafka-favorite-events/blob/main/config/config.yml"

    %% Styles
    classDef api fill:#AED6F1,stroke:#1f77b4,stroke-width:2px;
    classDef kafka fill:#58D68D,stroke:#2ca02c,stroke-width:2px;
    classDef consumer fill:#F5B7B1,stroke:#e377c2,stroke-width:2px;
    classDef mysql fill:#F7DC6F,stroke:#ff7f0e,stroke-width:2px;
    classDef batch fill:#D2B4DE,stroke:#9467bd,stroke-width:2px;
    classDef management fill:#E59866,stroke:#8c564b,stroke-width:2px;
    classDef config fill:#A3E4D7,stroke:#17becf,stroke-width:2px;
```
