How kaji-sugiura/kafka-favorite-events works
This project is designed as an asynchronous event-driven system that processes “favorite” events, aggregates rankings, and leverages Kafka for message passing between services. The key architectural goal is to decouple user event submission from asynchronous processing, data persistence, and aggregation – a PoC aimed at demonstrating scalability, non-blocking processing, and DLQ mechanisms.
Main Components
• API Server (located under the api-server directory): This service handles incoming HTTP requests, processes “like” or “favorite” events, and publishes these events to Kafka. It includes handlers (e.g., like_handler.go), models (like_event.go), and Kafka producer logic (kafka_producer.go).
• Kafka: Acts as the message broker. It is set up via Docker Compose and is the central component for asynchronous messaging.
• Consumer (found in the consumer directory): This service listens on Kafka topics, retrieves “like” events, and writes the event data into MySQL. It contains consumer logic (like_consumer.go) to process events.
• MySQL Database: Launched via Docker Compose and initialized with the provided schema (schema/init.sql), this service stores event data and aggregated results.
• Batch Job/CLI Application (in the batch directory): This Go-based batch program (aggregate.go) periodically aggregates the event data to update rankings.
• Additional Supporting Configurations and Artifacts:
• Docker Compose: Manages the orchestration and startup of Kafka and MySQL.
• Configuration Files: e.g., config/config.yml, which likely holds environment settings.
• DLQ (Dead Letter Queue) Handling: While not shown as a separate component file, DLQ functionality is mentioned in the README and should be visualized as a dedicated path within Kafka processing flows.
Relationships and Interactions
• The API Server is the entry point for “favorite” events. It sends events to Kafka using the Kafka producer.
• Kafka acts as the buffer between the API and downstream processing. It also handles DLQ paths for events that fail processing.
• The Consumer listens to Kafka messages and updates the MySQL database accordingly.
• The Batch job fetches data from MySQL to compute the top rankings and may update MySQL with the aggregation results.
• There is also an implicit relation for system monitoring and error handling, especially with the DLQ mechanism.
Architectural Patterns and Design Principles
• Asynchronous Message Processing: Utilizing Kafka decouples components enabling scalability and fault isolation.
• Layered Architecture: API server (presentation layer), Kafka (middleware), Consumer & Batch (processing layer), and MySQL (data layer).
• Event-driven Design: Internal event flows (publishing, processing, persisting) are key to handling dynamic event streams.