How choudharylakshyaveer/wikievent-stream-kafka works
Project type and purpose
• This is an event-driven, microservice-style Kafka streaming project built with Spring Boot.
• It consists of two independent services: a producer that reads Wikimedia live events and pushes them to a Kafka topic, and a consumer that subscribes to that topic and persists events into a MySQL database.
Core components and their code residence
• Parent (root) Maven module (pom.xml, docker-compose.yml)
• wikievent-producer (Spring Boot app with KafkaTemplate)
• wikievent-consumer (Spring Boot app with KafkaListener and Spring Data JPA)
• External systems: Wikimedia SSE feed, Apache Kafka (broker + Zookeeper via docker-compose), MySQL
a. Main components:
• Wikimedia Event Stream (third-party SSE)
• Producer Service (wikievent-producer)
• Kafka Cluster (broker + Zookeeper)
• Consumer Service (wikievent-consumer)
• MySQL Database
1. Producer polls or listens to Wikimedia SSE → parses events
2. Producer sends messages → Kafka topic (e.g. “wikievents”)
3. Consumer subscribes to topic → receives message batches
4. Consumer writes records → MySQL via JPA repository
c. Architectural patterns / principles:
• Event‐driven messaging (decoupling producer and consumer)
• Single‐responsibility microservices (one service for produce, one for consume)
• Containerized infrastructure (Docker Compose for Kafka/Zookeeper)
• Spring Boot conventions (auto‐configuration, dependency injection)
d. Key technologies / frameworks:
• Apache Kafka (broker + Zookeeper)
• Spring Boot (producer & consumer apps)
• Spring Kafka (KafkaTemplate, @KafkaListener)
• Spring Data JPA (Consumer → MySQL)
• Maven multimodule build
• Docker Compose