How sarthak1/kafka-pain-areas works
Project Type and Purpose
• This is a backend, event-driven Spring Boot microservice focused on two pain areas in a Kafka ecosystem:
– Real-time handling of reverse movements (Store → DC).
– One-time “cut-over” processing of historical data via a REST API plus Kafka.
• It is not full “frontend” + “backend” in the UI sense; it’s a single service with:
– Kafka inbound listener
– Business logic modules
– Database persistence
– External REST API integration for cut-over
– Unit tests
Key Architectural Components
A. Ingestion Layer
– KafkaListenerService
• Subscribes to one or more Kafka topics.
• At startup, can be paused during cut-over.
B. Cut-over Startup Task
– CutoverDataService
• Triggered on app start if system.cutover.enabled=true.
• Fetches historical data via SA-Team REST API.
• (Optionally) reads from Kafka for gap days.
C. Processing Layer (Business Logic)
– MovementProcessor
• Core reverse/normal movement detection.
• Determines MovementType (NORMAL/REVERSE).
• Maps actualOrigin/actualDestination.
– LocationService
• Identifies location types (Store/DC).
• Caching, prefix‐matching, pattern rules.
– MovementValidationService
• Validates processed movements against business rules.
D. Persistence Layer
– MovementDataRepository
• Spring Data JPA repository.
• Persists MovementData entities to a relational database (PostgreSQL/MySQL).
E. Infrastructure & Configuration
– Kafka cluster (external)
– SA Team REST API (external)
– Relational DB (PostgreSQL/MySQL)
– application.properties & KafkaPainAreasConfig.java
Interactions & Data Flow
1. Application Startup
– Check system.cutover.enabled.
– If true:
· Pause KafkaListenerService
· Call CutoverDataService.fetchAndProcessHistorical()
· Persist historical data (MovementData)
· Resume KafkaListenerService
2. Real-time Kafka Processing
– KafkaListenerService receives KafkaMovementMessage
– Delegates to MovementProcessor.processMovement()
– MovementProcessor calls LocationService → MovementValidationService
– On success, MovementDataRepository.save(…)
3. External Dependencies
– Kafka cluster for real-time messages
– SA Team REST API for cut-over data
– Relational DB for persisted state
Architectural Patterns & Principles
• Event-Driven Architecture (EDA) using Kafka as the message bus
• Single Application Bounded Context (no microservices split)
• Layered architecture:
– Ingestion → Processing → Persistence
• “Startup job” pattern for cut-over
• Dependency Injection (Spring Boot) for wiring services