How xebia-functional/poc-scala-data-streaming works
Main Components
• Core Domain & Models: (Module “01-c-domain”)
– Contains definitions for business models and codecs that are used across the system.
• API & Configuration:
– “02-c-api” provides common API endpoints and configuration interfaces.
– “02-i-api” and “02-o-api” offer specific APIs for input (e.g., Kafka Consumer APIs) and output (e.g., Flink and Spark Processor APIs).
– “03-c-config-ciris” and “03-c-config-pureconfig” contain configuration loaders, following different configuration libraries.
• Data Generation: (Module “03-u-data-generator”)
– Acts as a dummy data generator that produces events and pushes them to Kafka topics.
• Data Ingestion: (Module “04-i-consumer-kafka”)
– This component consumes events from Kafka, translating raw stream data into domain-specific events using codecs.
• Data Processing:
– Apache Flink Processor: (Module “04-o-processor-flink”)
• Processes streaming data in real time.
– Apache Spark Processor: (Module “04-o-processor-spark”)
• Processes or further analyzes the streaming data via Spark’s streaming engine.
• Application Entry Points: (Module “05-c-main”)
– Contains main classes to kick off one or more service types (data generation, configuration based runs, or processor launches).
• External Systems:
– Apache Kafka: Acts as an event bus facilitating communication among modules.
– Docker & Container Orchestration: Provided via various docker-compose configurations for Kafka, Spark, and data generator environments.
Relationships and Data Flow
• Data Generation → Kafka Broker
– The “03-u-data-generator” module produces events (for example, GPS or PP events) that are published to specific Kafka topics.
• Kafka as the Central Event Bus
– Multiple topics (as shown in the README sample output) are maintained (like “data-generator-gps”) to which events are published.
• Ingestion → Processing
– “04-i-consumer-kafka” consumes events from Kafka.
– The consumed events are then fed to processing modules (both Flink and Spark) via the API / output adapters located in “02-o-api.”
• Processing Modules (Flink and Spark)
– Both “04-o-processor-flink” and “04-o-processor-spark” subscribe to Kafka topics (and possibly get additional configuration from the API modules) to handle event processing.
• Domain Logic and Configurations
– All modules use definitions from the core domain (“01-c-domain”) ensuring consistency in data models and codecs.
– Configuration modules (“03-c-config-ciris” and “03-c-config-pureconfig”) provide the necessary setup for various processing aspects (e.g., Kafka consumer configurations, Flink and Spark settings).
• Application Orchestration
– The “05-c-main” module acts as the entry point to run the entire application, coordinating between data generation and processors.
Architectural Patterns and Design Principles
• Diamond (Hexagonal) Architecture:
– Clearly separates the core domain (business logic and models) from the infrastructure and adapter layers (APIs, configuration, data ingestion/processing).
• Kappa Architecture:
– Emphasize that the pipeline processes immutable event streams (published by the Data Generator and consumed by both processing engines).
• Micro-modular Design:
– Each module is dedicated to a specific concern (data generation, consumption, transformation, processing) and they interact via well-defined APIs and event streams.