# xebia-functional/poc-scala-data-streaming architecture

> Architecture diagram of the GitHub repository xebia-functional/poc-scala-data-streaming, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xebia-functional/poc-scala-data-streaming
- Repository: https://github.com/xebia-functional/poc-scala-data-streaming

## Overview

##### Main Components
- Core Domain & Models: (Module “01-c-domain”)
– Contains definitions for business models and codecs that are used across the system.
- API & Configuration:
– “02-c-api” provides common API endpoints and configuration interfaces.
– “02-i-api” and “02-o-api” offer specific APIs for input (e.g., Kafka Consumer APIs) and output (e.g., Flink and Spark Processor APIs).
– “03-c-config-ciris” and “03-c-config-pureconfig” contain configuration loaders, following different configuration libraries.
- Data Generation: (Module “03-u-data-generator”)
– Acts as a dummy data generator that produces events and pushes them to Kafka topics.
- Data Ingestion: (Module “04-i-consumer-kafka”)
– This component consumes events from Kafka, translating raw stream data into domain-specific events using codecs.
- Data Processing:
– Apache Flink Processor: (Module “04-o-processor-flink”)
- Processes streaming data in real time.
– Apache Spark Processor: (Module “04-o-processor-spark”)
- Processes or further analyzes the streaming data via Spark’s streaming engine.
- Application Entry Points: (Module “05-c-main”)
– Contains main classes to kick off one or more service types (data generation, configuration based runs, or processor launches).
- External Systems:
– Apache Kafka: Acts as an event bus facilitating communication among modules.
– Docker & Container Orchestration: Provided via various docker-compose configurations for Kafka, Spark, and data generator environments.

##### Relationships and Data Flow
- Data Generation → Kafka Broker
– The “03-u-data-generator” module produces events (for example, GPS or PP events) that are published to specific Kafka topics.
- Kafka as the Central Event Bus
– Multiple topics (as shown in the README sample output) are maintained (like “data-generator-gps”) to which events are published.
- Ingestion → Processing
– “04-i-consumer-kafka” consumes events from Kafka.
– The consumed events are then fed to processing modules (both Flink and Spark) via the API / output adapters located in “02-o-api.”
- Processing Modules (Flink and Spark)
– Both “04-o-processor-flink” and “04-o-processor-spark” subscribe to Kafka topics (and possibly get additional configuration from the API modules) to handle event processing.
- Domain Logic and Configurations
– All modules use definitions from the core domain (“01-c-domain”) ensuring consistency in data models and codecs.
– Configuration modules (“03-c-config-ciris” and “03-c-config-pureconfig”) provide the necessary setup for various processing aspects (e.g., Kafka consumer configurations, Flink and Spark settings).
- Application Orchestration
– The “05-c-main” module acts as the entry point to run the entire application, coordinating between data generation and processors.

##### Architectural Patterns and Design Principles
- Diamond (Hexagonal) Architecture:
– Clearly separates the core domain (business logic and models) from the infrastructure and adapter layers (APIs, configuration, data ingestion/processing).
- Kappa Architecture:
– Emphasize that the pipeline processes immutable event streams (published by the Data Generator and consumed by both processing engines).
- Micro-modular Design:
– Each module is dedicated to a specific concern (data generation, consumption, transformation, processing) and they interact via well-defined APIs and event streams.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Domain & Config Cluster
    subgraph "Domain & Config"
        DC["Core Domain (01-c-domain)"]:::internal
        APIConfig["Common API & Config Interfaces (02-c-api)"]:::internal
        ConfigCiris["Config via Ciris (03-c-config-ciris)"]:::internal
        ConfigPure["Config via Pureconfig (03-c-config-pureconfig)"]:::internal
        DC --> APIConfig
    end

    %% API Adapters Cluster
    subgraph "API Adapters"
        InputAPI["Input API (02-i-api)"]:::internal
        OutputAPI["Output API (02-o-api)"]:::internal
        APIConfig --> InputAPI
        APIConfig --> OutputAPI
    end

    %% Data Handling Cluster
    subgraph "Data Handling"
        DataGen["Data Generator (03-u-data-generator)"]:::internal
        KafkaConsumer["Kafka Consumer (04-i-consumer-kafka)"]:::internal
        InputAPI --> KafkaConsumer
    end

    %% Data Processing Cluster
    subgraph "Data Processing"
        FlinkProc["Flink Processor (04-o-processor-flink)"]:::internal
        SparkProc["Spark Processor (04-o-processor-spark)"]:::internal
        OutputAPI --> FlinkProc
        OutputAPI --> SparkProc
    end

    %% External Systems Cluster
    subgraph "External Systems"
        Kafka["Apache Kafka"]:::external
        Docker["Docker Orchestration"]:::external
    end

    %% Application Entry Point
    App["Application Entry (05-c-main)"]:::internal

    %% Data Flow Connections
    App -->|"starts"| DataGen
    App -->|"starts"| KafkaConsumer
    App -->|"starts"| FlinkProc
    App -->|"starts"| SparkProc

    DataGen -->|"publishes"| Kafka
    Kafka -->|"delivers"| KafkaConsumer
    KafkaConsumer -->|"forwards"| OutputAPI

    %% Configuration Connections
    ConfigCiris -->|"configures"| KafkaConsumer
    ConfigCiris -->|"configures"| FlinkProc
    ConfigCiris -->|"configures"| SparkProc
    ConfigPure -->|"configures"| KafkaConsumer
    ConfigPure -->|"configures"| FlinkProc
    ConfigPure -->|"configures"| SparkProc

    %% Docker Orchestration Connections
    Docker -->|"orchestrates"| Kafka
    Docker -->|"orchestrates"| FlinkProc
    Docker -->|"orchestrates"| SparkProc

    %% Core Domain Dependencies
    DC ---|"defines"| KafkaConsumer
    DC ---|"defines"| FlinkProc
    DC ---|"defines"| SparkProc

    %% Styles
    classDef internal fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef external fill:#FCF3CF,stroke:#F1C40F,stroke-width:2px;

    %% Click Events
    click DC "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/01-c-domain"
    click APIConfig "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/02-c-api"
    click InputAPI "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/02-i-api"
    click OutputAPI "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/02-o-api"
    click ConfigCiris "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/03-c-config-ciris"
    click ConfigPure "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/03-c-config-pureconfig"
    click DataGen "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/03-u-data-generator"
    click KafkaConsumer "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/04-i-consumer-kafka"
    click FlinkProc "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/04-o-processor-flink"
    click SparkProc "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/04-o-processor-spark"
    click App "https://github.com/xebia-functional/poc-scala-data-streaming/tree/main/05-c-main"
    click Docker "https://github.com/xebia-functional/poc-scala-data-streaming/blob/main/docker/docker-compose-generator.yml"
```
