# amine759/kappa-architecture-spark-kafka-ek architecture

> Architecture diagram of the GitHub repository amine759/kappa-architecture-spark-kafka-ek, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amine759/kappa-architecture-spark-kafka-ek
- Repository: https://github.com/amine759/kappa-architecture-spark-kafka-ek

## Overview

##### Main Components
- Producer Service: Located under the “kappa/producer” directory. This Scala-based service reads configuration (see Config.scala) and uses a WebSocket-to-Kafka producer (WebSocketToKafkaProducer.scala) to push data into Kafka.
- Kafka Cluster: Not explicitly in the source code (Docker Compose handles this) but it is critical since the producer writes to a Kafka topic (e.g., “main-topic”) and consumers later pull from it.
- Spark Streaming Application: Found under “kappa/spark-streaming-app”. Its main class (Main.scala under sparkstreaming) is responsible for consuming Kafka data, processing it, and likely performing the necessary transformations before output.
- Elasticsearch: Acts as the storage engine where processed tweet or trade data is stored, as indicated by the instructions that include verifying data in an index (“trades-index”).
- Kibana: Provides a visualization layer for monitoring the data in Elasticsearch.
- Docker Compose Environment: The “docker-compose.yaml” file encapsulates the services, including Kafka, Spark master (and likely worker), Elasticsearch, Kibana, and any other supporting services.

##### Relationships and Interactions
- Data Flow: The producer sends data to Kafka (this is the entry point). Next, the Spark streaming app consumes this data from the Kafka topic, processes it, and writes the results to Elasticsearch.
- Service Orchestration: Docker Compose is responsible for setting up and linking these services, ensuring that container networks allow necessary communication between Kafka, Spark, Elasticsearch, and Kibana.
- External Interactions: Additionally, there might be interactions with external services (e.g., a WebSocket source for the producer) if applicable.

##### Architectural Patterns and Design Principles
- Microservices/Containerized Architecture: Each main service is containerized (as seen with Dockerfiles and the docker-compose.yml file), making the components loosely coupled.
- Kappa Architecture Principles: The approach uses a single stream processing engine (Spark Streaming) to process real-time data without a separate batch layer.
- Modular Scala Projects: The codebase is split into subprojects (e.g., the "producer" and "spark-streaming-app") under a common Scala build (sbt), ensuring separation of concerns.

##### Key Technologies and Frameworks
- Scala and sbt: For building the producer and Spark streaming jobs.
- Docker and Docker Compose: To package and orchestrate the multi-container environment.
- Apache Kafka: For message brokering between the producer and the Spark streaming consumer.
- Apache Spark: For the real-time stream processing application.
- Elasticsearch and Kibana: For storing and visualizing processed data.

##### Elements
- Clear labels for each component: Producer, Kafka, Spark Stream Processing, Elasticsearch, and Kibana.
- Data flow from the Producer to Kafka.
- The Spark Streaming App consuming from Kafka.
- Processed data being stored in Elasticsearch.
- Kibana reading data from Elasticsearch.
- An indication of Docker Compose’s role by showing a container orchestration block that surrounds/links these services.

##### Final Recommendations
- Keep details precise—include network flows, port configurations if needed, and note any dependency (e.g., the Spark job depends on the Kafka topic being available).
- Incorporate notes from the README regarding deployment and execution instructions, as these often hint at runtime relationships (e.g., the role of deploy scripts and docker commands).

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Docker Compose"]:::orchestration
    click A "https://github.com/amine759/kappa-architecture-spark-kafka-ek/blob/main/docker-compose.yaml"

    subgraph "Containerized Services"
        B["Producer Service"]:::compute
        click B "https://github.com/amine759/kappa-architecture-spark-kafka-ek/tree/main/kappa/producer/src/main/scala/producer"
        C["Kafka Cluster"]:::messaging
        D["Spark Streaming App"]:::compute
        click D "https://github.com/amine759/kappa-architecture-spark-kafka-ek/blob/main/kappa/spark-streaming-app/src/main/scala/sparkstreaming/Main.scala"
        E["Elasticsearch"]:::storage
        F["Kibana"]:::visual
    end

    %% Docker Compose manages all containerized services
    A ---|"manages"| B
    A ---|"manages"| C
    A ---|"manages"| D
    A ---|"manages"| E
    A ---|"manages"| F

    %% Data flow
    B -->|"sendsDataTo"| C
    C -->|"consumedBy"| D
    D -->|"writesTo"| E
    E -->|"visualizedBy"| F

    %% Styles
    classDef orchestration fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef compute fill:#AED6F1,stroke:#2471A3,stroke-width:2px;
    classDef messaging fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef storage fill:#ABEBC6,stroke:#27AE60,stroke-width:2px;
    classDef visual fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
```
