# happening-oss/kafka-client architecture

> Architecture diagram of the GitHub repository happening-oss/kafka-client, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/happening-oss/kafka-client
- Repository: https://github.com/happening-oss/kafka-client

## Overview

##### Main Components
- Elixir Application Layer:
- Core modules located in the lib directory (e.g., KafkaClient.Producer, KafkaClient.Consumer, KafkaClient.Consumer.Poller, KafkaClient.Consumer.Stream).
- Supporting modules like KafkaClient.gen_port, which likely manage inter-process communication (via ports) between BEAM and Java.
- Test and configuration components demonstrate the structure and expected behaviors.
- Java Port Layer:
- The “port” directory contains a complete Java project (using Maven as indicated by pom.xml) with source files under port/src/main/java/com/happening/kafka. This component encapsulates the actual calls to the Apache Kafka Java client (with subpackages admin, consumer, producer, and utility classes).
- External Kafka Cluster:
- Represented externally as the actual Kafka deployment, which can be initiated using the included docker-compose.yml file.
- CI/CD and DevOps Tools:
- The CI configuration in .github/workflows indicates automated testing/build processes.
- The existence of docker-compose and configuration files (config/config.exs) shows deployment and environment management considerations.

##### Relationships and Interactions
- The Elixir modules form the entry point for users and application logic; they call into the Java Port via the gen_port abstraction.
- The Java Port then makes synchronous/asynchronous calls to the Kafka cluster using the official Kafka client APIs.

##### Architectural Patterns and Principles
- Asynchronous and Concurrent Processing: For example, the Elixir consumer uses finer-grained concurrency to allow for at-least-once processing and backpressure management.
- External Dependency Integration: Show the Kafka cluster as an external, third-party dependency.

##### Specific Technologies and Tools
- Elixir/BEAM details: Modules, concurrency primitives, and the overarching supervision model.
- Java details: The use of the Apache Kafka Java client library along with Maven build management.
- Docker: Represent the Kafka cluster deployment facilitated by docker-compose.
- CI: Indicate the automated workflows (e.g., GitHub workflows, tests) that support continuous integration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Layer: External Kafka Cluster
    subgraph "External Kafka Cluster"
        KafkaCluster["Kafka Broker Cluster"]:::kafka
    end

    %% Middle Layer: Java Port Layer
    subgraph "Java Port Layer"
        JavaBridge["Java Port/Bridge"]:::java
        JavaProducer["Java Producer"]:::java
        JavaConsumer["Java Consumer"]:::java
        JavaAdmin["Java Admin"]:::java
    end

    %% Bottom Layer: Elixir API Layer
    subgraph "Elixir API Layer"
        ElixirCore["KafkaClient Core"]:::elixir
        ElixirProducer["Producer"]:::elixir
        ElixirConsumer["Consumer"]:::elixir
        ElixirAdmin["Admin"]:::elixir
        ElixirGenPort["GenPort"]:::elixir
        ConsumerPoller["Poller"]:::elixir
        ConsumerStream["Stream"]:::elixir
        ConsumerPartitionProcessor["Partition Processor"]:::elixir
    end

    %% Auxiliary: CI/CD Pipeline
    subgraph "CI/CD Pipeline"
        CICD["GitHub Actions"]:::ci
    end

    %% Elixir to GenPort connections (Producer, Consumer, Admin)
    ElixirProducer --> ElixirGenPort
    ElixirAdmin --> ElixirGenPort
    ElixirConsumer --> ElixirGenPort

    %% GenPort <--> Java Port/Bridge (bidirectional communication)
    ElixirGenPort --> JavaBridge
    JavaBridge --> ElixirGenPort

    %% Java Bridge to specific Java components
    JavaBridge --> JavaProducer
    JavaBridge --> JavaConsumer
    JavaBridge --> JavaAdmin

    %% Producer message flow: Java Producer to Kafka Cluster
    JavaProducer -->|"send"| KafkaCluster

    %% Consumer message flow: Kafka Cluster to Java Consumer, then to Elixir Consumer
    KafkaCluster -->|"deliver"| JavaConsumer
    JavaConsumer --> ElixirGenPort
    ElixirGenPort --> ElixirConsumer

    %% Elixir Consumer internal components
    ElixirConsumer --> ConsumerPoller
    ElixirConsumer --> ConsumerStream
    ElixirConsumer --> ConsumerPartitionProcessor

    %% Click Events for Elixir Components
    click ElixirCore "https://github.com/happening-oss/kafka-client/blob/develop/lib/kafka_client.ex"
    click ElixirProducer "https://github.com/happening-oss/kafka-client/blob/develop/lib/kafka_client/producer.ex"
    click ElixirConsumer "https://github.com/happening-oss/kafka-client/blob/develop/lib/kafka_client/consumer.ex"
    click ElixirAdmin "https://github.com/happening-oss/kafka-client/blob/develop/lib/kafka_client/admin.ex"
    click ElixirGenPort "https://github.com/happening-oss/kafka-client/blob/develop/lib/kafka_client/gen_port.ex"
    click ConsumerPoller "https://github.com/happening-oss/kafka-client/blob/develop/lib/kafka_client/consumer/poller.ex"
    click ConsumerStream "https://github.com/happening-oss/kafka-client/blob/develop/lib/kafka_client/consumer/stream.ex"
    click ConsumerPartitionProcessor "https://github.com/happening-oss/kafka-client/blob/develop/lib/kafka_client/consumer/partition_processor.ex"

    %% Click Events for Java Components
    click JavaBridge "https://github.com/happening-oss/kafka-client/tree/develop/port/src/main/java/com/happening/kafka/port"
    click JavaProducer "https://github.com/happening-oss/kafka-client/tree/develop/port/src/main/java/com/happening/kafka/producer"
    click JavaConsumer "https://github.com/happening-oss/kafka-client/tree/develop/port/src/main/java/com/happening/kafka/consumer"
    click JavaAdmin "https://github.com/happening-oss/kafka-client/tree/develop/port/src/main/java/com/happening/kafka/admin"

    %% Click Event for External Kafka Cluster Deployment
    click KafkaCluster "https://github.com/happening-oss/kafka-client/blob/develop/docker-compose.yml"

    %% Click Event for CI/CD Pipeline
    click CICD "https://github.com/happening-oss/kafka-client/blob/develop/.github/workflows/ci.yml"

    %% Styling Classes
    classDef elixir fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef java fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef kafka fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef ci fill:#FFA500,stroke:#333,stroke-width:2px;
```
