# schmackes/ps-getting-started-kafka architecture

> Architecture diagram of the GitHub repository schmackes/ps-getting-started-kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/schmackes/ps-getting-started-kafka
- Repository: https://github.com/schmackes/ps-getting-started-kafka

## Overview

##### Main Components
- Kafka Cluster: At the heart of the project, the Kafka cluster acts as the central messaging system. Although not explicitly defined with cluster configuration files here, it is implicit due to the “Kafka” theme.
- Producer/Consumer Services: Most modules contain Java applications (e.g., Main.java, Producer.java, Consumer.java, LogProducer.java, SimpleETL.java). These represent the Kafka producers and consumers. Some demos (like module6 and module8) additionally show AVRO usage or include security considerations.
- Docker Containers: Each demo utilizes Docker (docker-compose.yaml) to orchestrate containerized components (which likely include Kafka brokers, Zookeeper nodes, and the client applications).
- Configuration and Support Files: There are various configuration resources like pom.xml files (for Java/Maven dependencies), property files for workers and file sinks, and even JSON files (e.g., increase_replication.json) that tweak Kafka settings.
- Security Components: In module8 (and parts of module8’s demos), the presence of security folders with shell scripts (generate-ca.sh, generate-keystore.sh, generate-truststore.sh) suggests SSL/TLS or other security configurations are being demonstrated.

##### Relationships and Interactions
- Each demo module is an independent scenario that deploys a subset of components. For instance, one demo might show a simple Kafka producer-consumer connection while another might focus on secure communication or ETL processing.
- Shows communication flow from producers to the Kafka cluster and then from Kafka to the consumers.
- Where applicable, include interactions showing enhanced processes – for example, the ETL demo in module7 illustrating data transformation.
- The docker-compose files in each module likely start up a set of interdependent services (such as Kafka brokers, Zookeeper, and the Java client apps) that cooperate within that demo.

##### Important Architectural Patterns and Design Principles
- Containerization: The use of Docker Compose across demos reflects a microservice-like, loosely coupled design for experimenting with different Kafka configurations.
- Separation of Concerns: Notice that producers, consumers, and in some cases ETL components are modularized into separate folders and demos. This separation reinforces the concept of message-driven or event-driven architecture.
- Use of standard frameworks: With Maven (pom.xml) and Java source code structure present, it’s clear that the demos are built using common Java best practices. The presence of Avro schema files shows a focus on consistent data serialization.
- Security Best Practices (in module8): The inclusion of keystore/truststore generation scripts indicates a practical exploration of Kafka’s security configurations.

- Interplay between containerized environments (using docker-compose orientation) and the internal Java applications.
- To note any critical dependencies (like the Avro schema in module6 or security scripts in module8) that are unique to certain demos.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Module3 Demo"
        M3_DC["Docker Compose (module3)"]:::docker
        M3_Config["Replication Config (increase_replication.json)"]:::config
    end

    subgraph "Module4 Demo"
        M4_DC["Docker Compose (module4)"]:::docker
        M4_POM["Maven POM"]:::maven
    end

    subgraph "Module5 Demo2"
        M5_DC["Docker Compose (module5)"]:::docker
        M5_Producer["Producer: Producer.java"]:::producer
        M5_Consumer["Consumer: Consumer.java"]:::consumer
        M5_POM["Maven POM"]:::maven
    end

    subgraph "Module6 Demo"
        M6_DC["Docker Compose (module6)"]:::docker
        M6_POM["Maven POM"]:::maven
        M6_FSink["File Sink Properties"]:::config
        M6_Worker["Worker Properties"]:::config
        M6_Avro["Avro Schema: Album.avsc"]:::config
    end

    subgraph "Module7 Demo"
        M7_DC["Docker Compose (module7)"]:::docker
        M7_ETL["ETL: SimpleETL.java"]:::etl
        M7_POM["Maven POM"]:::maven
    end

    subgraph "Module8 Demo"
        M8_DC["Docker Compose (module8)"]:::docker
        M8_Producer["Producer: Producer.java"]:::producer
        M8_Consumer["Consumer: Consumer.java"]:::consumer
        M8_POM["Maven POM"]:::maven
        subgraph "Module8 Security"
            M8_sec1["generate-ca.sh"]:::security
            M8_sec2["generate-keystore.sh"]:::security
            M8_sec3["generate-truststore.sh"]:::security
        end
    end

    Kafka["Apache Kafka Cluster"]:::kafka

    %% Deployment: Docker Compose services linking to the Kafka Cluster
    M3_DC -.->|"deploy"| Kafka
    M4_DC -.->|"deploy"| Kafka
    M5_DC -.->|"deploy"| Kafka
    M6_DC -.->|"deploy"| Kafka
    M7_DC -.->|"deploy"| Kafka
    M8_DC -.->|"deploy"| Kafka

    %% Maven dependencies feeding into Docker deployment
    M4_POM --> M4_DC
    M5_POM --> M5_DC
    M6_POM --> M6_DC
    M7_POM --> M7_DC
    M8_POM --> M8_DC

    %% Module6 configuration files link to its Docker Compose
    M6_FSink --> M6_DC
    M6_Worker --> M6_DC
    M6_Avro --> M6_DC

    %% Module8 security scripts connect to its Docker Compose service
    M8_sec1 --> M8_DC
    M8_sec2 --> M8_DC
    M8_sec3 --> M8_DC

    %% Communication: Producers publishing messages to the Kafka Cluster
    M5_Producer -->|"publish"| Kafka
    M8_Producer -->|"publish"| Kafka

    %% Communication: Kafka Cluster delivering messages to Consumers
    Kafka -->|"deliver"| M5_Consumer
    Kafka -->|"deliver"| M8_Consumer

    %% ETL service consuming messages from Kafka Cluster
    Kafka -->|"consume"| M7_ETL

    %% Module3 configuration update affecting Kafka
    M3_Config -->|"config"| Kafka

    %% Internal module ordering
    M3_DC --> M3_Config
    M4_DC --> M4_POM
    M5_DC --> M5_Producer
    M5_DC --> M5_Consumer
    M6_DC --> M6_POM
    M7_DC --> M7_ETL
    M7_DC --> M7_POM
    M8_DC --> M8_Producer
    M8_DC --> M8_Consumer
    M8_DC --> M8_POM

click M3_DC "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module3/demo1/docker-compose.yaml"
click M3_Config "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module3/demo3/increase_replication.json"
click M4_DC "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module4/demo1/docker-compose.yaml"
click M4_POM "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module4/demo2/pom.xml"
click M5_DC "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module5/demo2/docker-compose.yaml"
click M5_Producer "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module5/demo2/src/main/java/com/globomantics/Producer.java"
click M5_Consumer "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module5/demo2/src/main/java/com/globomantics/Consumer.java"
click M5_POM "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module5/demo1/pom.xml"
click M6_DC "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module6/demo1/docker-compose.yaml"
click M6_POM "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module6/demo1/pom.xml"
click M6_FSink "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module6/demo1/filesink.properties"
click M6_Worker "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module6/demo1/worker.properties"
click M6_Avro "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module6/demo2/src/main/avro/Album.avsc"
click M7_DC "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module7/demo1/docker-compose.yaml"
click M7_ETL "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module7/demo1/src/main/java/com/globomantics/SimpleETL.java"
click M7_POM "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module7/demo1/pom.xml"
click M8_DC "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module8/demo1/docker-compose.yaml"
click M8_Producer "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module8/demo1/clients/src/main/java/com/globomantics/Producer.java"
click M8_Consumer "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module8/demo1/clients/src/main/java/com/globomantics/Consumer.java"
click M8_POM "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module8/demo1/clients/pom.xml"
click M8_sec1 "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module8/demo1/security/generate-ca.sh"
click M8_sec2 "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module8/demo1/security/generate-keystore.sh"
click M8_sec3 "https://github.com/schmackes/ps-getting-started-kafka/blob/main/module8/demo1/security/generate-truststore.sh"

classDef kafka fill:#f4cccc,stroke:#b45f06,stroke-width:2px;
classDef docker fill:#d9ead3,stroke:#38761d,stroke-width:2px;
classDef producer fill:#c9daf8,stroke:#1155cc,stroke-width:2px;
classDef consumer fill:#c9daf8,stroke:#1155cc,stroke-width:2px;
classDef maven fill:#fff2cc,stroke:#bf9000,stroke-width:2px;
classDef etl fill:#ead1dc,stroke:#b854a1,stroke-width:2px;
classDef config fill:#d0e0e3,stroke:#5f6b6d,stroke-width:2px;
classDef security fill:#fce5cd,stroke:#b45f06,stroke-width:2px;
```
