How happening-oss/kafka-client works
Main Components
• Elixir Application Layer:
• Core modules located in the lib directory (e.g., KafkaClient.Producer, KafkaClient.Consumer, KafkaClient.Consumer.Poller, KafkaClient.Consumer.Stream).
• Supporting modules like KafkaClient.gen_port, which likely manage inter-process communication (via ports) between BEAM and Java.
• Test and configuration components demonstrate the structure and expected behaviors.
• Java Port Layer:
• The “port” directory contains a complete Java project (using Maven as indicated by pom.xml) with source files under port/src/main/java/com/happening/kafka. This component encapsulates the actual calls to the Apache Kafka Java client (with subpackages admin, consumer, producer, and utility classes).
• External Kafka Cluster:
• Represented externally as the actual Kafka deployment, which can be initiated using the included docker-compose.yml file.
• CI/CD and DevOps Tools:
• The CI configuration in .github/workflows indicates automated testing/build processes.
• The existence of docker-compose and configuration files (config/config.exs) shows deployment and environment management considerations.
Relationships and Interactions
• The Elixir modules form the entry point for users and application logic; they call into the Java Port via the gen_port abstraction.
• The Java Port then makes synchronous/asynchronous calls to the Kafka cluster using the official Kafka client APIs.
Architectural Patterns and Principles
• Asynchronous and Concurrent Processing: For example, the Elixir consumer uses finer-grained concurrency to allow for at-least-once processing and backpressure management.
• External Dependency Integration: Show the Kafka cluster as an external, third-party dependency.
Specific Technologies and Tools
• Elixir/BEAM details: Modules, concurrency primitives, and the overarching supervision model.
• Java details: The use of the Apache Kafka Java client library along with Maven build management.
• Docker: Represent the Kafka cluster deployment facilitated by docker-compose.
• CI: Indicate the automated workflows (e.g., GitHub workflows, tests) that support continuous integration.