# mkroli/kafka-proxy architecture

> Architecture diagram of the GitHub repository mkroli/kafka-proxy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mkroli/kafka-proxy
- Repository: https://github.com/mkroli/kafka-proxy

## Overview

##### Project Type and Purpose
- Type: This is a protocol-proxy service (sidecar) meant to serve as an intermediary between different messaging protocols and Kafka.
- Purpose: It gathers messages from various endpoints, optionally translates or transforms them (for example, when handling Avro with a schema registry), and then dispatches them to Kafka.

##### Analyzing the File Structure
- Top-Level Organization:
– The src directory contains the application’s code.
– There are specific folders for CLI commands (src/cli) that parse command-line arguments and dispatch to functionality.
– The kafka folder inside src holds Kafka-specific modules, including a producer, schema registry support, and serialization (serde) utilities to manage message conversion.
– The server folder encapsulates the protocol-specific code, with distinct files or modules (e.g., coap, rest, stream) for handling incoming message streams.
- Additional files (Docker, GitHub workflows, and examples) indicate a well-tested, container-friendly environment with deployment automation.

##### Insights from the README
- The README clarifies the supported protocols and usage patterns.
- It explicitly outlines that the service’s end point is Kafka, with command-line options to set bootstrap servers, topics, and schema registry configurations.
- The instructions reveal that the application can be used in a containerized environment, via Cargo, or by downloading binaries, reinforcing a design that integrates with external systems seamlessly.

a. Components to include:
- Client/CLI Interface – Command dispatchers (stdin, file, tcp, udp, coap, rest, posixmq, nng).
- Inbound Protocol Handlers – Represented by the server components that accept external messages (each protocol handler as a separate module/node).
- Kafka Module – Covers the Kafka producer and schema registry support, including serialization (serde modules).
- Metrics and Telemetry – The metrics module (src/metrics.rs) for Prometheus output.
- Configuration/Environment – Showing dependencies to environment variables and configuration options (from CLI options and environment variables settings).
b. Relationships and Data Flow:
- Incoming messages flow from the protocol-specific handlers to the core application logic.
- Data processing / serialization happens within the Kafka module to prepare messages for dispatch.
- The CLI or server modules trigger the Kafka Producer which interacts with an external Kafka cluster.
c. Architectural Patterns:
- Separation of concerns is clear: CLI processing, protocol-specific message collection, Kafka message generation, and telemetry are all modularized.
d. Technologies/Frameworks:
- Rust (Cargo as the build tool)
- Kafka (and its ecosystem, including Avro and Schema Registry for schema management)
- Communication protocols like TCP, UDP, CoAP, etc.
- Containerization (Docker, docker-compose) and CI/CD (GitHub workflows)

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Configuration"
        Config["Configuration/Environment"]:::config
    end

    subgraph "Input Layer"
        CLI["Client/CLI Interface"]:::cli
        PROT["Inbound Protocol Handlers"]:::protocol
        subgraph "Protocols"
            TCP["TCP Handler"]:::protocol
            UDP["UDP Handler"]:::protocol
            REST["REST Handler"]:::protocol
            COAP["CoAP Handler"]:::protocol
            PosixMQ["PosixMQ Handler"]:::protocol
            NNG["NNG Handler"]:::protocol
        end
    end

    subgraph "Processing Layer"
        KMOD["Kafka Module"]:::kafka
        KPROD["Kafka Producer"]:::kafka
        SSR["Schema Registry"]:::kafka
        SERT["Message Serializer (Serde)"]:::kafka
    end

    KAFKA["Kafka Cluster"]:::external
    T["Metrics/Telemetry"]:::metrics

    Config --> CLI
    Config --> KMOD
    CLI --> PROT
    CLI --> KMOD
    PROT --> TCP
    PROT --> UDP
    PROT --> REST
    PROT --> COAP
    PROT --> PosixMQ
    PROT --> NNG
    PROT --> KMOD
    KMOD --> KPROD
    KMOD --> SSR
    KMOD --> SERT
    KPROD --> KAFKA
    KMOD --> T
    PROT --> T
    CLI --> T

    click CLI "https://github.com/mkroli/kafka-proxy/tree/main/src/cli/"
    click KMOD "https://github.com/mkroli/kafka-proxy/tree/main/src/kafka/"
    click PROT "https://github.com/mkroli/kafka-proxy/tree/main/src/server/"
    click T "https://github.com/mkroli/kafka-proxy/blob/main/src/metrics.rs"

    classDef cli fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    classDef protocol fill:#ccffcc,stroke:#00cc00,stroke-width:2px;
    classDef kafka fill:#ccccff,stroke:#0000cc,stroke-width:2px;
    classDef metrics fill:#fff2cc,stroke:#d4a017,stroke-width:2px;
    classDef config fill:#eafff3,stroke:#008080,stroke-width:2px;
    classDef external fill:#f2e6ff,stroke:#9932cc,stroke-width:2px;
```
