# twmb/franz-go architecture

> Architecture diagram of the GitHub repository twmb/franz-go, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/twmb/franz-go
- Repository: https://github.com/twmb/franz-go

## Overview

##### Main Components
- Core Client Library (kgo): This is the heart of the project that provides the functionality for producing and consuming messages. It contains the logic to interact with Kafka brokers, supports retries, timeouts, and graceful shutdowns.
- Protocol & Low-level Message Handling (kmsg & kbin): This layer is responsible for packing/unpacking Kafka protocol messages, serializing/deserializing, and handling request/response structures.
- Administration Module (kadm): This package provides functionality for Kafka administrative operations (e.g., managing topics, ACLs, consumer groups, configurations) and abstracts the necessary admin interfaces.
- Plugins (plugin subdirectories – e.g., kzap, klogr, kiprom, etc.): These modules allow users to hook in custom logging, metrics, or monitoring behavior. They act as extension points in the core client.
- Schema Registry Support (sr): Provides client-side integration for Apache Kafka Schema Registry interactions and convenient Serde (encoding/decoding) types.
- Code Generation (generate directory): Houses definition files and code that is automatically generated from Kafka protocol definitions. This helps keep the protocol handling current with Kafka’s evolving API.
- Supporting/Utility Packages (e.g., kerr for error handling, kversion for client version info, kfake for testing and simulation): These modules are used for handling errors, maintaining version compatibility, and simulating behavior for tests.
- Example Applications (examples directory): Although not part of the architecture per se, these demonstrate real-world usage (admin client, benchmarks, transactions, etc.) and can help in understanding how the components are intended to be integrated.

##### Relationships and Interactions
- The user-facing client (kgo) acts as the central gateway. It accepts configuration options and hooks (for logging, metrics, etc.) and, upon interacting with Kafka brokers, delegates low-level request/response tasks to the protocol layer (kmsg).
- The protocol layer (using components from kmsg and kbin) translates Go structures into Kafka wire formats and manages data flow between the client and Kafka brokers.
- For administrative operations, the kadm package leverages both the protocol layer (kmsg) and common utilities (kerr) to execute admin-specific operations.
- The plugin modules integrate with the core client using well-defined hook interfaces. They are loosely coupled and can be visualized as “sidecars” that intercept or augment the client's operations (e.g., logging, metrics collection).
- The code generation component (generate) periodically updates the protocol definitions that the kmsg and kbin packages rely on, ensuring the client can support all Kafka protocol features.
- Utility packages (like kerr and kversion) support error handling and version management across all these layers.

##### Architectural Patterns and Design Principles
- Modular Architecture: Each major functionality (client API, admin functions, protocol encoding/decoding, plugin support) is implemented as a separate module/package. This separation of concerns is key to the design.
- Dependency Injection/Hook System: The client allows plugging in user-provided hooks for logging and metrics, demonstrating a flexible plug-in design.
- Idiomatic Go Practices: The use of contexts for timeouts and cancellation, and avoiding goroutine/channel overhead where possible, are central to its performance-oriented design.
- Code Generation: This helps maintain strict alignment with the ever-evolving Kafka protocol while abstracting away boilerplate coding.

- Clear labels for each component (e.g., “Core Kafka Client (kgo)”, “Admin API (kadm)”, “Protocol Handling (kmsg/kbin)”, “Plugin Extensions”, “Schema Registry Client (sr)”, “Utility Libraries (kerr, kversion)”).
– From the Client (kgo) to the Protocol Layer (kmsg) for network communications.
– From the Client to the Admin Module when administrative requests are made.
– From the Client to Plugins via hook interfaces (bidirectional where plugins may influence client behavior).
– From Code Generation (generate) feeding into the Protocol Layer.
- Annotations on the interactions, such as how configuration options and context handling integrate at various stages of a request.

## Diagram (Mermaid)

```mermaid
flowchart TD 
    subgraph "Code Generation"
        CODEGEN["Code Generation"]
    end

    subgraph "Protocol Layer"
        KMSG["Protocol Handling (kmsg)"]
        KBIN["Binary Handling (kbin)"]
        KBIN_INT["Internal kbin (pkg/kmsg/internal/kbin)"]
    end

    CORE["Core Kafka Client (kgo)"]

    ADMIN["Admin API (kadm)"]

    SR["Schema Registry Client (sr)"]

    subgraph "Plugins"
        KZAP["Logging Plugin (kzap)"]
        KLOGR["Structured Logging (klogr)"]
        KOTEL["Observability Plugin (kotel)"]
    end

    subgraph "Utility Libraries"
        KERR["Error Handling (kerr)"]
        KVERSION["Version Info (kversion)"]
        KFAKE["Testing/Simulation (kfake)"]
    end

    BROKERS["Kafka Brokers"]

    EXAPPS["Example Applications"]

    CODEGEN -->|"updates"| KMSG
    CODEGEN -->|"updates"| KBIN
    CODEGEN -->|"updates"| KBIN_INT
    KMSG -->|"includes"| KBIN_INT
    CORE -->|"sends/receives"| KMSG
    CORE -->|"admin_requests"| ADMIN
    CORE -->|"schema_ops"| SR
    CORE <-->|"hooks"| KZAP
    CORE <-->|"hooks"| KLOGR
    CORE <-->|"hooks"| KOTEL
    CORE -->|"error_handling"| KERR
    CORE -->|"versioning"| KVERSION
    CORE -->|"simulation"| KFAKE
    CORE -->|"communicates"| BROKERS
    CORE -->|"example_usage"| EXAPPS

    click CODEGEN "https://github.com/twmb/franz-go/tree/master/generate"
    click KMSG "https://github.com/twmb/franz-go/tree/master/pkg/kmsg"
    click KBIN "https://github.com/twmb/franz-go/tree/master/pkg/kbin"
    click KBIN_INT "https://github.com/twmb/franz-go/tree/master/pkg/kmsg/internal/kbin"
    click CORE "https://github.com/twmb/franz-go/tree/master/pkg/kgo"
    click ADMIN "https://github.com/twmb/franz-go/tree/master/pkg/kadm"
    click SR "https://github.com/twmb/franz-go/tree/master/pkg/sr"
    click KZAP "https://github.com/twmb/franz-go/tree/master/plugin/kzap"
    click KLOGR "https://github.com/twmb/franz-go/tree/master/plugin/klogr"
    click KOTEL "https://github.com/twmb/franz-go/tree/master/plugin/kotel"
    click KERR "https://github.com/twmb/franz-go/tree/master/pkg/kerr"
    click KVERSION "https://github.com/twmb/franz-go/tree/master/pkg/kversion"
    click KFAKE "https://github.com/twmb/franz-go/tree/master/pkg/kfake"
    click EXAPPS "https://github.com/twmb/franz-go/tree/master/examples"

    class CODEGEN codegen
    class KMSG,KBIN,KBIN_INT protocol
    class CORE core
    class ADMIN admin
    class SR core
    class KZAP,KLOGR,KOTEL plugins
    class KERR,KVERSION,KFAKE utility
    class BROKERS external
    class EXAPPS examples

    classDef core fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef protocol fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef admin fill:#FFD580,stroke:#000,stroke-width:2px;
    classDef plugins fill:#dda0dd,stroke:#000,stroke-width:2px;
    classDef codegen fill:#F5B7B1,stroke:#000,stroke-width:2px;
    classDef utility fill:#FFFF99,stroke:#000,stroke-width:2px;
    classDef external fill:#E0FFFF,stroke:#000,stroke-width:2px;
    classDef examples fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
