# segmentio/kafka-go architecture

> Architecture diagram of the GitHub repository segmentio/kafka-go, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/segmentio/kafka-go
- Repository: https://github.com/segmentio/kafka-go

## Overview

a. Client to Broker:
- Connection establishment
- TLS/SASL handshake if configured
- Broker discovery

b. Reader Flow:
- Consumer group coordination
- Partition assignment
- Message fetching
- Offset management

c. Writer Flow:
- Topic resolution
- Partition selection
- Message batching
- Delivery acknowledgment

##### Important Subsystems (Use nested containers)

a. Balancer Implementations:
- Hash
- LeastBytes
- RoundRobin
- CRC32
- Murmur2

b. Error Handling:
- Retries
- Timeouts
- Error logging

- Blue: Core components
- Green: Successful operations
- Yellow: Supporting features
- Red: Error handling
- Gray: External systems

##### Scaling aspects
- Connection pooling
- Concurrent operations
- Resource management

##### Timing indicators for
- Connection timeouts
- Read/Write deadlines
- Retry intervals

Key Relationships to Highlight:

##### Partition assignment strategies

Remember to include version compatibility notes and any significant behavioral differences between Kafka versions, as mentioned in the README (Kafka 0.10.1.0 to 2.7.1 compatibility).

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Client ["Kafka Go Client Library"]
        direction TB
        
        CL[Connection Layer]:::core
        RC[Reader Component]:::core
        WC[Writer Component]:::core
        PL[Protocol Layer]:::core

        subgraph Support ["Supporting Components"]
            direction TB
            
            subgraph Compress ["Compression"]
                direction TB
                SN[Snappy]:::compress
                GZ[Gzip]:::compress
                LZ[LZ4]:::compress
                ZS[Zstd]:::compress
            end

            subgraph Auth ["Authentication"]
                direction TB
                SASL[SASL]:::auth
                TLS[TLS]:::auth
                Plain[Plain Auth]:::auth
                SCRAM[SCRAM Auth]:::auth
                MSK[AWS MSK IAM]:::auth
            end

            subgraph Balance ["Balancers"]
                direction TB
                Hash[Hash]:::balance
                LB[LeastBytes]:::balance
                RR[RoundRobin]:::balance
                CRC[CRC32]:::balance
                MUR[Murmur2]:::balance
            end

            subgraph Msg ["Message Processing"]
                direction TB
                MP[Message]:::msg
                BP[Batch]:::msg
                RP[Record]:::msg
                RBP[RecordBatch]:::msg
            end

            subgraph ErrHandle ["Error Handling"]
                direction TB
                Retry[Retries]:::error
                Timeout[Timeouts]:::error
                ErrLog[Error Logging]:::error
            end
        end

        subgraph CGM ["Consumer Group Management"]
            direction TB
            CG[Consumer Groups]:::cg
            PA[Partition Assignment]:::cg
            OM[Offset Management]:::cg
        end
    end

    subgraph External ["External Systems"]
        direction TB
        KB[Kafka Brokers]:::external
    end

    %% Connections
    CL --> KB
    RC --> CL
    WC --> CL
    CL --> PL
    RC --> CGM
    WC --> Balance
    MP --> Compress

    %% Click Events
    click CL "https://github.com/segmentio/kafka-go/blob/main/conn.go"
    click RC "https://github.com/segmentio/kafka-go/blob/main/reader.go"
    click WC "https://github.com/segmentio/kafka-go/blob/main/writer.go"
    click PL "https://github.com/segmentio/kafka-go/blob/main/protocol/protocol.go"
    click SASL "https://github.com/segmentio/kafka-go/blob/main/sasl/sasl.go"
    click Plain "https://github.com/segmentio/kafka-go/blob/main/sasl/plain/plain.go"
    click SCRAM "https://github.com/segmentio/kafka-go/blob/main/sasl/scram/scram.go"
    click MSK "https://github.com/segmentio/kafka-go/blob/main/sasl/aws_msk_iam/msk_iam.go"
    click SN "https://github.com/segmentio/kafka-go/blob/main/compress/snappy/snappy.go"
    click GZ "https://github.com/segmentio/kafka-go/blob/main/compress/gzip/gzip.go"
    click LZ "https://github.com/segmentio/kafka-go/blob/main/compress/lz4/lz4.go"
    click ZS "https://github.com/segmentio/kafka-go/blob/main/compress/zstd/zstd.go"
    click CG "https://github.com/segmentio/kafka-go/blob/main/consumergroup.go"
    click MP "https://github.com/segmentio/kafka-go/blob/main/message.go"
    click BP "https://github.com/segmentio/kafka-go/blob/main/batch.go"
    click RP "https://github.com/segmentio/kafka-go/blob/main/record.go"
    click RBP "https://github.com/segmentio/kafka-go/blob/main/recordbatch.go"
    click ErrLog "https://github.com/segmentio/kafka-go/blob/main/error.go"

    %% Styles
    classDef core fill:#2374ab
    classDef compress fill:#57a773
    classDef auth fill:#ffc857
    classDef balance fill:#a4bab7
    classDef msg fill:#95b8d1
    classDef error fill:#e84855
    classDef cg fill:#8b8c89
    classDef external fill:#d3d3d3

    %% Legend
    subgraph Legend
        direction TB
        L1[Core Components]:::core
        L2[Compression]:::compress
        L3[Authentication]:::auth
        L4[Balancers]:::balance
        L5[Message Processing]:::msg
        L6[Error Handling]:::error
        L7[Consumer Groups]:::cg
        L8[External Systems]:::external
    end
```
