# ibm/sarama architecture

> Architecture diagram of the GitHub repository ibm/sarama, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ibm/sarama
- Repository: https://github.com/ibm/sarama
- Video tour (about a minute): https://gitdiagram.com/ibm/sarama/video

## Overview

##### Main Components

a. Client Interface Layer
- Main client interfaces (Consumer, Producer, Admin)
- Both sync and async producer variants
- Consumer group functionality
- Admin client capabilities

b. Protocol Implementation Layer
- Request/Response handling
- Message encoding/decoding
- Protocol versioning support
- Transaction management

c. Connection Management Layer
- Broker communication
- Connection pooling
- TLS/SASL security

d. Core Features
- Partitioning logic
- Compression support
- Metrics collection
- Error handling

**┌─────────────────────────────────────────────────────────┐**
- Application Layer                      │
- ┬─────────────────┬───────────────────┤
- Sync Producer  │ Async Producer  │     Consumer      │

- ┴─────────────────┴───────────────────┘
- ▲                 ▲              ▲

**┌─────────────────────────────────────────────────────────┐**
- Protocol Implementation                   │
- ┬───────────────┬──────────────┬───────────┤
- Message   │    Record     │  Transaction │  Admin    │
- Handling   │   Batching    │  Management  │   API     │
- ┴───────────────┴──────────────┴───────────┘
- ▲                 ▲              ▲

**┌─────────────────────────────────────────────────────────┐**
- Connection Management                      │
- ┬───────────────┬──────────────┬───────────┤
- Broker    │     SASL      │     TLS      │  Network  │
- Discovery  │    Auth       │   Security   │   I/O     │
- ┴───────────────┴──────────────┴───────────┘
- ▲                 ▲              ▲

**┌─────────────────────────────────────────────────────────┐**
- Apache Kafka                         │
- ┘

##### Important Elements to Highlight

a. Client Interfaces:
- How the sync/async producers differ
- Consumer group coordination
- Admin client capabilities

b. Protocol Features:
- Message batching and compression
- Transaction coordination flow
- Error handling paths

c. Connection Management:
- Broker discovery mechanism
- Security layer integration
- Connection pooling

##### Implementation Details

- Numbered sequence indicators for complex flows

**┌─────────────┐**
- Components  │ - Major system components
- ┤
**→ Flow        │ - Data/request flow direction**
- ┤
**⟲ Async       │ - Asynchronous operations**
- ┤
**⚡ Events      │ - Event triggers**
- ┘

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Client Interface Layer"
        SyncProd["Sync Producer"]
        AsyncProd["Async Producer"]
        Consumer["Consumer"]
        Admin["Admin Client"]
        ConsumerGroup["Consumer Group"]
    end

    subgraph "Protocol Implementation Layer"
        MsgHandle["Message Handling"]
        RecordMgmt["Record Management"]
        TransMgmt["Transaction Management"]
        ReqResp["Request/Response"]
    end

    subgraph "Connection Management Layer"
        Broker["Broker Communication"]
        ConnClient["Connection Client"]
        Security["Security"]
        subgraph "Security Components"
            SASL["SASL Auth"]
            TLS["TLS Security"]
            Kerberos["Kerberos"]
        end
    end

    subgraph "Core Features"
        Partition["Partitioning"]
        Compress["Compression"]
        Metrics["Metrics"]
        ErrorHandle["Error Handling"]
        Config["Configuration"]
    end

    subgraph "Protocol Encoding"
        Encoders["Encoders"]
        Decoders["Decoders"]
    end

    subgraph "Consumer Group Management"
        GroupCoord["Group Coordination"]
        BalanceStrat["Balance Strategy"]
    end

    Kafka[("Apache Kafka")]

    %% Relationships
    SyncProd & AsyncProd & Consumer & Admin --> ReqResp
    Consumer --> ConsumerGroup
    ConsumerGroup --> GroupCoord
    GroupCoord --> BalanceStrat
    ReqResp --> MsgHandle
    MsgHandle --> RecordMgmt
    RecordMgmt --> TransMgmt
    ReqResp --> Broker
    Broker --> ConnClient
    ConnClient --> Security
    Security --> SASL & TLS & Kerberos
    MsgHandle --> Encoders & Decoders
    ConnClient --> Partition
    ConnClient --> Compress
    ConnClient --> Metrics
    ConnClient --> ErrorHandle
    ConnClient --> Config
    ConnClient --> Kafka

    %% Styling
    classDef interface fill:#a8d1ff
    classDef protocol fill:#90EE90
    classDef connection fill:#FFB366
    classDef core fill:#E6E6E6
    classDef kafka fill:#FF9999

    class SyncProd,AsyncProd,Consumer,Admin,ConsumerGroup interface
    class MsgHandle,RecordMgmt,TransMgmt,ReqResp,Encoders,Decoders protocol
    class Broker,ConnClient,Security,SASL,TLS,Kerberos connection
    class Partition,Compress,Metrics,ErrorHandle,Config core
    class Kafka kafka

    %% Click events
    click SyncProd "https://github.com/IBM/sarama/blob/main/sync_producer.go"
    click AsyncProd "https://github.com/IBM/sarama/blob/main/async_producer.go"
    click Consumer "https://github.com/IBM/sarama/blob/main/consumer.go"
    click Admin "https://github.com/IBM/sarama/blob/main/admin.go"
    click ConsumerGroup "https://github.com/IBM/sarama/blob/main/consumer_group.go"
    click MsgHandle "https://github.com/IBM/sarama/blob/main/message.go"
    click RecordMgmt "https://github.com/IBM/sarama/blob/main/record.go"
    click TransMgmt "https://github.com/IBM/sarama/blob/main/transaction_manager.go"
    click ReqResp "https://github.com/IBM/sarama/blob/main/request.go"
    click Broker "https://github.com/IBM/sarama/blob/main/broker.go"
    click ConnClient "https://github.com/IBM/sarama/blob/main/client.go"
    click SASL "https://github.com/IBM/sarama/blob/main/sasl_authenticate_request.go"
    click TLS "https://github.com/IBM/sarama/blob/main/client_tls_test.go"
    click Kerberos "https://github.com/IBM/sarama/blob/main/kerberos_client.go"
    click Partition "https://github.com/IBM/sarama/blob/main/partitioner.go"
    click Compress "https://github.com/IBM/sarama/blob/main/compress.go"
    click Metrics "https://github.com/IBM/sarama/blob/main/metrics.go"
    click ErrorHandle "https://github.com/IBM/sarama/blob/main/errors.go"
    click Config "https://github.com/IBM/sarama/blob/main/config.go"
    click Encoders "https://github.com/IBM/sarama/blob/main/real_encoder.go"
    click Decoders "https://github.com/IBM/sarama/blob/main/real_decoder.go"
    click GroupCoord "https://github.com/IBM/sarama/blob/main/consumer_group_members.go"
    click BalanceStrat "https://github.com/IBM/sarama/blob/main/balance_strategy.go"
```
