# edenhill/kcat architecture

> Architecture diagram of the GitHub repository edenhill/kcat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/edenhill/kcat
- Repository: https://github.com/edenhill/kcat

## Overview

##### Main Components Identification
The system should be represented with these key components:

a. Core Components:
- Message Producer Module
- Message Consumer Module
- Metadata List Module
- Mock Cluster Module
- Message Format/Serialization Module

b. External Dependencies:
- librdkafka (core dependency)
- libyajl (JSON support)
- libavro-c and libserdes (Avro support)
- Kafka Cluster (external system)
- Schema Registry (optional external system)

**[Client Layer]**

**[Core Processing Layer]**

**[Integration Layer]**

**[External Systems]**

##### Component Relationships

- stdin → Producer Module → Kafka Cluster
- Kafka Cluster → Consumer Module → stdout
- Kafka Cluster ↔ Metadata List Module
- Mock Cluster Module ← → Client Applications

b. Integration Points:
- Format Module ↔ Schema Registry
- All Modules ↔ librdkafka
- JSON Module ↔ libyajl
- Avro Module ↔ libavro-c/libserdes

##### Specific Implementation Details

Key features:
- Message delimiter handling (-D option)
- Partition management
- Consumer group support (-G option)
- Transaction support
- Compression options
- Authentication/Security modules
- Header handling
- Serialization/Deserialization options

##### Important Notes

a. Configuration Flow:
- How configuration can be loaded from:
- Command line arguments
- Configuration files
- Environment variables

b. Mode Switching:
- How the system switches between:
- Producer mode
- Consumer mode
- Metadata list mode
- Mock cluster mode

##### Technical Details

- Support for multiple message formats (Avro, JSON, etc.)
- Integration with Schema Registry
- Security protocol support (SSL, SASL)
- Compression support
- Transaction handling
- Header management
- Offset management

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client["Client Layer"]
        CLI["Command Line Interface"]
        STDIN["Standard Input"]:::io
        STDOUT["Standard Output"]:::io
    end

    subgraph Core["Core Processing Layer"]
        KCAT["Main kcat Program"]:::core
        click KCAT "https://github.com/edenhill/kcat/blob/master/kcat.c"
        
        subgraph Processing["Processing Modules"]
            PROD["Message Producer"]:::core
            CONS["Message Consumer"]:::core
            META["Metadata List"]:::core
            MOCK["Mock Cluster"]:::core
            INPUT["Input Processing"]:::core
            click INPUT "https://github.com/edenhill/kcat/blob/master/input.c"
            TOOLS["Utility Functions"]:::core
            click TOOLS "https://github.com/edenhill/kcat/blob/master/tools.c"
        end

        subgraph Format["Format/Serialization"]
            FMT["Format Module"]:::core
            click FMT "https://github.com/edenhill/kcat/blob/master/format.c"
            JSON["JSON Module"]:::core
            click JSON "https://github.com/edenhill/kcat/blob/master/json.c"
            AVRO["Avro Module"]:::core
            click AVRO "https://github.com/edenhill/kcat/blob/master/avro.c"
        end
    end

    subgraph Integration["Integration Layer"]
        RDKAFKA["librdkafka"]:::dep
        YAJL["libyajl"]:::dep
        AVROC["libavro-c"]:::dep
        SERDES["libserdes"]:::dep
    end

    subgraph External["External Systems"]
        KAFKA["Kafka Cluster"]:::external
        REGISTRY["Schema Registry"]:::external-opt
    end

    subgraph Platform["Platform Layer"]
        UNIX["Unix/Linux Layer"]:::platform
        click UNIX "https://github.com/edenhill/kcat/blob/master/rdport.h"
        WIN["Windows Layer"]:::platform
        click WIN "https://github.com/edenhill/kcat/tree/master/win32/"
    end

    subgraph Build["Build System"]
        MAKE["Makefile"]:::build
        click MAKE "https://github.com/edenhill/kcat/tree/master/Makefile"
        CONFIG["Configure"]:::build
        click CONFIG "https://github.com/edenhill/kcat/tree/master/configure"
        MKLOVE["mklove"]:::build
        click MKLOVE "https://github.com/edenhill/kcat/tree/master/mklove/"
    end

    %% Connections
    CLI --> KCAT
    STDIN --> INPUT
    KCAT --> STDOUT
    
    KCAT --> PROD
    KCAT --> CONS
    KCAT --> META
    KCAT --> MOCK
    
    PROD & CONS & META --> FMT
    FMT --> JSON & AVRO
    
    JSON --> YAJL
    AVRO --> AVROC
    AVRO --> SERDES
    
    PROD & CONS & META --> RDKAFKA
    RDKAFKA --> KAFKA
    SERDES --> REGISTRY
    
    KCAT --> UNIX & WIN

    %% Styling
    classDef core fill:#3498db,stroke:#2980b9,color:white
    classDef dep fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef external fill:#2ecc71,stroke:#27ae60,color:white
    classDef external-opt fill:#e67e22,stroke:#d35400,color:white
    classDef io fill:#f1c40f,stroke:#f39c12,color:black
    classDef platform fill:#9b59b6,stroke:#8e44ad,color:white
    classDef build fill:#34495e,stroke:#2c3e50,color:white

    %% Legend
    subgraph Legend
        L1["Core Components"]:::core
        L2["Dependencies"]:::dep
        L3["External Systems"]:::external
        L4["Optional Systems"]:::external-opt
        L5["I/O Streams"]:::io
        L6["Platform Specific"]:::platform
        L7["Build System"]:::build
    end
```
