# line/decaton architecture

> Architecture diagram of the GitHub repository line/decaton, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/line/decaton
- Repository: https://github.com/line/decaton

## Overview

##### Main System Components
- Client Module – This includes the “client” directory and its subpackages. It provides APIs to produce tasks (e.g., DecatonClient, KafkaProducerSupplier) that applications use to push records into Kafka.
- Processor Module – Found in the “processor” directory. This is the core engine that consumes records from Kafka partitions and processes them concurrently while ensuring ordering, at-least-once delivery, and other features such as retry queuing and rate limiting.
- Common Module – Under “common”, which contains shared classes such as serializers and deserializers.
- Integration Modules – There are modules for Spring (in the “spring” directory) that enable easy integration into Spring Boot apps, plus “centraldogma” which likely deals with dynamic property sourcing/centralized configuration support, and modules such as “brave” and “micrometer-tracing” for distributed tracing/instrumentation.
- Auxiliary/Testing Modules – The “benchmark”, “testing”, “docs”, “protobuf”, and “protocol” directories provide examples, performance benchmarks, tests, and serialization/deserialization support for interacting with external systems (Kafka, Zookeeper, etc.).

##### Data Flow and Interactions
- From Kafka, show how the Decaton Processor module consumes records. Indicate a consumer group configuration where tasks are consumed partition‑wise and processed concurrently.
- That the Processor module implements features such as concurrent processing, task batching/compaction, deleting or retrying tasks (as seen in the various “processor” classes dealing with RetryQueueing, Batching, etc.).
– To “Tracing” modules (from “brave” and “micrometer-tracing”) for capturing distributed tracing data.
– To “CentralDogma” if dynamic configuration via external property suppliers is employed.
– To the “Spring” integration for auto-configuration and bean wiring in Spring Boot applications.

##### Architectural Patterns and Design Principles
- Separation of concerns: the client is only concerned with producing tasks; the processor handles task consumption and execution.
- High-performance, concurrent processing pattern where ordering is maintained while parallelism is achieved.
- External dependency on Apache Kafka as the backbone for messaging.
- Incorporate testing components that ensure that Embeded Kafka, ZooKeeper, etc. are used to simulate real-world scenarios.
- Recognize the modular design: each module (client, processor, common, protocol, etc.) acts as a layered component that addresses specific aspects (task production, task processing, configuration, tracing, etc.). This modular approach also facilitates extensions via integration modules.

##### Technology and Framework Stack (As Indicated in the Project)
- Apache Kafka as the messaging backbone.
- Gradle as the build tool.
- JVM (Java) as the core runtime environment.
- Spring Framework for integration support.
- CentralDogma for property sourcing (if used).
- Distributed tracing libraries like Brave and Micrometer.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Application Layer
    subgraph "Application Layer"
        EA["External Application"]:::external
        DC["Decaton Client API (Client Module)"]:::client
    end
    EA -->|"calls"| DC

    %% Messaging Layer
    subgraph "Messaging Layer"
        KB["Kafka Broker"]:::external
    end
    DC -->|"produces_tasks"| KB

    %% Processing Layer
    subgraph "Processing Layer"
        DP["Decaton Processor Engine (Processor Module)"]:::processor
        CU["Common Utilities (Common Module)"]:::common
    end
    KB -->|"delivers_tasks"| DP
    DP -->|"uses"| CU

    %% Integration Modules
    subgraph "Integration Modules"
        SI["Spring Integration (Integration Module)"]:::integration
        CD["CentralDogma Integration (Integration Module)"]:::integration
        BT["Brave Tracing (Integration Module)"]:::integration
        MT["Micrometer Tracing (Integration Module)"]:::integration
    end
    DP -->|"integrates"| SI
    DP -->|"config_sources"| CD
    DP -->|"tracing"| BT
    DP -->|"tracing"| MT

    %% Auxiliary/Testing Modules
    subgraph "Auxiliary/Testing Modules"
        BM["Benchmarking Module (Auxiliary/Testing)"]:::auxiliary
        TM["Testing Module (Auxiliary/Testing)"]:::auxiliary
        DE1["Documentation (Docs)"]:::auxiliary
        DE2["Examples (Developer Docs)"]:::auxiliary
        PS1["Protobuf Support (Protocol)"]:::auxiliary
        PS2["Protocol Support (Protocol)"]:::auxiliary
    end
    DP ---|"auxiliary"| BM
    DP ---|"auxiliary"| TM
    DP ---|"auxiliary"| DE1
    DP ---|"auxiliary"| DE2
    DP ---|"auxiliary"| PS1
    DP ---|"auxiliary"| PS2

    %% Click Events
    click DC "https://github.com/line/decaton/tree/master/client/"
    click DP "https://github.com/line/decaton/tree/master/processor/"
    click CU "https://github.com/line/decaton/tree/master/common/"
    click SI "https://github.com/line/decaton/tree/master/spring/"
    click CD "https://github.com/line/decaton/tree/master/centraldogma/"
    click BT "https://github.com/line/decaton/tree/master/brave/"
    click MT "https://github.com/line/decaton/tree/master/micrometer-tracing/"
    click BM "https://github.com/line/decaton/tree/master/benchmark/"
    click TM "https://github.com/line/decaton/tree/master/testing/"
    click DE1 "https://github.com/line/decaton/tree/master/docs/"
    click DE2 "https://github.com/line/decaton/tree/master/developer-docs/"
    click PS1 "https://github.com/line/decaton/tree/master/protobuf/"
    click PS2 "https://github.com/line/decaton/tree/master/protocol/"

    %% Styles
    classDef external fill:#ffaaaa, stroke:#333, stroke-width:2px;
    classDef client fill:#9fdfbf, stroke:#333, stroke-width:2px;
    classDef processor fill:#ffcc99, stroke:#333, stroke-width:2px;
    classDef common fill:#cccccc, stroke:#333, stroke-width:2px;
    classDef integration fill:#99ccff, stroke:#333, stroke-width:2px;
    classDef auxiliary fill:#d1afff, stroke:#333, stroke-width:2px;
```
