# automq/automq architecture

> Architecture diagram of the GitHub repository automq/automq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/automq/automq
- Repository: https://github.com/automq/automq

## Overview

AutoMQ is a Kafka-compatible, diskless streaming platform that uses S3-backed storage. Its principal workflow is Kafka client traffic handled by the broker, with stream data persisted and read through S3 storage; group coordination and administration are separate broker capabilities. The tree and excerpts support the storage, coordination, telemetry, and CLI components shown. Wiring among unsampled modules is intentionally limited, so this is a capability map rather than a complete call graph.

Main entry points: [`core/src/main`](https://github.com/automq/automq/tree/main/core/src/main) (Kafka broker), [`clients/src/main/java/org/apache/kafka/clients`](https://github.com/automq/automq/tree/main/clients/src/main/java/org/apache/kafka/clients) (Kafka client APIs), [`clients/src/main/java/org/apache/kafka/clients/NetworkClient.java`](https://github.com/automq/automq/blob/main/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java) (Client networking), [`s3stream/src/main/java/com/automq/stream/s3/S3Storage.java`](https://github.com/automq/automq/blob/main/s3stream/src/main/java/com/automq/stream/s3/S3Storage.java) (S3 stream storage)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Kafka clients"]
  node_clients["Kafka client APIs"]
  node_network["Client networking<br/>[NetworkClient.java]"]
end

subgraph group_broker["Broker services"]
  node_broker["Kafka broker"]
  node_groupsvc["Group coordinator"]
  node_rebalance["Balancer actions"]
  node_service["Resumable services"]
end

subgraph group_storage["S3 storage"]
  node_s3storage["S3 stream storage<br/>[S3Storage.java]"]
  node_objectstorage["Object storage operations"]
  node_remoteapi["Remote log API"]
  node_remoteadapter["Remote manager adapter"]
end

subgraph group_operations["Operations and telemetry"]
  node_logupload["Log uploader<br/>[LogUploader.java]"]
  node_logrecorder["Log recorder<br/>[LogRecorder.java]"]
  node_telemetry["Telemetry manager"]
  node_metrics["Metrics exporters"]
  node_cli["AutoMQ CLI<br/>[AutoMQCLI.java]"]
  node_clustercommands["Cluster commands"]
end

node_users(("Applications"))
node_s3[("S3 service")]
node_operator(("Cluster operator"))

node_users -->|"uses"| node_clients
node_clients -->|"sends requests"| node_network
node_network -->|"Kafka protocol"| node_broker
node_broker -->|"coordinates groups"| node_groupsvc
node_broker -->|"stores stream data"| node_s3storage
node_s3storage -->|"object operations"| node_objectstorage
node_objectstorage -->|"reads and writes"| node_s3
node_remoteapi -->|"delegates calls"| node_remoteadapter
node_remoteadapter -.->|"remote segment I/O"| node_s3
node_operator -->|"runs commands"| node_cli
node_cli -->|"dispatches"| node_clustercommands
node_telemetry -->|"exports metrics"| node_metrics
node_logrecorder -->|"records logs"| node_logupload
node_logupload -.->|"uploads logs"| node_s3

click node_clients "https://github.com/automq/automq/tree/main/clients/src/main/java/org/apache/kafka/clients"
click node_network "https://github.com/automq/automq/blob/main/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java"
click node_broker "https://github.com/automq/automq/tree/main/core/src/main"
click node_groupsvc "https://github.com/automq/automq/blob/main/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java"
click node_rebalance "https://github.com/automq/automq/blob/main/core/src/main/java/kafka/autobalancer/executor/ControllerActionExecutorService.java"
click node_service "https://github.com/automq/automq/blob/main/core/src/main/java/kafka/autobalancer/services/AbstractResumableService.java"
click node_s3storage "https://github.com/automq/automq/blob/main/s3stream/src/main/java/com/automq/stream/s3/S3Storage.java"
click node_objectstorage "https://github.com/automq/automq/blob/main/s3stream/src/main/java/com/automq/stream/s3/operator/AbstractObjectStorage.java"
click node_remoteapi "https://github.com/automq/automq/blob/main/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java"
click node_remoteadapter "https://github.com/automq/automq/blob/main/storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java"
click node_logupload "https://github.com/automq/automq/blob/main/automq-log-uploader/src/main/java/com/automq/log/uploader/LogUploader.java"
click node_logrecorder "https://github.com/automq/automq/blob/main/automq-log-uploader/src/main/java/com/automq/log/uploader/LogRecorder.java"
click node_telemetry "https://github.com/automq/automq/blob/main/automq-metrics/src/main/java/com/automq/opentelemetry/AutoMQTelemetryManager.java"
click node_metrics "https://github.com/automq/automq/tree/main/automq-metrics/src/main/java/com/automq/opentelemetry/exporter"
click node_cli "https://github.com/automq/automq/blob/main/automq-shell/src/main/java/com/automq/shell/AutoMQCLI.java"
click node_clustercommands "https://github.com/automq/automq/tree/main/automq-shell/src/main/java/com/automq/shell/commands/cluster"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_clients,node_network toneBlue
class node_broker,node_groupsvc,node_rebalance,node_service,node_s3 toneAmber
class node_s3storage,node_objectstorage,node_remoteapi,node_remoteadapter toneMint
class node_logupload,node_logrecorder,node_telemetry,node_metrics,node_cli,node_clustercommands toneRose
class node_users,node_operator toneIndigo
```

## Components

### Kafka clients

- [Kafka client APIs](https://github.com/automq/automq/tree/main/clients/src/main/java/org/apache/kafka/clients): `clients/src/main/java/org/apache/kafka/clients`
- [Client networking](https://github.com/automq/automq/blob/main/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java): `clients/src/main/java/org/apache/kafka/clients/NetworkClient.java`

### Broker services

- [Kafka broker](https://github.com/automq/automq/tree/main/core/src/main): `core/src/main`
- [Group coordinator](https://github.com/automq/automq/blob/main/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java): `group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java`
- [Balancer actions](https://github.com/automq/automq/blob/main/core/src/main/java/kafka/autobalancer/executor/ControllerActionExecutorService.java): `core/src/main/java/kafka/autobalancer/executor/ControllerActionExecutorService.java`
- [Resumable services](https://github.com/automq/automq/blob/main/core/src/main/java/kafka/autobalancer/services/AbstractResumableService.java): `core/src/main/java/kafka/autobalancer/services/AbstractResumableService.java`

### S3 storage

- [S3 stream storage](https://github.com/automq/automq/blob/main/s3stream/src/main/java/com/automq/stream/s3/S3Storage.java): `s3stream/src/main/java/com/automq/stream/s3/S3Storage.java`
- [Object storage operations](https://github.com/automq/automq/blob/main/s3stream/src/main/java/com/automq/stream/s3/operator/AbstractObjectStorage.java): `s3stream/src/main/java/com/automq/stream/s3/operator/AbstractObjectStorage.java`
- [Remote log API](https://github.com/automq/automq/blob/main/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java): `storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java`
- [Remote manager adapter](https://github.com/automq/automq/blob/main/storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java): `storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java`

### Operations and telemetry

- [Log uploader](https://github.com/automq/automq/blob/main/automq-log-uploader/src/main/java/com/automq/log/uploader/LogUploader.java): `automq-log-uploader/src/main/java/com/automq/log/uploader/LogUploader.java`
- [Log recorder](https://github.com/automq/automq/blob/main/automq-log-uploader/src/main/java/com/automq/log/uploader/LogRecorder.java): `automq-log-uploader/src/main/java/com/automq/log/uploader/LogRecorder.java`
- [Telemetry manager](https://github.com/automq/automq/blob/main/automq-metrics/src/main/java/com/automq/opentelemetry/AutoMQTelemetryManager.java): `automq-metrics/src/main/java/com/automq/opentelemetry/AutoMQTelemetryManager.java`
- [Metrics exporters](https://github.com/automq/automq/tree/main/automq-metrics/src/main/java/com/automq/opentelemetry/exporter): `automq-metrics/src/main/java/com/automq/opentelemetry/exporter`
- [AutoMQ CLI](https://github.com/automq/automq/blob/main/automq-shell/src/main/java/com/automq/shell/AutoMQCLI.java): `automq-shell/src/main/java/com/automq/shell/AutoMQCLI.java`
- [Cluster commands](https://github.com/automq/automq/tree/main/automq-shell/src/main/java/com/automq/shell/commands/cluster): `automq-shell/src/main/java/com/automq/shell/commands/cluster`

### Other components

- **Applications**
- **S3 service**
- **Cluster operator**

## Connections

- Applications → Kafka client APIs: uses
- Kafka client APIs → Client networking: sends requests
- Client networking → Kafka broker: Kafka protocol
- Kafka broker → Group coordinator: coordinates groups
- Kafka broker → S3 stream storage: stores stream data
- S3 stream storage → Object storage operations: object operations
- Object storage operations → S3 service: reads and writes
- Remote log API → Remote manager adapter: delegates calls
- Remote manager adapter → S3 service: remote segment I/O
- Cluster operator → AutoMQ CLI: runs commands
- AutoMQ CLI → Cluster commands: dispatches
- Telemetry manager → Metrics exporters: exports metrics
- Log recorder → Log uploader: records logs
- Log uploader → S3 service: uploads logs