How automq/automq works
AutoMQ is a Kafka-compatible, diskless streaming platform that uses S3-backed storage. Its principal workflow is Kafka client traffic handled by the broker, with stream data persisted and read through S3 storage; group coordination and administration are separate broker capabilities. The tree and excerpts support the storage, coordination, telemetry, and CLI components shown. Wiring among unsampled modules is intentionally limited, so this is a capability map rather than a complete call graph.
Main entry points: main Kafka broker · clients Kafka client APIs · NetworkClient.java Client networking · S3Storage.java S3 stream storage
Components 19
Kafka clients
- Kafka client APIs
clients/src/main/java/org/apache/kafka/clients - Client networking
clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
Broker services
- Kafka broker
core/src/main - Group coordinator
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java - Balancer actions
core/src/main/java/kafka/autobalancer/executor/ControllerActionExecutorService.java - Resumable services
core/src/main/java/kafka/autobalancer/services/AbstractResumableService.java
S3 storage
- S3 stream storage
s3stream/src/main/java/com/automq/stream/s3/S3Storage.java - Object storage operations
s3stream/src/main/java/com/automq/stream/s3/operator/AbstractObjectStorage.java - Remote log API
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java - Remote manager adapter
storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java
Operations and telemetry
- Log uploader
automq-log-uploader/src/main/java/com/automq/log/uploader/LogUploader.java - Log recorder
automq-log-uploader/src/main/java/com/automq/log/uploader/LogRecorder.java - Telemetry manager
automq-metrics/src/main/java/com/automq/opentelemetry/AutoMQTelemetryManager.java - Metrics exporters
automq-metrics/src/main/java/com/automq/opentelemetry/exporter - AutoMQ CLI
automq-shell/src/main/java/com/automq/shell/AutoMQCLI.java - Cluster commands
automq-shell/src/main/java/com/automq/shell/commands/cluster
Other components
- Applications
- S3 service
- Cluster operator
Connections 14
- Applications → Kafka client APIs: uses
- Kafka client APIs → Client networking: sends requests
- Client networking → Kafka broker: Kafka protocol
- Kafka broker → Group coordinator: coordinates groups
- Kafka broker → S3 stream storage: stores stream data
- S3 stream storage → Object storage operations: object operations
- Object storage operations → S3 service: reads and writes
- Remote log API → Remote manager adapter: delegates calls
- Remote manager adapter → S3 service: remote segment I/O
- Cluster operator → AutoMQ CLI: runs commands
- AutoMQ CLI → Cluster commands: dispatches
- Telemetry manager → Metrics exporters: exports metrics
- Log recorder → Log uploader: records logs
- Log uploader → S3 service: uploads logs