# raystack/firehose architecture

> Architecture diagram of the GitHub repository raystack/firehose, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raystack/firehose
- Repository: https://github.com/raystack/firehose

## Overview

Firehose is a Kafka-to-destination streaming service. Its principal workflow starts at the service runtime, configures a Kafka consumer, processes messages, and delivers them to a configured sink; filtering, error handling, retries, dead-letter output, and metrics support that flow. The README documents the sink capabilities, while sampled source and the SOURCE INDEX establish several internal relationships. Some sink-specific wiring is not shown, so the graph keeps those branches at the documented capability level and avoids asserting unverified calls.

Main entry points: [`src/main/java/org/raystack/firehose/message/Message.java`](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/message/Message.java) (Message envelope), [`src/main/java/org/raystack/firehose/launch/Main.java`](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/launch/Main.java) (Runtime entry), [`src/main/java/org/raystack/firehose/metrics/FirehoseInstrumentation.java`](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/metrics/FirehoseInstrumentation.java) (Instrumentation), [`src/main/java/org/raystack/firehose/consumer/kafka/FirehoseKafkaConsumer.java`](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/consumer/kafka/FirehoseKafkaConsumer.java) (Kafka consumer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime and input"]
  node_main["Runtime entry<br/>[Main.java]"]
  node_config["Application config<br/>[AppConfig.java]"]
  node_consumer_factory["Consumer factory"]
  node_kafka_consumer["Kafka consumer"]
  node_offsets["Offset manager"]
end

subgraph group_processing["Message processing"]
  node_message["Message envelope<br/>[Message.java]"]
  node_filter["Message filters"]
  node_proto["Proto mapping"]
  node_serializer["Message serialization<br/>[MessageToJson.java]"]
end

subgraph group_delivery["Delivery and recovery"]
  node_sink_pool["Sink pool<br/>[SinkPool.java]"]
  node_sink_factory["Sink factory<br/>[SinkFactory.java]"]
  node_sink_decorators["Retries and DLQ<br/>[SinkWithRetry.java]"]
  node_error_handler["Error handling<br/>[ErrorHandler.java]"]
  node_service_sinks["Service and DB sinks"]
  node_blob_sink["Blob sink<br/>[BlobSink.java]"]
  node_local_files["Local file writer<br/>[LocalStorage.java]"]
  node_remote_upload["Remote uploader"]
  node_blob_storage["Blob storage<br/>[BlobStorage.java]"]
  node_gcs["Google Cloud Storage"]
  node_dlq["Blob DLQ writer"]
end

subgraph group_operations["Observability"]
  node_metrics["Instrumentation"]
  node_metric_defs["Metric definitions<br/>[Metrics.java]"]
end

node_operator(("Operator"))
node_kafka[("Kafka")]
node_statsd(("StatsD reporter"))
node_destinations(("Configured destinations"))

node_operator -.->|"starts"| node_main
node_main -->|"creates"| node_consumer_factory
node_main -.->|"configures"| node_config
node_consumer_factory -.->|"selects"| node_kafka_consumer
node_kafka -->|"supplies records"| node_kafka_consumer
node_kafka_consumer -.->|"wraps records"| node_message
node_offsets -->|"manages offsets"| node_kafka_consumer
node_offsets -->|"tracks offsets"| node_message
node_message -.->|"is filtered"| node_filter
node_message -.->|"maps fields"| node_proto
node_message -.->|"is serialized"| node_serializer
node_sink_pool -.->|"uses"| node_sink_factory
node_sink_factory -.->|"creates sink"| node_service_sinks
node_sink_factory -.->|"creates sink"| node_blob_sink
node_service_sinks -->|"delivers to"| node_destinations
node_blob_sink -.->|"writes files"| node_local_files
node_local_files -.->|"uploads files"| node_remote_upload
node_remote_upload -->|"stores files"| node_blob_storage
node_blob_storage -->|"implemented by"| node_gcs
node_error_handler -.->|"routes failures"| node_dlq
node_dlq -->|"stores messages"| node_blob_storage
node_sink_decorators -->|"handles failures"| node_error_handler
node_metrics -.->|"reports metrics"| node_statsd
node_metrics -->|"uses definitions"| node_metric_defs
node_main -->|"initializes"| node_metrics
node_offsets -->|"records telemetry"| node_metrics

click node_main "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/launch/Main.java"
click node_config "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/config/AppConfig.java"
click node_consumer_factory "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/consumer/FirehoseConsumerFactory.java"
click node_kafka_consumer "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/consumer/kafka/FirehoseKafkaConsumer.java"
click node_offsets "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/consumer/kafka/ConsumerAndOffsetManager.java"
click node_message "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/message/Message.java"
click node_filter "https://github.com/raystack/firehose/tree/main/src/main/java/org/raystack/firehose/filter"
click node_proto "https://github.com/raystack/firehose/tree/main/src/main/java/org/raystack/firehose/proto"
click node_serializer "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/serializer/MessageToJson.java"
click node_sink_pool "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/SinkPool.java"
click node_sink_factory "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/SinkFactory.java"
click node_sink_decorators "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sinkdecorator/SinkWithRetry.java"
click node_error_handler "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/error/ErrorHandler.java"
click node_service_sinks "https://github.com/raystack/firehose/tree/main/src/main/java/org/raystack/firehose/sink"
click node_blob_sink "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/blob/BlobSink.java"
click node_local_files "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/blob/writer/local/LocalStorage.java"
click node_remote_upload "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/blob/writer/remote/BlobStorageChecker.java"
click node_blob_storage "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/common/blobstorage/BlobStorage.java"
click node_gcs "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/common/blobstorage/gcs/GoogleCloudStorage.java"
click node_dlq "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/dlq/blobstorage/BlobStorageDlqWriter.java"
click node_metrics "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/metrics/FirehoseInstrumentation.java"
click node_metric_defs "https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/metrics/Metrics.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_config,node_consumer_factory,node_kafka_consumer,node_offsets toneBlue
class node_message,node_filter,node_proto,node_serializer,node_kafka toneAmber
class node_sink_pool,node_sink_factory,node_sink_decorators,node_error_handler,node_service_sinks,node_blob_sink,node_local_files,node_remote_upload,node_blob_storage,node_gcs,node_dlq toneMint
class node_metrics,node_metric_defs toneRose
class node_operator,node_statsd,node_destinations toneIndigo
```

## Components

### Runtime and input

- [Runtime entry](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/launch/Main.java): `src/main/java/org/raystack/firehose/launch/Main.java`
- [Application config](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/config/AppConfig.java): `src/main/java/org/raystack/firehose/config/AppConfig.java`
- [Consumer factory](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/consumer/FirehoseConsumerFactory.java): `src/main/java/org/raystack/firehose/consumer/FirehoseConsumerFactory.java`
- [Kafka consumer](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/consumer/kafka/FirehoseKafkaConsumer.java): `src/main/java/org/raystack/firehose/consumer/kafka/FirehoseKafkaConsumer.java`
- [Offset manager](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/consumer/kafka/ConsumerAndOffsetManager.java): `src/main/java/org/raystack/firehose/consumer/kafka/ConsumerAndOffsetManager.java`

### Message processing

- [Message envelope](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/message/Message.java): `src/main/java/org/raystack/firehose/message/Message.java`
- [Message filters](https://github.com/raystack/firehose/tree/main/src/main/java/org/raystack/firehose/filter): `src/main/java/org/raystack/firehose/filter`
- [Proto mapping](https://github.com/raystack/firehose/tree/main/src/main/java/org/raystack/firehose/proto): `src/main/java/org/raystack/firehose/proto`
- [Message serialization](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/serializer/MessageToJson.java): `src/main/java/org/raystack/firehose/serializer/MessageToJson.java`

### Delivery and recovery

- [Sink pool](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/SinkPool.java): `src/main/java/org/raystack/firehose/sink/SinkPool.java`
- [Sink factory](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/SinkFactory.java): `src/main/java/org/raystack/firehose/sink/SinkFactory.java`
- [Retries and DLQ](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sinkdecorator/SinkWithRetry.java): `src/main/java/org/raystack/firehose/sinkdecorator/SinkWithRetry.java`
- [Error handling](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/error/ErrorHandler.java): `src/main/java/org/raystack/firehose/error/ErrorHandler.java`
- [Service and DB sinks](https://github.com/raystack/firehose/tree/main/src/main/java/org/raystack/firehose/sink): `src/main/java/org/raystack/firehose/sink`
- [Blob sink](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/blob/BlobSink.java): `src/main/java/org/raystack/firehose/sink/blob/BlobSink.java`
- [Local file writer](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/blob/writer/local/LocalStorage.java): `src/main/java/org/raystack/firehose/sink/blob/writer/local/LocalStorage.java`
- [Remote uploader](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/blob/writer/remote/BlobStorageChecker.java): `src/main/java/org/raystack/firehose/sink/blob/writer/remote/BlobStorageChecker.java`
- [Blob storage](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/common/blobstorage/BlobStorage.java): `src/main/java/org/raystack/firehose/sink/common/blobstorage/BlobStorage.java`
- [Google Cloud Storage](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/common/blobstorage/gcs/GoogleCloudStorage.java): `src/main/java/org/raystack/firehose/sink/common/blobstorage/gcs/GoogleCloudStorage.java`
- [Blob DLQ writer](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/sink/dlq/blobstorage/BlobStorageDlqWriter.java): `src/main/java/org/raystack/firehose/sink/dlq/blobstorage/BlobStorageDlqWriter.java`

### Observability

- [Instrumentation](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/metrics/FirehoseInstrumentation.java): `src/main/java/org/raystack/firehose/metrics/FirehoseInstrumentation.java`
- [Metric definitions](https://github.com/raystack/firehose/blob/main/src/main/java/org/raystack/firehose/metrics/Metrics.java): `src/main/java/org/raystack/firehose/metrics/Metrics.java`

### Other components

- **Operator**
- **Kafka**
- **StatsD reporter**
- **Configured destinations**

## Connections

- Operator → Runtime entry: starts
- Runtime entry → Consumer factory: creates (evidence: `src/main/java/org/raystack/firehose/launch/Main.java`)
- Runtime entry → Application config: configures
- Consumer factory → Kafka consumer: selects
- Kafka → Kafka consumer: supplies records (evidence: `README.md`)
- Kafka consumer → Message envelope: wraps records
- Offset manager → Kafka consumer: manages offsets (evidence: `src/main/java/org/raystack/firehose/consumer/kafka/ConsumerAndOffsetManager.java`)
- Offset manager → Message envelope: tracks offsets (evidence: `src/main/java/org/raystack/firehose/consumer/kafka/ConsumerAndOffsetManager.java`)
- Message envelope → Message filters: is filtered
- Message envelope → Proto mapping: maps fields
- Message envelope → Message serialization: is serialized
- Sink pool → Sink factory: uses
- Sink factory → Service and DB sinks: creates sink
- Sink factory → Blob sink: creates sink
- Service and DB sinks → Configured destinations: delivers to (evidence: `README.md`)
- Blob sink → Local file writer: writes files
- Local file writer → Remote uploader: uploads files
- Remote uploader → Blob storage: stores files (evidence: `src/main/java/org/raystack/firehose/sink/blob/writer/remote/BlobStorageChecker.java`)
- Blob storage → Google Cloud Storage: implemented by (evidence: `src/main/java/org/raystack/firehose/sink/common/blobstorage/gcs/GoogleCloudStorage.java`)
- Error handling → Blob DLQ writer: routes failures
- Blob DLQ writer → Blob storage: stores messages (evidence: `src/main/java/org/raystack/firehose/sink/dlq/blobstorage/BlobStorageDlqWriter.java`)
- Retries and DLQ → Error handling: handles failures
- Instrumentation → StatsD reporter: reports metrics
- Instrumentation → Metric definitions: uses definitions (evidence: `src/main/java/org/raystack/firehose/metrics/FirehoseInstrumentation.java`)
- Runtime entry → Instrumentation: initializes (evidence: `src/main/java/org/raystack/firehose/launch/Main.java`)
- Offset manager → Instrumentation: records telemetry (evidence: `src/main/java/org/raystack/firehose/consumer/kafka/ConsumerAndOffsetManager.java`)
