How raystack/firehose works
Firehose is a Kafka-to-destination streaming service. Its principal workflow starts at the service runtime, configures a Kafka consumer, processes messages, and delivers them to a configured sink; filtering, error handling, retries, dead-letter output, and metrics support that flow. The README documents the sink capabilities, while sampled source and the SOURCE INDEX establish several internal relationships. Some sink-specific wiring is not shown, so the graph keeps those branches at the documented capability level and avoids asserting unverified calls.
Main entry points: Message.java Message envelope · Main.java Runtime entry · FirehoseInstrumentation.java Instrumentation · FirehoseKafkaConsumer.java Kafka consumer
Components 26
Runtime and input
- Runtime entry
src/main/java/org/raystack/firehose/launch/Main.java - Application config
src/main/java/org/raystack/firehose/config/AppConfig.java - Consumer factory
src/main/java/org/raystack/firehose/consumer/FirehoseConsumerFactory.java - Kafka consumer
src/main/java/org/raystack/firehose/consumer/kafka/FirehoseKafkaConsumer.java - Offset manager
src/main/java/org/raystack/firehose/consumer/kafka/ConsumerAndOffsetManager.java
Message processing
- Message envelope
src/main/java/org/raystack/firehose/message/Message.java - Message filters
src/main/java/org/raystack/firehose/filter - Proto mapping
src/main/java/org/raystack/firehose/proto - Message serialization
src/main/java/org/raystack/firehose/serializer/MessageToJson.java
Delivery and recovery
- Sink pool
src/main/java/org/raystack/firehose/sink/SinkPool.java - Sink factory
src/main/java/org/raystack/firehose/sink/SinkFactory.java - Retries and DLQ
src/main/java/org/raystack/firehose/sinkdecorator/SinkWithRetry.java - Error handling
src/main/java/org/raystack/firehose/error/ErrorHandler.java - Service and DB sinks
src/main/java/org/raystack/firehose/sink - Blob sink
src/main/java/org/raystack/firehose/sink/blob/BlobSink.java - Local file writer
src/main/java/org/raystack/firehose/sink/blob/writer/local/LocalStorage.java - Remote uploader
src/main/java/org/raystack/firehose/sink/blob/writer/remote/BlobStorageChecker.java - Blob storage
src/main/java/org/raystack/firehose/sink/common/blobstorage/BlobStorage.java - Google Cloud Storage
src/main/java/org/raystack/firehose/sink/common/blobstorage/gcs/GoogleCloudStorage.java - Blob DLQ writer
src/main/java/org/raystack/firehose/sink/dlq/blobstorage/BlobStorageDlqWriter.java
Observability
- Instrumentation
src/main/java/org/raystack/firehose/metrics/FirehoseInstrumentation.java - Metric definitions
src/main/java/org/raystack/firehose/metrics/Metrics.java
Other components
- Operator
- Kafka
- StatsD reporter
- Configured destinations
Connections 26
- Operator → Runtime entry: starts
- Runtime entry → Consumer factory: creates
src/main/java/org/raystack/firehose/launch/Main.java - Runtime entry → Application config: configures
- Consumer factory → Kafka consumer: selects
- Kafka → Kafka consumer: supplies records
README.md - Kafka consumer → Message envelope: wraps records
- Offset manager → Kafka consumer: manages offsets
src/main/java/org/raystack/firehose/consumer/kafka/ConsumerAndOffsetManager.java - Offset manager → Message envelope: tracks offsets
src/main/java/org/raystack/firehose/consumer/kafka/ConsumerAndOffsetManager.java - Message envelope → Message filters: is filtered
- Message envelope → Proto mapping: maps fields
- Message envelope → Message serialization: is serialized
- Sink pool → Sink factory: uses
- Sink factory → Service and DB sinks: creates sink
- Sink factory → Blob sink: creates sink
- Service and DB sinks → Configured destinations: delivers to
README.md - Blob sink → Local file writer: writes files
- Local file writer → Remote uploader: uploads files
- Remote uploader → Blob storage: stores files
src/main/java/org/raystack/firehose/sink/blob/writer/remote/BlobStorageChecker.java - Blob storage → Google Cloud Storage: implemented by
src/main/java/org/raystack/firehose/sink/common/blobstorage/gcs/GoogleCloudStorage.java - Error handling → Blob DLQ writer: routes failures
- Blob DLQ writer → Blob storage: stores messages
src/main/java/org/raystack/firehose/sink/dlq/blobstorage/BlobStorageDlqWriter.java - Retries and DLQ → Error handling: handles failures
- Instrumentation → StatsD reporter: reports metrics
- Instrumentation → Metric definitions: uses definitions
src/main/java/org/raystack/firehose/metrics/FirehoseInstrumentation.java - Runtime entry → Instrumentation: initializes
src/main/java/org/raystack/firehose/launch/Main.java - Offset manager → Instrumentation: records telemetry
src/main/java/org/raystack/firehose/consumer/kafka/ConsumerAndOffsetManager.java