How streamingfast/firehose-ethereum works
Project Type and Purpose
• This is a streaming data‐pipeline / CLI service (“fireeth”) for Ethereum data, built in Go.
• It connects to Ethereum clients (via JSON-RPC or a Firehose‐instrumented node), decodes block logs, applies filters/transforms, and exposes data via gRPC or STDOUT to downstream consumers.
File Structure Analysis
• Top-level:
• cmd/fireeth – the main CLI entrypoint, subcommands (poller, bootstrapper, tools)
• blockfetcher – modules fetching blocks (RPC & Firehose dmlog)
• block – conversion from fetched blocks to internal model
• codec – decoding and postprocessing of dmlog formats, metrics
• transform – actions/filters on decoded blocks (log_filter, call_filter, etc.)
• substreams – glue code for StreamingFast’s Substreams pipeline definitions
• proto/types – protobuf schemas and gRPC definitions
• devel – local/dev scripts and configurations for various networks
• tools – shell scripts and utilities for node management
• Dockerfiles & CI workflows for building images and releases
• Patterns:
• Modular pipeline: fetch → convert → decode → transform → serve
• Plugin-style support for multiple input sources (RPC vs dmlog) and networks
• Protobuf/gRPC for API contracts
README Insights
• Emphasizes Firehose concepts: components, data flow, storage, design principles
• Deployable via Homebrew, Docker, or compiled binary
• Uses GitHub Actions + goreleaser for multi‐arch releases
• Integrates with instrumented Ethereum clients (Firehose V2)
a. Main Components
• External Ethereum Node(s)
– JSON-RPC endpoint (e.g. Geth/Parity)
– Firehose‐instrumented node producing dmlog streams
• fireeth CLI/Service
– blockfetcher module (rpc.go, optimism.go)
– converter (block/converter.go)
– codec (parsing, metrics, postprocess)
– transform filters (log_filter, call_filter, combined_filter)
– substreams integration (substreams.yaml, rpccalls.go)
– gRPC server (exposed via proto/sf/ethereum/substreams/v1/rpc.proto)
• Downstream Consumers
– Custom clients via gRPC
– STDOUT pipe (console_reader)
• CI/CD & Deployment
– Dockerfiles, GitHub workflows (buf.yml, docker.yml)
– goreleaser scripts
b. Relationships and Data Flow
1. fireeth starts and connects to Ethereum Node(s)
2. blockfetcher retrieves blocks (RPC calls or dmlog stream)
3. converter transforms raw data into internal Block structs
4. codec decodes, normalizes, and emits intermediate protobuf messages
5. transform modules apply filters/aggregations
6. substreams pipeline optionally further processes data
7. gRPC server pushes streams to clients or writes to STDOUT
c. Architectural Patterns / Principles
• Pipeline architecture: clear stages, each stage testable
• Separation of concerns: fetching, decoding, transforming, serving
• Protocol-first design: protobuf definitions drive code generation
• Extensibility: add new network fetchers or transform modules as plugins
d. Key Technologies
• Go modules & unit tests (block, codec, transform, substreams packages)
• Protocol Buffers & gRPC (proto/ & types/pb/)
• Docker & GitHub Actions for CI/CD
• StreamingFast Substreams framework