# streamingfast/firehose-ethereum architecture

> Architecture diagram of the GitHub repository streamingfast/firehose-ethereum, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/streamingfast/firehose-ethereum
- Repository: https://github.com/streamingfast/firehose-ethereum

## Overview

##### Project Type and Purpose
- This is a streaming data‐pipeline / CLI service (“fireeth”) for Ethereum data, built in Go.
- It connects to Ethereum clients (via JSON-RPC or a Firehose‐instrumented node), decodes block logs, applies filters/transforms, and exposes data via gRPC or STDOUT to downstream consumers.

##### File Structure Analysis
- Top-level:
- cmd/fireeth – the main CLI entrypoint, subcommands (poller, bootstrapper, tools)
- blockfetcher – modules fetching blocks (RPC & Firehose dmlog)
- block – conversion from fetched blocks to internal model
- codec – decoding and postprocessing of dmlog formats, metrics
- transform – actions/filters on decoded blocks (log_filter, call_filter, etc.)
- substreams – glue code for StreamingFast’s Substreams pipeline definitions
- proto/types – protobuf schemas and gRPC definitions
- devel – local/dev scripts and configurations for various networks
- tools – shell scripts and utilities for node management
- Dockerfiles & CI workflows for building images and releases
- Patterns:
- Modular pipeline: fetch → convert → decode → transform → serve
- Plugin-style support for multiple input sources (RPC vs dmlog) and networks
- Protobuf/gRPC for API contracts

##### README Insights
- Emphasizes Firehose concepts: components, data flow, storage, design principles
- Deployable via Homebrew, Docker, or compiled binary
- Uses GitHub Actions + goreleaser for multi‐arch releases
- Integrates with instrumented Ethereum clients (Firehose V2)

a. Main Components
- External Ethereum Node(s)
– JSON-RPC endpoint (e.g. Geth/Parity)
– Firehose‐instrumented node producing dmlog streams
- fireeth CLI/Service
– blockfetcher module (rpc.go, optimism.go)
– converter (block/converter.go)
– codec (parsing, metrics, postprocess)
– transform filters (log_filter, call_filter, combined_filter)
– substreams integration (substreams.yaml, rpccalls.go)
– gRPC server (exposed via proto/sf/ethereum/substreams/v1/rpc.proto)
- Downstream Consumers
– Custom clients via gRPC
– STDOUT pipe (console_reader)
- CI/CD & Deployment
– Dockerfiles, GitHub workflows (buf.yml, docker.yml)
– goreleaser scripts

b. Relationships and Data Flow
1. fireeth starts and connects to Ethereum Node(s)
2. blockfetcher retrieves blocks (RPC calls or dmlog stream)
3. converter transforms raw data into internal Block structs
4. codec decodes, normalizes, and emits intermediate protobuf messages
5. transform modules apply filters/aggregations
6. substreams pipeline optionally further processes data
7. gRPC server pushes streams to clients or writes to STDOUT

c. Architectural Patterns / Principles
- Pipeline architecture: clear stages, each stage testable
- Separation of concerns: fetching, decoding, transforming, serving
- Protocol-first design: protobuf definitions drive code generation
- Extensibility: add new network fetchers or transform modules as plugins

d. Key Technologies
- Go modules & unit tests (block, codec, transform, substreams packages)
- Protocol Buffers & gRPC (proto/ & types/pb/)
- Docker & GitHub Actions for CI/CD
- StreamingFast Substreams framework

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layer 1 - External Systems
    subgraph "Layer 1: External Systems"
        direction TB
        ETH_RPC["/ Ethereum JSON-RPC Endpoint"]:::input
        ETH_FH["/ Firehose Instrumented Node (dmlog Stream)"]:::input
    end

    %% Layer 2 - fireeth Pipeline
    subgraph "Layer 2: fireeth Pipeline"
        direction TB
        CLI["CLI/Service (cmd/fireeth)"]:::pipeline
        BF["Blockfetcher (blockfetcher)"]:::pipeline
        CV["Converter (block/converter.go)"]:::pipeline
        CD["Codec (codec)"]:::pipeline
        TF["Transform (transform)"]:::pipeline
        SS["Substreams Integration"]:::pipeline
        subgraph "Schema Definitions"
            direction TB
            RPC_PROTO["RPC Proto (rpc.proto)"]:::pipeline
            TRANSFORM_PROTO["Transform Proto (transforms.proto)"]:::pipeline
            TYPE_PROTO["Type Proto (type.proto)"]:::pipeline
        end
    end

    %% Layer 3 - API & Consumers
    subgraph "Layer 3: API & Consumers"
        direction TB
        GRPC["gRPC Server"]:::api
        STDOUT["STDOUT Pipe (console_reader)"]:::api
    end

    %% Layer 4 - CI/CD & Dev Environments
    subgraph "Layer 4: CI/CD & Dev & Tools"
        direction TB
        DEV["Dev Scripts (devel/)"]:::dev
        TOOLS["Utilities (tools/fireeth/)"]:::dev
        DOCKER["Dockerfiles"]:::dev
        CI["GitHub Workflows"]:::dev
    end

    %% Connections
    ETH_RPC -->|connects| CLI
    ETH_FH -->|connects| CLI
    CLI -->|fetch blocks| BF
    BF -->|raw blocks| CV
    CV -->|internal model| CD
    CD -->|decoded protobuf| TF
    TF -->|filtered data| SS
    SS -->|stream data| GRPC
    SS -->|stream data| STDOUT

    %% Click Events
    click CLI "https://github.com/streamingfast/firehose-ethereum/blob/develop/cmd/fireeth/main.go"
    click BF "https://github.com/streamingfast/firehose-ethereum/blob/develop/blockfetcher/rpc.go"
    click CV "https://github.com/streamingfast/firehose-ethereum/blob/develop/block/converter.go"
    click CD "https://github.com/streamingfast/firehose-ethereum/blob/develop/codec/postprocess.go"
    click TF "https://github.com/streamingfast/firehose-ethereum/blob/develop/transform/log_filter.go"
    click SS "https://github.com/streamingfast/firehose-ethereum/blob/develop/substreams/substreams.yaml"
    click RPC_PROTO "https://github.com/streamingfast/firehose-ethereum/blob/develop/proto/sf/ethereum/substreams/v1/rpc.proto"
    click TRANSFORM_PROTO "https://github.com/streamingfast/firehose-ethereum/blob/develop/proto/sf/ethereum/transform/v1/transforms.proto"
    click TYPE_PROTO "https://github.com/streamingfast/firehose-ethereum/blob/develop/proto/sf/ethereum/type/v2/type.proto"
    click DEV "https://github.com/streamingfast/firehose-ethereum/tree/develop/devel/"
    click TOOLS "https://github.com/streamingfast/firehose-ethereum/tree/develop/tools/fireeth/"
    click DOCKER "https://github.com/streamingfast/firehose-ethereum/tree/develop/Dockerfile"
    click CI "https://github.com/streamingfast/firehose-ethereum/blob/develop/.github/workflows/docker.yml"

    %% Styles
    classDef input fill:#B3DDF2,stroke:#333,stroke-width:1px
    classDef pipeline fill:#C2F0C2,stroke:#333,stroke-width:1px
    classDef api fill:#FFD59E,stroke:#333,stroke-width:1px
    classDef dev fill:#E0E0E0,stroke:#333,stroke-width:1px
```
