# blockchain-etl/ethereum-etl architecture

> Architecture diagram of the GitHub repository blockchain-etl/ethereum-etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/blockchain-etl/ethereum-etl
- Repository: https://github.com/blockchain-etl/ethereum-etl

## Overview

The Ethereum ETL project is a command-line tool and library that extracts Ethereum blockchain data (like blocks, transactions, token transfers, and traces) and converts it into convenient formats such as CSV files or exports it to databases and streaming platforms. The overall design of the system is modular, with distinct components handling data extraction, transformation, and export.

##### Project Type and Purpose
- The project is primarily an ETL (Extract, Transform, Load) tool tailored for Ethereum blockchain data.
- Its purpose is to connect to Ethereum nodes via various provider URIs (RPC, IPC), extract large volumes of blockchain data, transform the data using mapping and conversion utilities, and then export that data to files (CSV), databases (like Postgres), or stream the data to external platforms (Kafka, Google Pub/Sub).

##### Analyzing the File Structure
- Two major modules exist: “blockchainetl” and “ethereumetl”. Although they share similarities, “ethereumetl” is tailored specifically for Ethereum with additional CLI commands and job types.
- Top-level directories include configuration files (Dockerfile, .travis.yml, tox.ini), version control and CI/CD directories (.github), documentation (docs, mkdocs.yml), and schemas (e.g., SQL schema files for AWS).
- Within the code, subdirectories such as “jobs”, “exporters”, “cli”, “domain”, “mappers”, “providers”, “services”, and “streaming” indicate the separation of concerns:
- jobs: Define and manage different ETL tasks (exporting blocks, contracts, receipts, traces, token transfers).
- cli: Evidence of a command-line interface that invokes jobs.
- domain and mappers: Represent the data models and their transformations.
- exporters and converters: Handle the transformation of raw fetched data into user-friendly formats.
- providers: Modules that interface with Ethereum nodes (via RPC, IPC).
- streaming: Components that support continuous data ingestion and streaming to various outputs.
- The “tests” directory further demonstrates the importance of validating each component (jobs, services, streaming, etc.).

##### Key Components and Interactions
a. Data Ingestion:
– Providers (found in ethereumetl/providers and blockchainetl providers) connect to Ethereum nodes using provider URIs.
– They interact with the Ethereum network to get blocks and related data.
b. Data Processing Pipeline:
– CLI commands invoke extraction jobs (in ethereumetl/cli) which in turn use job modules (jobs/ directory) to coordinate ETL operations.
– Mappers (in ethereumetl/mappers) convert raw RPC responses to structured domain objects.
– Converters (found in blockchainetl/jobs/exporters/converters) further transform numeric or data types (e.g., converting integer to decimal).
c. Data Export:
– Exporters (modules under jobs/exporters) write data into CSV formats, databases (like Postgres), or stream them to external systems (Kafka, Google Pub/Sub, etc.).
– A separate streaming module supports continuous data pipelines.
d. Execution and Concurrency:
– The executors (batch_work_executor, bounded_executor, fail_safe_executor) manage concurrent job execution, ensuring robust and efficient pipeline operation.
e. Utility and Support:
– Additional modules (logging_utils, file_utils, progress_logger) provide logging, status monitoring, and helpful utilities.
– Docker and CI/CD configurations indicate the deployment and integration pipeline for the project.

##### Architectural Patterns and Design Principles
- Modular design: Components are clearly separated by functionality (extraction jobs, mapping logic, data export), making the system extensible and maintainable.
- Pipeline architecture: Data flows from extraction (providers) → transformation (mappers, converters) → export (exporters, streaming).
- Command-driven interface: The CLI commands initiate ETL workflows that chain together these components.
- Robust execution: Use of different executors to handle batch processing and fault tolerance.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI and Execution
    CLI["CLI Interface"]:::cli
    EXEC["ETL Executors"]:::executor

    CLI -->|"triggers"| EXEC

    %% ETL Core Jobs
    subgraph "ETL Core Jobs"
        ETH_JOB["EthereumETL Jobs"]:::extraction
        BCH_JOB["BlockchainETL Jobs"]:::extraction
        EXEC --> ETH_JOB
        EXEC --> BCH_JOB
    end

    %% Data Providers
    subgraph "Data Providers"
        ETH_PROVIDER["EthereumETL Providers"]:::provider
        BCH_PROVIDER["BlockchainETL Providers"]:::provider
        ETH_JOB -->|"fetches data"| ETH_PROVIDER
        BCH_JOB -->|"fetches data"| BCH_PROVIDER
    end

    %% External Ethereum Node
    EXT_ETHNODE["Ethereum Node"]:::external
    ETH_PROVIDER -->|"RPC/IPC calls"| EXT_ETHNODE
    BCH_PROVIDER -->|"RPC/IPC calls"| EXT_ETHNODE

    %% Transformation Layer
    subgraph "Transformation Layer"
        ETH_MAPPER["EthereumETL Mappers"]:::transformation
        BCH_CONVERTER["BlockchainETL Converters"]:::transformation
        ETH_JOB -->|"processes"| ETH_MAPPER
        BCH_JOB -->|"processes"| BCH_CONVERTER
    end

    %% Domain Model
    DOMAIN["Domain Model"]:::domain
    ETH_MAPPER --> DOMAIN
    BCH_CONVERTER --> DOMAIN

    %% Data Export
    subgraph "Data Export"
        ETH_EXPORTER["EthereumETL Exporters"]:::exporter
        BCH_EXPORTER["BlockchainETL Exporters"]:::exporter
        ETH_MAPPER -->|"transforms"| ETH_EXPORTER
        BCH_CONVERTER -->|"transforms"| BCH_EXPORTER
    end

    %% Streaming Module
    subgraph "Streaming Module"
        ETH_STREAMING["EthereumETL Streaming"]:::streaming
        BCH_STREAMING["BlockchainETL Streaming"]:::streaming
        ETH_MAPPER -->|"optional stream"| ETH_STREAMING
        BCH_CONVERTER -->|"optional stream"| BCH_STREAMING
    end

    %% External Destination Systems
    EXT_DEST["External Destination"]:::external
    ETH_EXPORTER -->|"exports to"| EXT_DEST
    BCH_EXPORTER -->|"exports to"| EXT_DEST
    ETH_STREAMING -->|"streams to"| EXT_DEST
    BCH_STREAMING -->|"streams to"| EXT_DEST

    %% Utility and Support Modules
    subgraph "Utility & Support"
        UTIL_LOG["Logging Utils"]:::utility
        UTIL_FILE["File Utils"]:::utility
        UTIL_PROG["Progress Logger"]:::utility
        UTIL_MISC["Misc Utils"]:::utility
    end
    EXEC --- UTIL_LOG
    EXEC --- UTIL_FILE
    EXEC --- UTIL_PROG
    EXEC --- UTIL_MISC

    %% Click Events
    click CLI "https://github.com/blockchain-etl/ethereum-etl/tree/develop/ethereumetl/cli"
    click ETH_JOB "https://github.com/blockchain-etl/ethereum-etl/tree/develop/ethereumetl/jobs"
    click BCH_JOB "https://github.com/blockchain-etl/ethereum-etl/tree/develop/blockchainetl/jobs"
    click ETH_PROVIDER "https://github.com/blockchain-etl/ethereum-etl/tree/develop/ethereumetl/providers"
    click BCH_PROVIDER "https://github.com/blockchain-etl/ethereum-etl/tree/develop/blockchainetl"
    click ETH_MAPPER "https://github.com/blockchain-etl/ethereum-etl/tree/develop/ethereumetl/mappers"
    click BCH_CONVERTER "https://github.com/blockchain-etl/ethereum-etl/tree/develop/blockchainetl/jobs/exporters/converters"
    click ETH_EXPORTER "https://github.com/blockchain-etl/ethereum-etl/tree/develop/ethereumetl/jobs/exporters"
    click BCH_EXPORTER "https://github.com/blockchain-etl/ethereum-etl/tree/develop/blockchainetl/jobs/exporters"
    click ETH_STREAMING "https://github.com/blockchain-etl/ethereum-etl/tree/develop/ethereumetl/streaming"
    click BCH_STREAMING "https://github.com/blockchain-etl/ethereum-etl/tree/develop/blockchainetl/streaming"
    click EXEC "https://github.com/blockchain-etl/ethereum-etl/tree/develop/ethereumetl/executors"
    click DOMAIN "https://github.com/blockchain-etl/ethereum-etl/tree/develop/ethereumetl/domain"
    click UTIL_LOG "https://github.com/blockchain-etl/ethereum-etl/blob/develop/blockchainetl/logging_utils.py"
    click UTIL_FILE "https://github.com/blockchain-etl/ethereum-etl/blob/develop/blockchainetl/file_utils.py"
    click UTIL_PROG "https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/progress_logger.py"
    click UTIL_MISC "https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/misc_utils.py"

    %% Styles
    classDef cli fill:#A2D9CE,stroke:#333,stroke-width:2px;
    classDef executor fill:#F9E79F,stroke:#333,stroke-width:2px;
    classDef extraction fill:#F5B7B1,stroke:#333,stroke-width:2px;
    classDef provider fill:#AED6F1,stroke:#333,stroke-width:2px;
    classDef transformation fill:#D2B4DE,stroke:#333,stroke-width:2px;
    classDef domain fill:#FAD7A0,stroke:#333,stroke-width:2px;
    classDef exporter fill:#C39BD3,stroke:#333,stroke-width:2px;
    classDef streaming fill:#A9DFBF,stroke:#333,stroke-width:2px;
    classDef utility fill:#E8DAEF,stroke:#333,stroke-width:2px;
    classDef external fill:#D6EAF8,stroke:#333,stroke-width:2px;
```
