# denniswon/evm-indexer architecture

> Architecture diagram of the GitHub repository denniswon/evm-indexer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/denniswon/evm-indexer
- Repository: https://github.com/denniswon/evm-indexer

## Overview

##### Project Type and Purpose
- This is a Go-based, monolithic backend service (“Evm-Indexer”) that syncs and persists recent Ethereum/EVM blocks, transactions, and events into a Postgres database, and exposes historical queries via REST & GraphQL APIs as well as real-time notifications over WebSockets. It also uses Redis for pub/sub when dispatching events to subscribers.

##### File Structure Analysis
- Top-level:
- main.go, Makefile, go.mod/go.sum
- .env.example, scripts/setup.sh, demo/ (JS subscription examples)
- app/ contains all core code, organized into:
- block/ – fetch, pack, publish, retry, sync logic (block and tx processing pipeline)
- client.go – Ethereum JSON-RPC client wrappers
- config/ – configuration loader (.env)
- data/ – domain-level structs for blocks, txs, events
- db/ – Postgres models, queries, transactions
- proto/ – Protobuf definitions for block, event, transaction
- pubsub/ – Redis pub/sub integration for internal or cross-instance messaging
- queue/ – in-memory or Redis-backed job queue abstraction
- rest/ – GraphQL (gqlgen) + REST HTTP server and schema/resolvers
- setup.go, util/ – application initialization, helpers
- External dependencies:
- Ethereum node (HTTP & WS RPC)
- PostgreSQL (with pgcrypto)
- Redis (v6.0.6)

##### Key Architectural Patterns
- Layered modular monolith: separation into client, domain (data), persistence (db), processing (block/queue), API (rest), integration (pubsub) layers.
- Producer–consumer / worker pool: block syncer pushes jobs to queue; worker pool picks up, fetches data, writes to DB, then publishes events.
- Publish–subscribe: Redis is used to fan out real-time block/tx/event messages to WebSocket subscribers.
- API gateway pattern: single Go binary exposing REST endpoints, GraphQL endpoint, and WebSocket endpoint.

##### Main Components
- External Systems:
- Ethereum JSON-RPC (HTTP & WebSocket)
- PostgreSQL
- Redis
- Client Apps (demo JS, other consumers)
- Internal Modules:
1. Config Loader
2. EthClient – communicates with Ethereum RPC
3. Block Listener / Syncer – receives new block notifications via WS or polls via HTTP
4. Job Queue – holds block/tx persistence jobs
5. Worker Pool – fetches block data, transactions, logs; packs blocks & txs
6. DB Layer – writes blocks, txs, events to Postgres; runs migrations on startup
7. Pub/Sub Layer – publishes block/tx/event messages to Redis channels
8. API Layer:
- REST Controllers (app/rest/rest.go)
- GraphQL Server (gqlgen: schema.graphqls, resolvers)
- WebSocket Server (/v1/ws subscription handler)
9. Demo Scripts – sample subscription clients in demo/

##### Relationships and Data Flows
- Ethereum RPC → EthClient → Block Listener
- Block Listener → Job Queue → Worker Pool
- Worker Pool → DB Layer → Postgres
- Worker Pool → Pub/Sub Layer → Redis Channels
- Client Apps connect to:
- REST/GraphQL → API Layer → DB Layer (read queries)
- WebSocket → API Layer → Pub/Sub Layer → Redis → Broadcast to subscribers

##### Architectural Highlights
- ConcurrencyFactor × CPU cores → worker pool size
- Delayed vs. real-time processing: configurable block confirmations, queue vs. immediate DB write
- GraphQLPlayground toggle for development
- Protobuf definitions support potential gRPC extension
- Separation of domain models (app/data) vs. DB models/queries (app/db)

##### Essential Elements
- Component names and technology stacks (Go, gqlgen, ethclient, pgx or database/sql, Redis)
- Annotations for key patterns: “Worker Pool,” “Pub/Sub,” “GraphQL Resolver,” “Database ORM/Queries”

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    subgraph "Clients & External" 
        direction TB
        EthRPC["Ethereum JSON-RPC\n(HTTP & WS)"]:::external
        Postgres["PostgreSQL"]:::external
        Redis["Redis Pub/Sub"]:::external
        ClientApps["Client Apps\n(demo JS, other)"]:::external
    end

    %% Config & Utils
    subgraph "Config & Utils"
        direction TB
        ConfigLoader["Config Loader\n(BlockConfirmations,\nConcurrencyFactor)"]:::config
        Setup["App Initialization"]:::config
        Util["Utilities"]:::config
        DomainModels["Domain Models"]:::domain
        ProtobufDefs["Protobuf Definitions"]:::domain
    end

    %% Ingestion & Processing Pipeline
    subgraph "Ingestion & Processing" 
        direction TB
        EthClient["EthClient\n(ethclient wrappers)"]:::processing
        BlockListener["Block Listener / Syncer"]:::processing
        JobQueue["Job Queue"]:::processing
        WorkerPool["Worker Pool"]:::processing
    end

    %% Persistence
    subgraph "Persistence"
        direction TB
        DBLayer["DB Layer\n(models & queries)"]:::storage
    end

    %% Real-Time Pub/Sub
    subgraph "Real-Time Pub/Sub"
        direction TB
        PubSubLayer["Pub/Sub Layer"]:::processing
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        RESTAPI["REST API\n(/v1/block, etc.)"]:::api
        GraphQLAPI["GraphQL API\n(/v1/graphql)"]:::api
        WebSocketAPI["WebSocket API\n(/v1/ws)"]:::api
    end

    %% Demo Scripts
    DemoScripts["Demo Scripts"]:::external

    %% Connections
    EthRPC -->|HTTP/WS RPC| EthClient
    EthClient -->|new blocks| BlockListener
    BlockListener -->|enqueue jobs| JobQueue
    JobQueue -->|distribute jobs| WorkerPool
    WorkerPool -->|write| DBLayer
    WorkerPool -->|publish| PubSubLayer
    RESTAPI -->|read queries| DBLayer
    GraphQLAPI -->|read queries| DBLayer
    WebSocketAPI -->|subscribe| PubSubLayer
    PubSubLayer -->|fan-out| Redis
    ClientApps -->|HTTP| RESTAPI
    ClientApps -->|HTTP| GraphQLAPI
    ClientApps -->|WebSocket| WebSocketAPI
    DemoScripts -->|samples| ClientApps

    ConfigLoader -->|load settings| EthClient
    ConfigLoader --> Setup
    Setup --> ConfigLoader
    Setup --> DBLayer
    Setup --> PubSubLayer
    Setup --> RESTAPI
    Setup --> GraphQLAPI
    Setup --> WebSocketAPI

    %% Click Events
    click ConfigLoader "https://github.com/denniswon/evm-indexer/blob/main/app/config/config.go"
    click Setup "https://github.com/denniswon/evm-indexer/blob/main/app/setup.go"
    click Util "https://github.com/denniswon/evm-indexer/blob/main/app/util/util.go"
    click DomainModels "https://github.com/denniswon/evm-indexer/blob/main/app/data/data.go"
    click ProtobufDefs "https://github.com/denniswon/evm-indexer/blob/main/app/proto/block.proto"
    click EthClient "https://github.com/denniswon/evm-indexer/blob/main/app/client.go"
    click BlockListener "https://github.com/denniswon/evm-indexer/blob/main/app/block/listener.go"
    click JobQueue "https://github.com/denniswon/evm-indexer/blob/main/app/queue/queue.go"
    click DBLayer "https://github.com/denniswon/evm-indexer/blob/main/app/db/db.go"
    click PubSubLayer "https://github.com/denniswon/evm-indexer/blob/main/app/pubsub/consumption.go"
    click RESTAPI "https://github.com/denniswon/evm-indexer/blob/main/app/rest/rest.go"
    click GraphQLAPI "https://github.com/denniswon/evm-indexer/blob/main/app/rest/graph/schema.graphqls"
    click DemoScripts "https://github.com/denniswon/evm-indexer/blob/main/demo/all_subscription.js"

    %% Styles
    classDef external fill:#D8BFD8,stroke:#9370DB,stroke-width:2px
    classDef config fill:#D3D3D3,stroke:#A9A9A9,stroke-width:2px
    classDef domain fill:#F0F8FF,stroke:#4682B4,stroke-width:2px
    classDef processing fill:#ADD8E6,stroke:#1E90FF,stroke-width:2px
    classDef storage fill:#E6E6FA,stroke:#6A5ACD,stroke-width:2px
    classDef api fill:#90EE90,stroke:#228B22,stroke-width:2px
```
