How denniswon/evm-indexer works
Project Type and Purpose
• This is a Go-based, monolithic backend service (“Evm-Indexer”) that syncs and persists recent Ethereum/EVM blocks, transactions, and events into a Postgres database, and exposes historical queries via REST & GraphQL APIs as well as real-time notifications over WebSockets. It also uses Redis for pub/sub when dispatching events to subscribers.
File Structure Analysis
• Top-level:
• main.go, Makefile, go.mod/go.sum
• .env.example, scripts/setup.sh, demo/ (JS subscription examples)
• app/ contains all core code, organized into:
• block/ – fetch, pack, publish, retry, sync logic (block and tx processing pipeline)
• client.go – Ethereum JSON-RPC client wrappers
• config/ – configuration loader (.env)
• data/ – domain-level structs for blocks, txs, events
• db/ – Postgres models, queries, transactions
• proto/ – Protobuf definitions for block, event, transaction
• pubsub/ – Redis pub/sub integration for internal or cross-instance messaging
• queue/ – in-memory or Redis-backed job queue abstraction
• rest/ – GraphQL (gqlgen) + REST HTTP server and schema/resolvers
• setup.go, util/ – application initialization, helpers
• External dependencies:
• Ethereum node (HTTP & WS RPC)
• PostgreSQL (with pgcrypto)
• Redis (v6.0.6)
Key Architectural Patterns
• Layered modular monolith: separation into client, domain (data), persistence (db), processing (block/queue), API (rest), integration (pubsub) layers.
• Producer–consumer / worker pool: block syncer pushes jobs to queue; worker pool picks up, fetches data, writes to DB, then publishes events.
• Publish–subscribe: Redis is used to fan out real-time block/tx/event messages to WebSocket subscribers.
• API gateway pattern: single Go binary exposing REST endpoints, GraphQL endpoint, and WebSocket endpoint.
Main Components
• External Systems:
• Ethereum JSON-RPC (HTTP & WebSocket)
• PostgreSQL
• Redis
• Client Apps (demo JS, other consumers)
• Internal Modules:
1. Config Loader
2. EthClient – communicates with Ethereum RPC
3. Block Listener / Syncer – receives new block notifications via WS or polls via HTTP
4. Job Queue – holds block/tx persistence jobs
5. Worker Pool – fetches block data, transactions, logs; packs blocks & txs
6. DB Layer – writes blocks, txs, events to Postgres; runs migrations on startup
7. Pub/Sub Layer – publishes block/tx/event messages to Redis channels
8. API Layer:
• REST Controllers (app/rest/rest.go)
• GraphQL Server (gqlgen: schema.graphqls, resolvers)
• WebSocket Server (/v1/ws subscription handler)
9. Demo Scripts – sample subscription clients in demo/
Relationships and Data Flows
• Ethereum RPC → EthClient → Block Listener
• Block Listener → Job Queue → Worker Pool
• Worker Pool → DB Layer → Postgres
• Worker Pool → Pub/Sub Layer → Redis Channels
• Client Apps connect to:
• REST/GraphQL → API Layer → DB Layer (read queries)
• WebSocket → API Layer → Pub/Sub Layer → Redis → Broadcast to subscribers
Architectural Highlights
• ConcurrencyFactor × CPU cores → worker pool size
• Delayed vs. real-time processing: configurable block confirmations, queue vs. immediate DB write
• GraphQLPlayground toggle for development
• Protobuf definitions support potential gRPC extension
• Separation of domain models (app/data) vs. DB models/queries (app/db)
Essential Elements
• Component names and technology stacks (Go, gqlgen, ethclient, pgx or database/sql, Redis)
• Annotations for key patterns: “Worker Pool,” “Pub/Sub,” “GraphQL Resolver,” “Database ORM/Queries”