# dodoex/web3-rpc-proxy architecture

> Architecture diagram of the GitHub repository dodoex/web3-rpc-proxy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dodoex/web3-rpc-proxy
- Repository: https://github.com/dodoex/web3-rpc-proxy

## Overview

##### Project Type and Purpose
- This is a backend middleware service (“Web3 RPC Proxy”) written in Go.
- It exposes a JSON‑RPC HTTP API (using fasthttp) to upstream clients, implements multi‑tenant rate limiting, caching, retries and endpoint selection, and forwards requests to one or more EVM blockchain nodes.
- It supports dynamic configuration via etcd, persistence via PostgreSQL, distributed rate limits via Redis, and optional event publication via AMQP.

##### File Structure
- cmd/main.go: application entrypoint and server bootstrap.
- config/: YAML and JSON config templates (default.yaml, ethereum‑openrpc.json, grafana.json).
- internal/app/: “Agent” MVC‑style module holding controllers (HTTP handlers), services (business logic), repositories (DB access), plus shared modules (config loader, etcd, logger, Redis, AMQP).
- internal/core/: core RPC and endpoint‑selection engine, request context, JSON‑RPC validation.
- internal/app/database/: DB schema, migrations, seeding (tenant table).
- providers/web3‑rpc‑provider: external package for discovering and wrapping free RPC endpoints.
- utils/: shared helper functions (config parsing, compression, sharding, constants, metrics).
- docker-compose.yaml / Dockerfile: containerization and multi‑service orchestration.

##### README for Additional Insights
- The README confirms a “cluster‑deployable EVM RPC proxy” pattern.
- It describes dependencies (PostgreSQL, Redis, etcd, AMQP) and features (caching, rate limits, retries, JSON‑RPC schema validation, Prometheus metrics, Grafana dashboards).

##### Main Components & Their Roles
a. API Layer (fasthttp)
- Accepts inbound JSON‑RPC batches at “/<CHAIN>”.
- Parses headers (x_api_key, x_api_bucket, cache flags).
b. Controller (“agent_controller.go”)
- Translates HTTP request into internal ReqCtx.
- Applies parameter validation.
- Calls the AgentService.
c. Service Layer (agent_service.go)
- Enforces tenant isolation, rate limiting (via Redis), caching (via Redis scripts), retries (rotation or same), timeout handling.
- Publishes events to AMQP if configured.
- Delegates the actual JSON‑RPC call to the core endpoint client.
d. Core Endpoint Engine (internal/core/endpoint)
- Maintains a pool of HTTP/WebSocket clients to blockchain nodes.
- Implements endpoint scoring, selection strategies, failover.
e. Persistence & Config Stores
- PostgreSQL (tenant metadata via repository).
- etcd (dynamic config for chains and endpoints).
- redis (distributed rate‑limiting counters, cache).
f. Optional MQ (AMQP)
- Publish each request/response record for auditing or analytics.
g. External Blockchains
- Multiple EVM JSON‑RPC endpoints (HTTP or WSS).

##### Relationships & Data Flows
1. Client → HTTP server (fasthttp) → AgentController
2. Controller → AgentService
3. AgentService ←→ Redis (rate limit & cache)
4. AgentService → PostgreSQL via TenantRepository (tenant metadata)
5. AgentService → CoreEndpointClient → External EVM RPC nodes
6. AgentService → etcd (read config)
7. AgentService → AMQP (publish events)
8. Prometheus scrape metrics from in‑process registry

##### Architectural Patterns & Principles
- Layered / Clean Architecture: clear separation between HTTP adapter (controllers), business logic (services), infrastructure (repositories, shared modules).
- Hexagonal touches in core endpoint module: pluggable HTTP/WebSocket transports.
- Configuration‑as‑code in etcd via shared config loader.
- Dependency Injection via module wiring in internal/app/bootstrap.go.

##### Key Technologies & Frameworks
- Go 1.21, Fasthttp, Viper (for config), fx or a simple DI pattern in bootstrap.
- PostgreSQL (tenant table), Redis (cache & counters), etcd (config), RabbitMQ/AMQP.
- Prometheus client for metrics, Grafana dashboards.
- Docker & Docker‑Compose for containerization.

##### Final Checklist
- All major modules labeled clearly.
- Notes on protocols (HTTP, JSON‑RPC, AMQP, gRPC etc. if any).
- Deployment context: one node of cluster behind a load balancer; scalable Redis, etcd and PostgreSQL clusters; Grafana dashboards.
- Call out Docker Compose / Dockerfile for container orchestration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    LB["Load Balancer"]:::external
    LB --> API["fasthttp API Gateway"]:::api
    click API "https://github.com/dodoex/web3-rpc-proxy/blob/main/cmd/main.go"

    subgraph "API & Adapters"
        Router["router.go"]:::api
        click Router "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/router.go"
        API --> Router

        C1["AgentController"]:::api
        click C1 "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/agent/controller/agent_controller.go"
        Router --> C1

        C2["OtherController"]:::api
        click C2 "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/agent/controller/other_controller.go"
        Router --> C2
    end

    subgraph "Business Logic"
        S1["AgentService"]:::service
        click S1 "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/agent/service/agent_service.go"
        S2["TenantService"]:::service
        click S2 "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/agent/service/tenant_service.go"
        S3["EndpointService"]:::service
        click S3 "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/agent/service/endpoint_service.go"
        CacheLogic["Cache Logic"]:::service
        click CacheLogic "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/agent/service/cache.go"

        C1 --> S1
        S1 --> S2
        S1 --> S3
        S1 --> CacheLogic
    end

    subgraph "Core Endpoint Engine"
        ECHTTP["http_client.go"]:::core
        click ECHTTP "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/core/endpoint/http_client.go"
        ECWS["websocket_client.go"]:::core
        click ECWS "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/core/endpoint/websocket_client.go"
        Selector["selector.go"]:::core
        click Selector "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/core/endpoint/selector.go"
        ProviderComp["provider.go"]:::core
        click ProviderComp "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/core/endpoint/provider.go"
        EndpointComp["endpoint.go"]:::core
        click EndpointComp "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/core/endpoint/endpoint.go"
        CoreCache["core cache.go"]:::core
        click CoreCache "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/core/endpoint/cache.go"

        EndpointClient["EndpointClient"]:::core
        EndpointClient --> ECHTTP
        EndpointClient --> ECWS
        EndpointClient --> Selector
        EndpointClient --> ProviderComp
        EndpointClient --> EndpointComp
        EndpointClient --> CoreCache
    end

    S3 --> EndpointClient

    ReqCtx["ReqCtx"]:::core
    click ReqCtx "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/core/reqctx/reqctxs.go"
    S1 --> ReqCtx

    subgraph "Core RPC & Validation"
        JSONRPC["jsonrpc.go"]:::core
        click JSONRPC "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/core/rpc/jsonrpc.go"
        JSONSchema["jsonrpc_schema.go"]:::core
        click JSONSchema "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/core/rpc/jsonrpc_schema.go"
        S1 --> JSONRPC
        JSONRPC --> JSONSchema
    end

    subgraph "Shared Modules"
        ConfigLoader["config.go"]:::infra
        click ConfigLoader "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/shared/config.go"
        EtcdComp["etcd.go"]:::infra
        click EtcdComp "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/shared/etcd.go"
        RedisComp["redis.go"]:::infra
        click RedisComp "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/shared/redis.go"
        RedisScripts["redis_scripts.go"]:::infra
        click RedisScripts "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/shared/redis_scripts.go"
        AMQPComp["amqp.go"]:::infra
        click AMQPComp "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/shared/amqp.go"
        LoggerComp["logger.go"]:::infra
        click LoggerComp "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/shared/logger.go"
        WatcherComp["watcher.go"]:::infra
        click WatcherComp "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/shared/watcher.go"
        TransportComp["transport.go"]:::infra
        click TransportComp "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/shared/transport.go"
        Bootstrap["bootstrap.go"]:::infra
        click Bootstrap "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/bootstrap.go"
        ApplicationComp["application.go"]:::infra
        click ApplicationComp "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/application.go"

        S1 --> RedisComp
        S1 --> RedisScripts
        S1 --> ConfigLoader
        S1 --> EtcdComp
        S1 --> AMQPComp
    end

    subgraph "Repositories & DB"
        TenantRepo["TenantRepository"]:::repo
        click TenantRepo "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/agent/repository/tenant_repository.go"
        DBModule["database.go"]:::db
        click DBModule "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/database/database.go"
        SchemaBase["schema/base.go"]:::db
        click SchemaBase "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/database/schema/base.go"
        SchemaTenant["schema/tenant.go"]:::db
        click SchemaTenant "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/database/schema/tenant.go"
        Seeder["tenant_seeder.go"]:::db
        click Seeder "https://github.com/dodoex/web3-rpc-proxy/blob/main/internal/app/database/seeds/tenant_seeder.go"

        S2 --> TenantRepo
        TenantRepo --> DBModule
        DBModule --> SchemaBase
        DBModule --> SchemaTenant
        Seeder -.-> DBModule
    end

    subgraph "External Systems"
        Postgres["PostgreSQL"]:::storage
        RedisDS["Redis"]:::storage
        EtcdDS["etcd"]:::storage
        AMQPDS["AMQP"]:::storage
        EVMNodes["EVM RPC Nodes"]:::external
        Prom["Prometheus"]:::external
        Graf["Grafana"]:::external

        DBModule --> Postgres
        RedisComp --> RedisDS
        EtcdComp --> EtcdDS
        AMQPComp --> AMQPDS
        EndpointClient --> EVMNodes
        S1 --> Prom
        Prom --> Graf
    end

    Docs["docs/architecture.png"]:::doc
    click Docs "https://github.com/dodoex/web3-rpc-proxy/blob/main/docs/architecture.png"

    subgraph "Containerization"
        Dockerfile["Dockerfile"]:::infra
        click Dockerfile "https://github.com/dodoex/web3-rpc-proxy/tree/main/Dockerfile"
        Compose["docker-compose.yaml"]:::infra
        click Compose "https://github.com/dodoex/web3-rpc-proxy/blob/main/docker-compose.yaml"
        LB --> Dockerfile
    end

    classDef api fill:#ADD8E6,stroke:#000
    classDef service fill:#90EE90,stroke:#000
    classDef core fill:#98FB98,stroke:#000
    classDef infra fill:#F0E68C,stroke:#000
    classDef repo fill:#FFE4B5,stroke:#000
    classDef db fill:#FFDAB9,stroke:#000
    classDef storage fill:#FFA500,stroke:#000
    classDef external fill:#D3D3D3,stroke:#000
    classDef doc fill:#FFFFFF,stroke:#000
```
