How dodoex/web3-rpc-proxy works
Project Type and Purpose
• This is a backend middleware service (“Web3 RPC Proxy”) written in Go.
• It exposes a JSON‑RPC HTTP API (using fasthttp) to upstream clients, implements multi‑tenant rate limiting, caching, retries and endpoint selection, and forwards requests to one or more EVM blockchain nodes.
• It supports dynamic configuration via etcd, persistence via PostgreSQL, distributed rate limits via Redis, and optional event publication via AMQP.
File Structure
• cmd/main.go: application entrypoint and server bootstrap.
• config/: YAML and JSON config templates (default.yaml, ethereum‑openrpc.json, grafana.json).
• internal/app/: “Agent” MVC‑style module holding controllers (HTTP handlers), services (business logic), repositories (DB access), plus shared modules (config loader, etcd, logger, Redis, AMQP).
• internal/core/: core RPC and endpoint‑selection engine, request context, JSON‑RPC validation.
• internal/app/database/: DB schema, migrations, seeding (tenant table).
• providers/web3‑rpc‑provider: external package for discovering and wrapping free RPC endpoints.
• utils/: shared helper functions (config parsing, compression, sharding, constants, metrics).
• docker-compose.yaml / Dockerfile: containerization and multi‑service orchestration.
README for Additional Insights
• The README confirms a “cluster‑deployable EVM RPC proxy” pattern.
• It describes dependencies (PostgreSQL, Redis, etcd, AMQP) and features (caching, rate limits, retries, JSON‑RPC schema validation, Prometheus metrics, Grafana dashboards).
Main Components & Their Roles
a. API Layer (fasthttp)
• Accepts inbound JSON‑RPC batches at “/<CHAIN>”.
• Parses headers (x_api_key, x_api_bucket, cache flags).
b. Controller (“agent_controller.go”)
• Translates HTTP request into internal ReqCtx.
• Applies parameter validation.
• Calls the AgentService.
c. Service Layer (agent_service.go)
• Enforces tenant isolation, rate limiting (via Redis), caching (via Redis scripts), retries (rotation or same), timeout handling.
• Publishes events to AMQP if configured.
• Delegates the actual JSON‑RPC call to the core endpoint client.
d. Core Endpoint Engine (internal/core/endpoint)
• Maintains a pool of HTTP/WebSocket clients to blockchain nodes.
• Implements endpoint scoring, selection strategies, failover.
e. Persistence & Config Stores
• PostgreSQL (tenant metadata via repository).
• etcd (dynamic config for chains and endpoints).
• redis (distributed rate‑limiting counters, cache).
f. Optional MQ (AMQP)
• Publish each request/response record for auditing or analytics.
g. External Blockchains
• Multiple EVM JSON‑RPC endpoints (HTTP or WSS).
Relationships & Data Flows
1. Client → HTTP server (fasthttp) → AgentController
2. Controller → AgentService
3. AgentService ←→ Redis (rate limit & cache)
4. AgentService → PostgreSQL via TenantRepository (tenant metadata)
5. AgentService → CoreEndpointClient → External EVM RPC nodes
6. AgentService → etcd (read config)
7. AgentService → AMQP (publish events)
8. Prometheus scrape metrics from in‑process registry
Architectural Patterns & Principles
• Layered / Clean Architecture: clear separation between HTTP adapter (controllers), business logic (services), infrastructure (repositories, shared modules).
• Hexagonal touches in core endpoint module: pluggable HTTP/WebSocket transports.
• Configuration‑as‑code in etcd via shared config loader.
• Dependency Injection via module wiring in internal/app/bootstrap.go.
Key Technologies & Frameworks
• Go 1.21, Fasthttp, Viper (for config), fx or a simple DI pattern in bootstrap.
• PostgreSQL (tenant table), Redis (cache & counters), etcd (config), RabbitMQ/AMQP.
• Prometheus client for metrics, Grafana dashboards.
• Docker & Docker‑Compose for containerization.
Final Checklist
• All major modules labeled clearly.
• Notes on protocols (HTTP, JSON‑RPC, AMQP, gRPC etc. if any).
• Deployment context: one node of cluster behind a load balancer; scalable Redis, etcd and PostgreSQL clusters; Grafana dashboards.
• Call out Docker Compose / Dockerfile for container orchestration.