How ethpandaops/xatu works
Main Components
• External Ethereum Network Elements:
– Consensus P2P Network (the source of data, including ETHP2P connections)
– Execution P2P Network (from which Mimicry collects execution data)
• Xatu Clients:
– Sentry: Runs alongside an Ethereum consensus client to collect beacon chain data.
– Cannon: Also runs alongside an Ethereum consensus client, capturing canonical finalized data.
– Discovery: Uses discovery protocols (both v4 and v5) to find network nodes and even execution layer metadata.
– Mimicry: Connects to the execution P2P network to collect execution layer data.
– (Other example modes referenced in README like SAGE or Relay Monitor if applicable.)
• Centralized Xatu Server:
– Acts as the hub for collecting events from all the different clients.
– Provides interfaces (HTTP endpoints) for ingestion and may be responsible for coordinating data flows.
• Data Pipeline and Persistence:
– Persistence: A Clickhouse database (with configuration and migrations provided) stores the monitored data.
• Deployment & Observability Layers:
– Docker Compose configurations, Kubernetes/Helm charts (for containerized deployments).
– Other services such as Grafana (for dashboard and monitoring), Prometheus, Nginx (as reverse proxy) and possibly Tempo for tracing.
How Components Interact
• The Ethereum networks (consensus and execution) are the source systems.
• Each client (Sentry, Cannon, Discovery, Mimicry) connects to the appropriate Ethereum service (often via Beacon API) to collect information.
• All these clients send data to the Xatu Server, forming the central node.
• The Xatu Server then forwards data into the data pipeline, which feeds persistence (Clickhouse) and can be visualized through dashboards (Grafana) and monitored by alerts.
• There is also a feedback or control loop where, for example, deployment configuration (e.g., Dockerfiles, Docker Compose, helm charts) ensures all components work together.
Recognize Architectural Patterns and Principles
• Modular Client-Server Architecture: Each client mode is designed as an independent module (referenced in cmd and pkg directories) that connects to a centralized server.
• Microservices-like Separation: Although packaged in one repository, the different modes (Server, Sentry, Discovery, Mimicry, Cannon) are clearly separated with dedicated subdirectories and have their own configurations.
• Data Pipeline Design: The system clearly envisions a pipeline flow – from client data ingestion to persistence and visualization.
• Use of Containerization & CI/CD: File tree shows Dockerfiles, GitHub workflows, and deployment manifests indicating a strong containerized and automated deployment approach.
Technologies and Frameworks
• Programming Language: Go (as seen by the presence of .go files in cmd/ and pkg/ directories).
• Containerization: Docker/Docker Compose, along with support for Kubernetes via Helm.
• Databases and Observability: Clickhouse for persistence, Grafana dashboards for visualization, Prometheus for metrics, and possibly Nginx for routing.
• Protocols: Beacon API for consensus client interaction, Ethereum’s discovery protocols (discv4/discv5) for network discovery, and P2P protocols.
• Build/CI Tools: GitHub Actions workflows, Makefile, Buf for protobuf development, and goreleaser for releases.