# ethpandaops/xatu architecture

> Architecture diagram of the GitHub repository ethpandaops/xatu, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ethpandaops/xatu
- Repository: https://github.com/ethpandaops/xatu

## Overview

##### Main Components
- External Ethereum Network Elements:
– Consensus P2P Network (the source of data, including ETHP2P connections)
– Execution P2P Network (from which Mimicry collects execution data)
- Xatu Clients:
– Sentry: Runs alongside an Ethereum consensus client to collect beacon chain data.
– Cannon: Also runs alongside an Ethereum consensus client, capturing canonical finalized data.
– Discovery: Uses discovery protocols (both v4 and v5) to find network nodes and even execution layer metadata.
– Mimicry: Connects to the execution P2P network to collect execution layer data.
– (Other example modes referenced in README like SAGE or Relay Monitor if applicable.)
- Centralized Xatu Server:
– Acts as the hub for collecting events from all the different clients.
– Provides interfaces (HTTP endpoints) for ingestion and may be responsible for coordinating data flows.
- Data Pipeline and Persistence:
– Persistence: A Clickhouse database (with configuration and migrations provided) stores the monitored data.
- Deployment & Observability Layers:
– Docker Compose configurations, Kubernetes/Helm charts (for containerized deployments).
– Other services such as Grafana (for dashboard and monitoring), Prometheus, Nginx (as reverse proxy) and possibly Tempo for tracing.

##### How Components Interact
- The Ethereum networks (consensus and execution) are the source systems.
- Each client (Sentry, Cannon, Discovery, Mimicry) connects to the appropriate Ethereum service (often via Beacon API) to collect information.
- All these clients send data to the Xatu Server, forming the central node.
- The Xatu Server then forwards data into the data pipeline, which feeds persistence (Clickhouse) and can be visualized through dashboards (Grafana) and monitored by alerts.
- There is also a feedback or control loop where, for example, deployment configuration (e.g., Dockerfiles, Docker Compose, helm charts) ensures all components work together.

##### Recognize Architectural Patterns and Principles
- Modular Client-Server Architecture: Each client mode is designed as an independent module (referenced in cmd and pkg directories) that connects to a centralized server.
- Microservices-like Separation: Although packaged in one repository, the different modes (Server, Sentry, Discovery, Mimicry, Cannon) are clearly separated with dedicated subdirectories and have their own configurations.
- Data Pipeline Design: The system clearly envisions a pipeline flow – from client data ingestion to persistence and visualization.
- Use of Containerization & CI/CD: File tree shows Dockerfiles, GitHub workflows, and deployment manifests indicating a strong containerized and automated deployment approach.

##### Technologies and Frameworks
- Programming Language: Go (as seen by the presence of .go files in cmd/ and pkg/ directories).
- Containerization: Docker/Docker Compose, along with support for Kubernetes via Helm.
- Databases and Observability: Clickhouse for persistence, Grafana dashboards for visualization, Prometheus for metrics, and possibly Nginx for routing.
- Protocols: Beacon API for consensus client interaction, Ethereum’s discovery protocols (discv4/discv5) for network discovery, and P2P protocols.
- Build/CI Tools: GitHub Actions workflows, Makefile, Buf for protobuf development, and goreleaser for releases.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Networks
    subgraph "External Ethereum Networks"
        Consensus["Consensus P2P Network"]:::external
        Execution["Execution P2P Network"]:::external
    end

    %% Xatu Clients
    subgraph "Xatu Clients"
        Sentry["Sentry Client"]:::client
        Cannon["Cannon Client"]:::client
        Discovery["Discovery Client"]:::client
        Mimicry["Mimicry Client"]:::client
        RelayMonitor["Relay Monitor"]:::client
    end

    %% Central Hub
    subgraph "Central Hub"
        XatuServer["Xatu Server"]:::server
    end

    %% Data Pipeline & Persistence
    subgraph "Data Pipeline & Persistence"
        DataPipeline["Data Pipeline & Persistence (Clickhouse)"]:::persistence
    end

    %% Observability & Dashboards
    subgraph "Observability & Dashboards"
        Grafana["Grafana"]:::observability
        Prometheus["Prometheus"]:::observability
        Tempo["Tempo"]:::observability
    end

    %% Deployment & CI/CD
    subgraph "Deployment & CI/CD"
        Deployment["Container Deployment & CI/CD"]:::deployment
    end

    %% Connections from External Networks to Clients
    Consensus -->|"feeds"| Sentry
    Consensus -->|"feeds"| Cannon
    Consensus -->|"feeds"| Discovery
    Consensus -->|"feeds"| RelayMonitor
    Execution -->|"feeds"| Mimicry

    %% Clients send data to Central Hub
    Sentry -->|"sends data"| XatuServer
    Cannon -->|"sends data"| XatuServer
    Discovery -->|"sends data"| XatuServer
    Mimicry -->|"sends data"| XatuServer
    RelayMonitor -->|"sends data"| XatuServer

    %% Central Hub to Data Pipeline
    XatuServer -->|"forwards data"| DataPipeline

    %% Data Pipeline to Observability components
    DataPipeline -->|"data for viz"| Grafana
    DataPipeline -->|"metrics"| Prometheus
    DataPipeline -->|"traces"| Tempo

    %% Deployment influences all major components (dotted relationships)
    Deployment -.-> Sentry
    Deployment -.-> Cannon
    Deployment -.-> Discovery
    Deployment -.-> Mimicry
    Deployment -.-> RelayMonitor
    Deployment -.-> XatuServer
    Deployment -.-> DataPipeline
    Deployment -.-> Grafana
    Deployment -.-> Prometheus
    Deployment -.-> Tempo

    %% Click Events for Xatu Clients
    click Sentry "https://github.com/ethpandaops/xatu/blob/master/cmd/sentry.go"
    click Sentry "https://github.com/ethpandaops/xatu/tree/master/pkg/sentry"
    click Cannon "https://github.com/ethpandaops/xatu/blob/master/cmd/cannon.go"
    click Cannon "https://github.com/ethpandaops/xatu/tree/master/pkg/cannon"
    click Discovery "https://github.com/ethpandaops/xatu/blob/master/cmd/discovery.go"
    click Discovery "https://github.com/ethpandaops/xatu/tree/master/pkg/discovery"
    click Mimicry "https://github.com/ethpandaops/xatu/blob/master/cmd/mimicry.go"
    click Mimicry "https://github.com/ethpandaops/xatu/blob/master/cmd/cl-mimicry.go"
    click Mimicry "https://github.com/ethpandaops/xatu/tree/master/pkg/mimicry"
    click Mimicry "https://github.com/ethpandaops/xatu/tree/master/pkg/clmimicry"
    click RelayMonitor "https://github.com/ethpandaops/xatu/blob/master/cmd/relay-monitor.go"
    click RelayMonitor "https://github.com/ethpandaops/xatu/tree/master/pkg/relaymonitor"

    %% Click Events for Central Hub
    click XatuServer "https://github.com/ethpandaops/xatu/blob/master/cmd/server.go"
    click XatuServer "https://github.com/ethpandaops/xatu/tree/master/pkg/server"

    %% Click Events for Data Pipeline & Persistence
    click DataPipeline "https://github.com/ethpandaops/xatu/blob/master/deploy/local/docker-compose/xatu-server.yaml"
    click DataPipeline "https://github.com/ethpandaops/xatu/tree/master/deploy/local/docker-compose/clickhouse"

    %% Click Events for Observability & Dashboards
    click Grafana "https://github.com/ethpandaops/xatu/tree/master/deploy/local/docker-compose/grafana"
    click Prometheus "https://github.com/ethpandaops/xatu/blob/master/deploy/local/docker-compose/prometheus.yaml"
    click Tempo "https://github.com/ethpandaops/xatu/blob/master/deploy/local/docker-compose/tempo.yaml"

    %% Click Events for Deployment & CI/CD
    click Deployment "https://github.com/ethpandaops/xatu/tree/master/Dockerfile"
    click Deployment "https://github.com/ethpandaops/xatu/tree/master/deploy"
    click Deployment "https://github.com/ethpandaops/xatu/tree/master/.github/workflows"

    %% Styles
    classDef external fill:#FFD580,stroke:#FF8C00,stroke-width:2px;
    classDef client fill:#C2E0FF,stroke:#1F77B4,stroke-width:2px;
    classDef server fill:#FFE6E6,stroke:#CC0000,stroke-width:2px;
    classDef persistence fill:#E2F0CB,stroke:#7FBF7F,stroke-width:2px;
    classDef observability fill:#E0BBE4,stroke:#9D4EDD,stroke-width:2px;
    classDef deployment fill:#FADADD,stroke:#FF1493,stroke-width:2px;
```
