# farcasterxyz/hub-monorepo architecture

> Architecture diagram of the GitHub repository farcasterxyz/hub-monorepo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/farcasterxyz/hub-monorepo
- Repository: https://github.com/farcasterxyz/hub-monorepo

## Overview

##### Main Components

a. Core Applications:
- Hubble Hub (Primary Application)
- Shuttle (Event Streaming Service)
- Client Libraries (Hub-NodeJS and Hub-Web)

b. External Services:
- Ethereum Network (for on-chain events)
- RocksDB (for storage)
- PostgreSQL (for Shuttle)
- Redis (for Shuttle)

##### Key Architectural Layers

a. Network Layer:
- P2P Communication (src/network/p2p)
- Sync Engine (src/network/sync)
- RPC Services (src/rpc)

b. Storage Layer:
- RocksDB Integration (src/storage/db)
- Store Management (src/storage/stores)
- Message Processing (src/storage/engine)

c. Protocol Layer:
- Protocol Buffers (protobufs/schemas)
- Message Validation
- Crypto Operations (src/crypto)

**[External Clients] <---> [Client Libraries (hub-nodejs/hub-web)]**

- v
**[Ethereum Network] <---> [Hubble Hub]**

- v
**[P2P Network] <----------> [Network Layer]**

- v
- [Protocol Layer]

- v
- [Storage Layer]

- [RocksDB/Redis]

- v
- [Shuttle Event Streaming]

- v
- [PostgreSQL]

##### Key Interactions to Highlight

a. Message Flow:
- Client message submission
- P2P message propagation
- Message validation and storage
- Event streaming to Shuttle

b. Synchronization:
- P2P network sync
- Merkle trie synchronization
- Event reconciliation

c. Data Storage:
- Message persistence
- State management
- Event logging

##### Additional Elements

a. Security Components:
- Ed25519 and EIP712 signers
- Verification systems
- Rate limiting

b. Monitoring:
- Grafana integration
- StatsD metrics
- Health checks

##### Implementation Details

- The modular architecture with clear separation of concerns
- The hybrid implementation using TypeScript and Rust
- The event-driven nature of the system
- The scalability and reliability features

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Clients
        ExtClients[External Clients]:::client
        HubNodeJS["Hub NodeJS Client"]:::client
        HubWeb["Hub Web Client"]:::client
    end

    subgraph Core["Core Hub"]
        HubbleHub["Hubble Hub"]:::core
        P2PNetwork["P2P Network"]:::network
        
        subgraph NetworkLayer["Network Layer"]
            P2PComm["P2P Communication"]:::network
            SyncEngine["Sync Engine"]:::network
            RPCServices["RPC Services"]:::network
        end
        
        subgraph ProtocolLayer["Protocol Layer"]
            Protobufs["Protocol Buffers"]:::protocol
            MsgValidation["Message Validation"]:::protocol
            CryptoOps["Crypto Operations"]:::protocol
        end
        
        subgraph StorageLayer["Storage Layer"]
            RocksDBInt["RocksDB Integration"]:::storage
            StoreMgmt["Store Management"]:::storage
            MsgProcess["Message Processing"]:::storage
        end
        
        subgraph Security["Security Components"]
            Ed25519["Ed25519 Signer"]:::security
            EIP712["EIP712 Signer"]:::security
            RateLimiting["Rate Limiting"]:::security
        end
        
        subgraph Monitoring["Monitoring"]
            Grafana["Grafana"]:::monitoring
            StatsD["StatsD Metrics"]:::monitoring
        end
    end

    subgraph External["External Services"]
        Ethereum["Ethereum Network"]:::external
        RocksDB[(RocksDB)]:::database
        Redis[(Redis)]:::database
        PostgreSQL[(PostgreSQL)]:::database
    end

    subgraph EventStreaming["Event Streaming"]
        Shuttle["Shuttle"]:::streaming
    end

    %% Connections
    ExtClients --> HubNodeJS & HubWeb
    HubNodeJS & HubWeb --> HubbleHub
    
    Ethereum <--> HubbleHub
    P2PNetwork <--> P2PComm
    
    NetworkLayer --> ProtocolLayer
    ProtocolLayer --> StorageLayer
    StorageLayer --> RocksDB
    
    HubbleHub --> Shuttle
    Shuttle --> PostgreSQL
    Shuttle --> Redis

    %% Click Events
    click HubbleHub "https://github.com/farcasterxyz/hub-monorepo/tree/main/apps/hubble"
    click Shuttle "https://github.com/farcasterxyz/hub-monorepo/tree/main/packages/shuttle"
    click HubNodeJS "https://github.com/farcasterxyz/hub-monorepo/tree/main/packages/hub-nodejs"
    click HubWeb "https://github.com/farcasterxyz/hub-monorepo/tree/main/packages/hub-web"
    click P2PComm "https://github.com/farcasterxyz/hub-monorepo/tree/main/apps/hubble/src/network/p2p"
    click SyncEngine "https://github.com/farcasterxyz/hub-monorepo/tree/main/apps/hubble/src/network/sync"
    click RPCServices "https://github.com/farcasterxyz/hub-monorepo/tree/main/apps/hubble/src/rpc"
    click RocksDBInt "https://github.com/farcasterxyz/hub-monorepo/tree/main/apps/hubble/src/storage/db"
    click StoreMgmt "https://github.com/farcasterxyz/hub-monorepo/tree/main/apps/hubble/src/storage/stores"
    click MsgProcess "https://github.com/farcasterxyz/hub-monorepo/tree/main/apps/hubble/src/storage/engine"
    click Protobufs "https://github.com/farcasterxyz/hub-monorepo/tree/main/protobufs/schemas"
    click MsgValidation "https://github.com/farcasterxyz/hub-monorepo/blob/main/apps/hubble/src/storage/engine/validation.worker.ts"
    click CryptoOps "https://github.com/farcasterxyz/hub-monorepo/tree/main/packages/core/src/crypto"
    click Ethereum "https://github.com/farcasterxyz/hub-monorepo/tree/main/apps/hubble/src/eth"
    click Ed25519 "https://github.com/farcasterxyz/hub-monorepo/blob/main/packages/core/src/signers/nobleEd25519Signer.ts"
    click EIP712 "https://github.com/farcasterxyz/hub-monorepo/blob/main/packages/core/src/signers/eip712Signer.ts"
    click RateLimiting "https://github.com/farcasterxyz/hub-monorepo/blob/main/apps/hubble/src/utils/rateLimits.ts"
    click Grafana "https://github.com/farcasterxyz/hub-monorepo/tree/main/apps/hubble/grafana"
    click StatsD "https://github.com/farcasterxyz/hub-monorepo/blob/main/apps/hubble/src/utils/statsd.ts"

    %% Styles
    classDef client fill:#A5D6A7,stroke:#000
    classDef core fill:#90CAF9,stroke:#000
    classDef network fill:#81D4FA,stroke:#000
    classDef protocol fill:#B39DDB,stroke:#000
    classDef storage fill:#FFE082,stroke:#000
    classDef security fill:#EF9A9A,stroke:#000
    classDef monitoring fill:#CE93D8,stroke:#000
    classDef external fill:#B0BEC5,stroke:#000
    classDef database fill:#FFCC80,stroke:#000
    classDef streaming fill:#80CBC4,stroke:#000
```
