# sriramko/arbitrageengine architecture

> Architecture diagram of the GitHub repository sriramko/arbitrageengine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sriramko/arbitrageengine
- Repository: https://github.com/sriramko/arbitrageengine

## Overview

This repository implements a multi-venue market-data pipeline: exchange feeds are ingested by venue producers, normalized into a shared event envelope, transported through Kafka (optionally via Zenoh), stored in Postgres, analyzed, and presented in a live dashboard. Its analysis branches cover rolling correlations and divergence, cross-market arbitrage, and ingest and gossip latency; a registry discovers and classifies markets for analysis. The README documents these system-level flows; source excerpts provide additional support for database writes, registry construction, and the latency sampler. Some producer and analyzer internals were not sampled, so those links follow documented architecture.

Main entry points: [`consumer/db/postgres.py`](https://github.com/sriramko/arbitrageengine/blob/main/consumer/db/postgres.py) (Postgres), [`coinbase-rust-client/src/main.rs`](https://github.com/sriramko/arbitrageengine/blob/main/coinbase-rust-client/src/main.rs) (Coinbase producer), [`polymarket-rust-client/src/main.rs`](https://github.com/sriramko/arbitrageengine/blob/main/polymarket-rust-client/src/main.rs) (Polymarket producer), [`kalshi-cpp-client/producer/main.cpp`](https://github.com/sriramko/arbitrageengine/blob/main/kalshi-cpp-client/producer/main.cpp) (Kalshi producer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Market ingestion"]
  node_coinbase_producer["Coinbase producer<br/>[main.rs]"]
  node_polymarket_producer["Polymarket producer<br/>[main.rs]"]
  node_kalshi_producer["Kalshi producer<br/>[main.cpp]"]
  node_event_envelope["Unified event schema<br/>[event.py]"]
end

subgraph group_transport["Event transport"]
  node_kafka["Kafka"]
  node_zenoh["Zenoh gossip"]
  node_zenoh_bridge["Zenoh bridge<br/>[main.rs]"]
  node_kafka_relay["Kafka relay<br/>[main.rs]"]
end

subgraph group_storage["Storage and registry"]
  node_unified_consumer["Unified consumer<br/>[unified.py]"]
  node_postgres[("Postgres<br/>[postgres.py]")]
  node_market_registry["Market registry<br/>[registry.py]"]
  node_market_classification["Market classification<br/>[classify.py]"]
end

subgraph group_analysis["Analysis"]
  node_correlation_engine["Correlation engine<br/>[engine.py]"]
  node_cross_arb_scanner["Cross-market arbitrage<br/>[cross_venue.py]"]
  node_latency_monitor["Latency sampler<br/>[sampler.rs]"]
  node_latency_publisher["Latency publisher<br/>[kafka_producer.rs]"]
  node_latency_consumer["Latency consumer"]
  node_gossip_latency["Gossip latency monitor<br/>[gossip_latency.py]"]
end

subgraph group_presentation["Monitoring"]
  node_dashboard["Flask dashboard<br/>[app.py]"]
end

node_exchange_feeds(("Exchange feeds"))
node_dashboard_user(("Dashboard user"))

node_exchange_feeds -->|"stream data"| node_coinbase_producer
node_exchange_feeds -->|"stream data"| node_polymarket_producer
node_exchange_feeds -->|"stream data"| node_kalshi_producer
node_coinbase_producer -->|"emit envelope"| node_event_envelope
node_polymarket_producer -->|"emit envelope"| node_event_envelope
node_kalshi_producer -->|"emit envelope"| node_event_envelope
node_coinbase_producer -->|"publish"| node_kafka
node_polymarket_producer -->|"publish"| node_kafka
node_kalshi_producer -->|"publish"| node_kafka
node_coinbase_producer -.->|"optional gossip"| node_zenoh
node_polymarket_producer -.->|"optional gossip"| node_zenoh
node_kalshi_producer -.->|"optional gossip"| node_zenoh
node_zenoh -.->|"bridge events"| node_zenoh_bridge
node_zenoh_bridge -.->|"republish"| node_kafka
node_kafka -.->|"relay events"| node_kafka_relay
node_kafka_relay -.->|"forward events"| node_kafka
node_kafka -->|"consume events"| node_unified_consumer
node_unified_consumer -->|"parse events"| node_event_envelope
node_unified_consumer -->|"write ticks"| node_postgres
node_market_registry -->|"classify markets"| node_market_classification
node_postgres -->|"read price series"| node_correlation_engine
node_market_registry -->|"select pairs"| node_correlation_engine
node_correlation_engine -->|"write signals"| node_postgres
node_market_registry -.->|"supply markets"| node_cross_arb_scanner
node_postgres -.->|"read market data"| node_cross_arb_scanner
node_cross_arb_scanner -.->|"write signals"| node_postgres
node_latency_monitor -->|"publish samples"| node_latency_publisher
node_latency_publisher -->|"publish metrics"| node_kafka
node_kafka -->|"consume metrics"| node_latency_consumer
node_latency_consumer -->|"write latency"| node_postgres
node_zenoh -.->|"measure gossip"| node_gossip_latency
node_gossip_latency -.->|"record latency"| node_postgres
node_postgres -->|"query ticks and signals"| node_dashboard
node_dashboard_user -->|"view dashboard"| node_dashboard

click node_coinbase_producer "https://github.com/sriramko/arbitrageengine/blob/main/coinbase-rust-client/src/main.rs"
click node_polymarket_producer "https://github.com/sriramko/arbitrageengine/blob/main/polymarket-rust-client/src/main.rs"
click node_kalshi_producer "https://github.com/sriramko/arbitrageengine/blob/main/kalshi-cpp-client/producer/main.cpp"
click node_event_envelope "https://github.com/sriramko/arbitrageengine/blob/main/consumer/event.py"
click node_zenoh_bridge "https://github.com/sriramko/arbitrageengine/blob/main/zenoh-bridge/src/main.rs"
click node_kafka_relay "https://github.com/sriramko/arbitrageengine/blob/main/relay/src/main.rs"
click node_unified_consumer "https://github.com/sriramko/arbitrageengine/blob/main/consumer/unified.py"
click node_postgres "https://github.com/sriramko/arbitrageengine/blob/main/consumer/db/postgres.py"
click node_market_registry "https://github.com/sriramko/arbitrageengine/blob/main/consumer/registry/registry.py"
click node_market_classification "https://github.com/sriramko/arbitrageengine/blob/main/consumer/registry/classify.py"
click node_correlation_engine "https://github.com/sriramko/arbitrageengine/blob/main/consumer/correlation/engine.py"
click node_cross_arb_scanner "https://github.com/sriramko/arbitrageengine/blob/main/consumer/cross_arb/cross_venue.py"
click node_latency_monitor "https://github.com/sriramko/arbitrageengine/blob/main/latency/src/sampler.rs"
click node_latency_publisher "https://github.com/sriramko/arbitrageengine/blob/main/latency/src/kafka_producer.rs"
click node_latency_consumer "https://github.com/sriramko/arbitrageengine/blob/main/consumer/latency_consumer.py"
click node_gossip_latency "https://github.com/sriramko/arbitrageengine/blob/main/scripts/gossip_latency.py"
click node_dashboard "https://github.com/sriramko/arbitrageengine/blob/main/frontend/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_coinbase_producer,node_polymarket_producer,node_kalshi_producer,node_event_envelope,node_dashboard_user toneBlue
class node_kafka,node_zenoh,node_zenoh_bridge,node_kafka_relay toneAmber
class node_unified_consumer,node_postgres,node_market_registry,node_market_classification toneMint
class node_correlation_engine,node_cross_arb_scanner,node_latency_monitor,node_latency_publisher,node_latency_consumer,node_gossip_latency toneRose
class node_dashboard,node_exchange_feeds toneIndigo
```

## Components

### Market ingestion

- [Coinbase producer](https://github.com/sriramko/arbitrageengine/blob/main/coinbase-rust-client/src/main.rs): `coinbase-rust-client/src/main.rs`
- [Polymarket producer](https://github.com/sriramko/arbitrageengine/blob/main/polymarket-rust-client/src/main.rs): `polymarket-rust-client/src/main.rs`
- [Kalshi producer](https://github.com/sriramko/arbitrageengine/blob/main/kalshi-cpp-client/producer/main.cpp): `kalshi-cpp-client/producer/main.cpp`
- [Unified event schema](https://github.com/sriramko/arbitrageengine/blob/main/consumer/event.py): `consumer/event.py`

### Event transport

- **Kafka**
- **Zenoh gossip**
- [Zenoh bridge](https://github.com/sriramko/arbitrageengine/blob/main/zenoh-bridge/src/main.rs): `zenoh-bridge/src/main.rs`
- [Kafka relay](https://github.com/sriramko/arbitrageengine/blob/main/relay/src/main.rs): `relay/src/main.rs`

### Storage and registry

- [Unified consumer](https://github.com/sriramko/arbitrageengine/blob/main/consumer/unified.py): `consumer/unified.py`
- [Postgres](https://github.com/sriramko/arbitrageengine/blob/main/consumer/db/postgres.py): `consumer/db/postgres.py`
- [Market registry](https://github.com/sriramko/arbitrageengine/blob/main/consumer/registry/registry.py): `consumer/registry/registry.py`
- [Market classification](https://github.com/sriramko/arbitrageengine/blob/main/consumer/registry/classify.py): `consumer/registry/classify.py`

### Analysis

- [Correlation engine](https://github.com/sriramko/arbitrageengine/blob/main/consumer/correlation/engine.py): `consumer/correlation/engine.py`
- [Cross-market arbitrage](https://github.com/sriramko/arbitrageengine/blob/main/consumer/cross_arb/cross_venue.py): `consumer/cross_arb/cross_venue.py`
- [Latency sampler](https://github.com/sriramko/arbitrageengine/blob/main/latency/src/sampler.rs): `latency/src/sampler.rs`
- [Latency publisher](https://github.com/sriramko/arbitrageengine/blob/main/latency/src/kafka_producer.rs): `latency/src/kafka_producer.rs`
- [Latency consumer](https://github.com/sriramko/arbitrageengine/blob/main/consumer/latency_consumer.py): `consumer/latency_consumer.py`
- [Gossip latency monitor](https://github.com/sriramko/arbitrageengine/blob/main/scripts/gossip_latency.py): `scripts/gossip_latency.py`

### Monitoring

- [Flask dashboard](https://github.com/sriramko/arbitrageengine/blob/main/frontend/app.py): `frontend/app.py`

### Other components

- **Exchange feeds**
- **Dashboard user**

## Connections

- Exchange feeds → Coinbase producer: stream data (evidence: `README.md`)
- Exchange feeds → Polymarket producer: stream data (evidence: `README.md`)
- Exchange feeds → Kalshi producer: stream data (evidence: `README.md`)
- Coinbase producer → Unified event schema: emit envelope (evidence: `README.md`)
- Polymarket producer → Unified event schema: emit envelope (evidence: `README.md`)
- Kalshi producer → Unified event schema: emit envelope (evidence: `README.md`)
- Coinbase producer → Kafka: publish (evidence: `README.md`)
- Polymarket producer → Kafka: publish (evidence: `README.md`)
- Kalshi producer → Kafka: publish (evidence: `README.md`)
- Coinbase producer → Zenoh gossip: optional gossip (evidence: `README.md`)
- Polymarket producer → Zenoh gossip: optional gossip (evidence: `README.md`)
- Kalshi producer → Zenoh gossip: optional gossip (evidence: `README.md`)
- Zenoh gossip → Zenoh bridge: bridge events (evidence: `README.md`)
- Zenoh bridge → Kafka: republish (evidence: `README.md`)
- Kafka → Kafka relay: relay events (evidence: `README.md`)
- Kafka relay → Kafka: forward events (evidence: `README.md`)
- Kafka → Unified consumer: consume events (evidence: `README.md`)
- Unified consumer → Unified event schema: parse events (evidence: `consumer/unified.py`)
- Unified consumer → Postgres: write ticks (evidence: `consumer/unified.py`)
- Market registry → Market classification: classify markets (evidence: `consumer/registry/registry.py`)
- Postgres → Correlation engine: read price series (evidence: `consumer/correlation/engine.py`)
- Market registry → Correlation engine: select pairs (evidence: `consumer/correlation/engine.py`)
- Correlation engine → Postgres: write signals (evidence: `consumer/correlation/engine.py`)
- Market registry → Cross-market arbitrage: supply markets (evidence: `README.md`)
- Postgres → Cross-market arbitrage: read market data (evidence: `README.md`)
- Cross-market arbitrage → Postgres: write signals (evidence: `README.md`)
- Latency sampler → Latency publisher: publish samples (evidence: `latency/src/app.rs`)
- Latency publisher → Kafka: publish metrics (evidence: `README.md`)
- Kafka → Latency consumer: consume metrics (evidence: `README.md`)
- Latency consumer → Postgres: write latency (evidence: `consumer/latency_consumer.py`)
- Zenoh gossip → Gossip latency monitor: measure gossip (evidence: `README.md`)
- Gossip latency monitor → Postgres: record latency (evidence: `README.md`)
- Postgres → Flask dashboard: query ticks and signals (evidence: `frontend/app.py`)
- Dashboard user → Flask dashboard: view dashboard (evidence: `README.md`)