How sriramko/arbitrageengine works
This repository implements a multi-venue market-data pipeline: exchange feeds are ingested by venue producers, normalized into a shared event envelope, transported through Kafka (optionally via Zenoh), stored in Postgres, analyzed, and presented in a live dashboard. Its analysis branches cover rolling correlations and divergence, cross-market arbitrage, and ingest and gossip latency; a registry discovers and classifies markets for analysis. The README documents these system-level flows; source excerpts provide additional support for database writes, registry construction, and the latency sampler. Some producer and analyzer internals were not sampled, so those links follow documented architecture.
Main entry points: postgres.py Postgres · src/main.rs Coinbase producer · src/main.rs Polymarket producer · main.cpp Kalshi producer
Components 21
Market ingestion
- Coinbase producer
coinbase-rust-client/src/main.rs - Polymarket producer
polymarket-rust-client/src/main.rs - Kalshi producer
kalshi-cpp-client/producer/main.cpp - Unified event schema
consumer/event.py
Event transport
- Kafka
- Zenoh gossip
- Zenoh bridge
zenoh-bridge/src/main.rs - Kafka relay
relay/src/main.rs
Storage and registry
- Unified consumer
consumer/unified.py - Postgres
consumer/db/postgres.py - Market registry
consumer/registry/registry.py - Market classification
consumer/registry/classify.py
Analysis
- Correlation engine
consumer/correlation/engine.py - Cross-market arbitrage
consumer/cross_arb/cross_venue.py - Latency sampler
latency/src/sampler.rs - Latency publisher
latency/src/kafka_producer.rs - Latency consumer
consumer/latency_consumer.py - Gossip latency monitor
scripts/gossip_latency.py
Monitoring
- Flask dashboard
frontend/app.py
Other components
- Exchange feeds
- Dashboard user
Connections 34
- Exchange feeds → Coinbase producer: stream data
README.md - Exchange feeds → Polymarket producer: stream data
README.md - Exchange feeds → Kalshi producer: stream data
README.md - Coinbase producer → Unified event schema: emit envelope
README.md - Polymarket producer → Unified event schema: emit envelope
README.md - Kalshi producer → Unified event schema: emit envelope
README.md - Coinbase producer → Kafka: publish
README.md - Polymarket producer → Kafka: publish
README.md - Kalshi producer → Kafka: publish
README.md - Coinbase producer → Zenoh gossip: optional gossip
README.md - Polymarket producer → Zenoh gossip: optional gossip
README.md - Kalshi producer → Zenoh gossip: optional gossip
README.md - Zenoh gossip → Zenoh bridge: bridge events
README.md - Zenoh bridge → Kafka: republish
README.md - Kafka → Kafka relay: relay events
README.md - Kafka relay → Kafka: forward events
README.md - Kafka → Unified consumer: consume events
README.md - Unified consumer → Unified event schema: parse events
consumer/unified.py - Unified consumer → Postgres: write ticks
consumer/unified.py - Market registry → Market classification: classify markets
consumer/registry/registry.py - Postgres → Correlation engine: read price series
consumer/correlation/engine.py - Market registry → Correlation engine: select pairs
consumer/correlation/engine.py - Correlation engine → Postgres: write signals
consumer/correlation/engine.py - Market registry → Cross-market arbitrage: supply markets
README.md - Postgres → Cross-market arbitrage: read market data
README.md - Cross-market arbitrage → Postgres: write signals
README.md - Latency sampler → Latency publisher: publish samples
latency/src/app.rs - Latency publisher → Kafka: publish metrics
README.md - Kafka → Latency consumer: consume metrics
README.md - Latency consumer → Postgres: write latency
consumer/latency_consumer.py - Zenoh gossip → Gossip latency monitor: measure gossip
README.md - Gossip latency monitor → Postgres: record latency
README.md - Postgres → Flask dashboard: query ticks and signals
frontend/app.py - Dashboard user → Flask dashboard: view dashboard
README.md