How openbmp/obmp-collector works
Project Type and Purpose
• OpenBMP is a network‐monitoring collector daemon implementing the BGP Monitoring Protocol (BMP).
• It ingests BMP messages from routers, parses them (BGP, EVPN, link‐state, etc.), and produces both raw and parsed streams to Apache Kafka.
• Downstream consumer apps (MySQL, file, UI, forwarders) subscribe to Kafka topics for real‐time route analytics, storage, or forwarding.
File Structure
• Top-level “Server/” directory contains the collector daemon’s C++ source:
• Config, Logger, safeQueue, md5 utilities
• “bgp/” subfolder: parsers and message formats for Open, Update, Notification, EVPN, extended communities, BGP‐LS
• “bmp/” subfolder: BMP listener, reader, and parsing logic
• “kafka/” subfolder: Kafka producer implementation (callbacks, topic selection, partitioning)
• main entrypoint “openbmp.cpp” and threading logic “client_thread.cpp”
• “docs/” holds architecture guidance, API spec, install/build instructions, user guides, and a high‐level flow image.
• “scripts/rv-kafka-to-bmp.py” is a Python example of consuming Kafka and re‐emitting BMP.
README for Additional Insights
• The architecture revolves around three logical tiers:
A. BMP device inputs (routers sending BMP v1–v3)
B. OpenBMP collector/daemon (BMP listener → parser → Kafka producer)
C. Consumers (MySQL, file, forwarders, UI via Docker)
• Kafka is the central message bus: “collector no longer forwards directly to MySQL” but instead publishes to Kafka topics.
Main Components
a. BMP Devices (routers) – external BGP/BMP speakers
b. OpenBMP Daemon –
• BMP Listener & Parser modules (bmp, bgp subfolders)
• Core services: Config, Logger, Thread Management, safeQueue
• Kafka Producer layer (kafka subfolder)
c. Message Bus – Apache Kafka cluster
d. Consumers / Forwarders – MySQL Consumer, File Consumer, UI (DB_REST/API), OpenBMP Forwarder, MRT dumper, custom scripts
e. External Systems – Databases (MySQL, Cassandra, Elasticsearch), Monitoring UIs, Analytics Platforms
Relationships and Data Flow
• Within the daemon, show BMP read → BGP parsing → safeQueue → KafkaProducer
• Kafka publishes to multiple topics (raw BMP, parsed BGP, EVPN, link‐state)
• Consumers subscribe: MySQL Consumer → DB, File Consumer → disk, Forwarder → other BMP collectors, UI → REST/API → browser
Architectural Patterns & Design Principles
• Modular parsing pipeline: separation of concerns (bmp vs. bgp vs. kafka modules)
• Producer/Consumer pattern via thread pools and safeQueue
• Event-driven messaging bus (Kafka) decouples collectors from storage and analytics
• Configuration‐driven: templated openbmpd.conf, CMake for build
Technologies and Frameworks to Call Out
• C++11 (server code) built with CMake
• Apache Kafka (librdkafka consumer/producer via MsgBusImpl_kafka)
• Python script example for custom consumers/forwarders
• Dockerized UI and DB_REST (external to this repo)