# openbmp/obmp-collector architecture

> Architecture diagram of the GitHub repository openbmp/obmp-collector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openbmp/obmp-collector
- Repository: https://github.com/openbmp/obmp-collector

## Overview

##### Project Type and Purpose
- OpenBMP is a network‐monitoring collector daemon implementing the BGP Monitoring Protocol (BMP).
- It ingests BMP messages from routers, parses them (BGP, EVPN, link‐state, etc.), and produces both raw and parsed streams to Apache Kafka.
- Downstream consumer apps (MySQL, file, UI, forwarders) subscribe to Kafka topics for real‐time route analytics, storage, or forwarding.

##### File Structure
- Top-level “Server/” directory contains the collector daemon’s C++ source:
- Config, Logger, safeQueue, md5 utilities
- “bgp/” subfolder: parsers and message formats for Open, Update, Notification, EVPN, extended communities, BGP‐LS
- “bmp/” subfolder: BMP listener, reader, and parsing logic
- “kafka/” subfolder: Kafka producer implementation (callbacks, topic selection, partitioning)
- main entrypoint “openbmp.cpp” and threading logic “client_thread.cpp”
- “docs/” holds architecture guidance, API spec, install/build instructions, user guides, and a high‐level flow image.
- “scripts/rv-kafka-to-bmp.py” is a Python example of consuming Kafka and re‐emitting BMP.

##### README for Additional Insights
- The architecture revolves around three logical tiers:
A. BMP device inputs (routers sending BMP v1–v3)
B. OpenBMP collector/daemon (BMP listener → parser → Kafka producer)
C. Consumers (MySQL, file, forwarders, UI via Docker)
- Kafka is the central message bus: “collector no longer forwards directly to MySQL” but instead publishes to Kafka topics.

##### Main Components
a. BMP Devices (routers) – external BGP/BMP speakers
b. OpenBMP Daemon –
- BMP Listener & Parser modules (bmp, bgp subfolders)
- Core services: Config, Logger, Thread Management, safeQueue
- Kafka Producer layer (kafka subfolder)
c. Message Bus – Apache Kafka cluster
d. Consumers / Forwarders – MySQL Consumer, File Consumer, UI (DB_REST/API), OpenBMP Forwarder, MRT dumper, custom scripts
e. External Systems – Databases (MySQL, Cassandra, Elasticsearch), Monitoring UIs, Analytics Platforms

##### Relationships and Data Flow
- Within the daemon, show BMP read → BGP parsing → safeQueue → KafkaProducer
- Kafka publishes to multiple topics (raw BMP, parsed BGP, EVPN, link‐state)
- Consumers subscribe: MySQL Consumer → DB, File Consumer → disk, Forwarder → other BMP collectors, UI → REST/API → browser

##### Architectural Patterns & Design Principles
- Modular parsing pipeline: separation of concerns (bmp vs. bgp vs. kafka modules)
- Producer/Consumer pattern via thread pools and safeQueue
- Event-driven messaging bus (Kafka) decouples collectors from storage and analytics
- Configuration‐driven: templated openbmpd.conf, CMake for build

##### Technologies and Frameworks to Call Out
- C++11 (server code) built with CMake
- Apache Kafka (librdkafka consumer/producer via MsgBusImpl_kafka)
- Python script example for custom consumers/forwarders
- Dockerized UI and DB_REST (external to this repo)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Devices
    Routers["BMP Devices (Routers)"]:::external

    %% Daemon Layer
    subgraph "OpenBMP Daemon"
        direction TB
        Config["Configuration Loader"]:::core
        Logger["Logger Service"]:::core
        ThreadMgmt["Thread Management & Client Handling"]:::core
        SafeQueue["Thread-Safe Queue (safeQueue)"]:::core
        MD5["MD5 Utility"]:::core

        subgraph "BMP Modules"
            direction TB
            BMPListener["BMP Listener Module"]:::core
            BMPReader["BMP Reader Module"]:::core
            BMPParser["BMP Parsing Logic"]:::core
        end

        subgraph "BGP Core Parsers"
            direction TB
            OpenMsg["OpenMsg"]:::core
            UpdateMsg["UpdateMsg"]:::core
            NotificationMsg["NotificationMsg"]:::core
            ExtCommunity["ExtCommunity"]:::core
            AddPathData["AddPathDataContainer"]:::core
            ParseBGP["parseBGP"]:::core
            BGPCommon["bgp_common"]:::core
        end

        subgraph "BGP Extensions"
            direction TB
            EVPN["EVPN"]:::core
            MPLinkState["MPLinkState"]:::core
            MPLinkStateAttr["MPLinkStateAttr"]:::core
            MPReachAttr["MPReachAttr"]:::core
            MPUnReachAttr["MPUnReachAttr"]:::core
        end

        subgraph "Kafka Producer Layer"
            direction TB
            MsgBusInt["MsgBusInterface"]:::core
            KafkaImpl["MsgBusImpl_kafka"]:::core
            DeliveryCB["KafkaDeliveryReportCallback"]:::core
            EventCB["KafkaEventCallback"]:::core
            Partitioner["KafkaPeerPartitionerCallback"]:::core
            TopicSelector["KafkaTopicSelector"]:::core
        end

        Entrypoint["Daemon Entrypoint\n(openbmp.cpp)"]:::core
    end

    %% Message Bus
    KafkaCluster["Apache Kafka Cluster"]:::kafka

    %% Consumers
    subgraph "Consumers / Forwarders"
        direction TB
        MySQLConsumer["MySQL Consumer"]:::optional
        FileConsumer["File Consumer"]:::optional
        Forwarder["OpenBMP Forwarder"]:::optional
        MRTDumper["MRT Dumper"]:::optional
        UI["UI (REST/API)"]:::optional
        CustomScript["Python Kafka→BMP Script"]:::optional
    end

    %% Data Flow
    Routers -->|"send BMP v1–v3"| BMPListener
    BMPListener --> BMPReader
    BMPReader --> BMPParser
    BMPParser --> OpenMsg
    BMPParser --> UpdateMsg
    BMPParser --> NotificationMsg
    BMPParser --> ExtCommunity
    BMPParser --> AddPathData
    BMPParser --> ParseBGP
    BMPParser --> BGPCommon
    OpenMsg --> EVPN
    UpdateMsg --> EVPN
    EVPN --> MPLinkState
    MPLinkState --> MPLinkStateAttr
    MPLinkStateAttr --> MPReachAttr
    MPReachAttr --> MPUnReachAttr
    ParseBGP --> SafeQueue
    SafeQueue --> MsgBusInt
    MsgBusInt --> KafkaImpl
    KafkaImpl --> DeliveryCB
    KafkaImpl --> EventCB
    KafkaImpl --> Partitioner
    KafkaImpl --> TopicSelector
    TopicSelector --> KafkaCluster

    KafkaCluster --> MySQLConsumer
    KafkaCluster --> FileConsumer
    KafkaCluster --> Forwarder
    KafkaCluster --> MRTDumper
    KafkaCluster --> UI
    KafkaCluster --> CustomScript

    %% Click Events
    click Config "https://github.com/openbmp/obmp-collector/blob/main/Server/src/Config.cpp"
    click Config "https://github.com/openbmp/obmp-collector/blob/main/Server/src/Config.h"
    click Logger "https://github.com/openbmp/obmp-collector/blob/main/Server/src/Logger.cpp"
    click Logger "https://github.com/openbmp/obmp-collector/blob/main/Server/src/Logger.h"
    click ThreadMgmt "https://github.com/openbmp/obmp-collector/blob/main/Server/src/client_thread.cpp"
    click ThreadMgmt "https://github.com/openbmp/obmp-collector/blob/main/Server/src/client_thread.h"
    click SafeQueue "https://github.com/openbmp/obmp-collector/blob/main/Server/src/safeQueue.hpp"
    click MD5 "https://github.com/openbmp/obmp-collector/blob/main/Server/src/md5.cpp"
    click MD5 "https://github.com/openbmp/obmp-collector/blob/main/Server/src/md5.h"
    click BMPListener "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bmp/BMPListener.cpp"
    click BMPListener "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bmp/BMPListener.h"
    click BMPReader "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bmp/BMPReader.cpp"
    click BMPReader "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bmp/BMPReader.h"
    click BMPParser "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bmp/parseBMP.cpp"
    click BMPParser "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bmp/parseBMP.h"
    click OpenMsg "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/OpenMsg.cpp"
    click OpenMsg "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/OpenMsg.h"
    click UpdateMsg "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/UpdateMsg.cpp"
    click UpdateMsg "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/UpdateMsg.h"
    click NotificationMsg "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/NotificationMsg.cpp"
    click NotificationMsg "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/NotificationMsg.h"
    click ExtCommunity "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/ExtCommunity.cpp"
    click ExtCommunity "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/ExtCommunity.h"
    click AddPathData "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/AddPathDataContainer.cpp"
    click AddPathData "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/AddPathDataContainer.h"
    click ParseBGP "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/parseBGP.cpp"
    click ParseBGP "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/parseBGP.h"
    click BGPCommon "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/bgp_common.h"
    click EVPN "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/EVPN.cpp"
    click EVPN "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/EVPN.h"
    click MPLinkState "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/linkstate/MPLinkState.cpp"
    click MPLinkState "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/linkstate/MPLinkState.h"
    click MPLinkStateAttr "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/linkstate/MPLinkStateAttr.cpp"
    click MPLinkStateAttr "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/linkstate/MPLinkStateAttr.h"
    click MPReachAttr "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/MPReachAttr.cpp"
    click MPReachAttr "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/MPReachAttr.h"
    click MPUnReachAttr "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/MPUnReachAttr.cpp"
    click MPUnReachAttr "https://github.com/openbmp/obmp-collector/blob/main/Server/src/bgp/MPUnReachAttr.h"
    click MsgBusInt "https://github.com/openbmp/obmp-collector/blob/main/Server/src/MsgBusInterface.hpp"
    click KafkaImpl "https://github.com/openbmp/obmp-collector/blob/main/Server/src/kafka/MsgBusImpl_kafka.cpp"
    click KafkaImpl "https://github.com/openbmp/obmp-collector/blob/main/Server/src/kafka/MsgBusImpl_kafka.h"
    click DeliveryCB "https://github.com/openbmp/obmp-collector/blob/main/Server/src/kafka/KafkaDeliveryReportCallback.cpp"
    click DeliveryCB "https://github.com/openbmp/obmp-collector/blob/main/Server/src/kafka/KafkaDeliveryReportCallback.h"
    click EventCB "https://github.com/openbmp/obmp-collector/blob/main/Server/src/kafka/KafkaEventCallback.cpp"
    click EventCB "https://github.com/openbmp/obmp-collector/blob/main/Server/src/kafka/KafkaEventCallback.h"
    click Partitioner "https://github.com/openbmp/obmp-collector/blob/main/Server/src/kafka/KafkaPeerPartitionerCallback.cpp"
    click Partitioner "https://github.com/openbmp/obmp-collector/blob/main/Server/src/kafka/KafkaPeerPartitionerCallback.h"
    click TopicSelector "https://github.com/openbmp/obmp-collector/blob/main/Server/src/kafka/KafkaTopicSelector.cpp"
    click TopicSelector "https://github.com/openbmp/obmp-collector/blob/main/Server/src/kafka/KafkaTopicSelector.h"
    click Entrypoint "https://github.com/openbmp/obmp-collector/blob/main/Server/src/openbmp.cpp"
    click CustomScript "https://github.com/openbmp/obmp-collector/blob/main/scripts/rv-kafka-to-bmp.py"

    %% Styles
    classDef external fill:#f9d7b9,stroke:#e27d60,color:#000
    classDef core fill:#a8d0e6,stroke:#374785,color:#000
    classDef kafka fill:#c8e6c9,stroke:#2e7d32,color:#000
    classDef optional fill:#f1f1f1,stroke:#b0b0b0,color:#666,stroke-dasharray: 5 5
```
