How openbmp/obmp-psql works
Project Type and Purpose
• This is an open-source backend consumer tool written in Java that reads parsed BMP/BGP messages from a Kafka message bus and writes them into a PostgreSQL database for real-time and historical reporting.
• Ancillary cron jobs and scripts supplement the main consumer by pulling additional data (WHOIS, PeeringDB, RPKI) and populating or augmenting the Postgres tables.
File‐Structure Analysis
• src/main/java: Java consumer app (org.openbmp.*)
• ConsumerApp, ConsumerRunnable, ConsumerRebalanceListener, PSQLHandler, WriterRunnable, plus SQL‐statement classes under psqlquery.
• Configuration under src/main/resources (obmp-psql.yml, log4j).
• cron_scripts: Python scripts invoked on schedule for WHOIS, PeeringDB, RPKI data.
• scripts: Helper scripts for bulk CSV import and geo data.
• database: Migration SQL files defining tables (base, aggregations), functions, views, triggers, and schema-version.
• docs: Build and run instructions.
• pom.xml: Maven build.
• test/src: Unit tests for core consumer.
Key README Insights
• The consumer is multi‐threaded; each thread owns a separate Postgres connection.
• Threads auto‐scale up/down based on queue depth; rebalancing occurs periodically.
• Fault tolerance is via Kafka consumer group rebalancing across hosts.
• Supports specific address families (IPv4, IPv6, link-state).
a. External Systems
– Kafka Cluster (topic partitions for parsed BMP messages)
– RPKI Validator Service (cron job)
– PeeringDB API (cron job)
– WHOIS sources (cron job)
– External Geo-IP CSV
b. Main Application
– ConsumerApp (bootstraps threads)
– Partitioned ConsumerRunnable instances (one per thread)
– Internal queue manager / load‐balancer
– WriterRunnable pool (per message type)
– PSQLHandler (executes SQL via JDBC)
– psqlquery classes (SQL generation)
c. Data Store
– PostgreSQL Instance
• Tables (peers, prefixes, link-state data)
• Views, Aggregations, Functions, Triggers
• schema-version tracking
d. Auxiliary Scripts
– scripts/db-ip-import.sh → bulk load IP blocks
– scripts/geo-csv-to-psql.py → load GeoIP data
e. Build & Deployment
– Maven (pom.xml)
– Docker/Kubernetes (optional, not in tree but can be added)
Relationships & Data Flows
• Scripts invoked manually/CI for initial/bulk loads.
Architectural Patterns & Principles
• Producer-Consumer with thread pool and queue‐based load balancing.
• Auto-scaling thread pools based on load thresholds.
• Clean separation of concerns: Kafka consumption, message processing, SQL execution.
• SQL encapsulation in dedicated query classes (psqlquery package).
• Schema‐versioned migrations for database.
Essential Elements to Capture
• Component boundaries and their configuration sources.
• Inter‐component queues (in-memory Java queues) and scaling logic.
• SQL generation and data persistence pipeline.
• External data enrichment pipelines running on cron.
• Build/test workflow (Maven → unit tests → packaging).