How finos/timebase-ce works
Big Picture
• The project is an enterprise‐grade, open‑source time‑series data management and messaging system used in financial services.
• It functions both as a time‑series database (storing and querying historical data) and as a high‑performance real‑time data streaming/messaging server.
• Its architecture needs to support low‐latency data ingestion, high throughput data transmission, and real‑time querying, while also exposing APIs (including JDBC, REST, websockets) and integration modules (e.g., cloud/Docker support, S3 clients, SNMP agents, web monitoring tools).
High-Level Components
• TimeBase Server Core:
– This is the heart of the system and is subdivided into subcomponents such as:
• Data Ingestion Layer: accepts incoming time-series messages—via native protocols (IPC/UDP), REST endpoints, and possibly other messaging APIs.
• Storage Engine (“TickDB”): responsible for persistent data storage, including mechanisms for organizing “streams” of time-series data in a resilient and scalable way. It also supports historical queries.
• Query Processing Engine: supports both SQL-style and low‑latency, real‑time stream consumption.
• Messaging/Streaming Infrastructure: handles publication and subscription to “topics” or streams. This part can integrate high‑performance messaging channels (for example, using Aeron) and low‑latency APIs.
• API and Connector Modules:
– JDBC Module: for SQL access.
– Client Libraries and SDKs for various integrations.
– REST/WS interfaces for external integration.
• Web Layers and Monitoring:
– Web Monitor: the webmonitor and web modules include a UI for system status (e.g., connections, active sessions, server metrics, logs).
– SNMP/Admin modules: there’s an SNMP agent component to expose operational metrics.
• Build & Deployment Support:
– There are Gradle-based build modules and distribution scripts for Docker, Kubernetes, and cloud deployment.
Architectural Patterns and Relationships
• The system follows a modular, component‑oriented design. Each module (server, client, pub, jdbc, etc.) is built as an independent Gradle sub‑project.
• Data ingress (multiple protocols) and differentiate between the persistent storage (“TickDB”) and the real‑time stream distribution.
• That Messaging components and the Query engine can work concurrently, with data “streams” having user-defined schemas.
• Integration points (SNMP agents, REST endpoints, JDBC API, and web UIs) that interface with external applications.
– Data flows from external data providers into the ingestion layer.
– Ingestion flows into the core storage engine and is “published” via the messaging system.
– Queries and JDBC calls being routed to the Query Processing Engine.
– Administration / monitoring calls going to the SNMP agent, web UI, and REST layer.
Technologies and Frameworks
• Java (11+): the whole system is built in Java with Gradle as the build tool.
• Gradle builds and buildSrc for custom build plugins.
• Cloud/Container orchestration via Docker and Kubernetes (as per provided Docker-related files).
• Low-level networking libraries, e.g., Aeron for high‑performance messaging.
• Complementary languages: There are CPP utilities integrated with TimeBase for performance‑sensitive use cases.
• API endpoints expose the system via REST endpoints and websockets.