How embarktrucks/embag works
Project Type and Purpose
• Project Type: A dual API library for reading ROS bag files – it’s a native C++ library (with a CLI demo in embag_echo) and a Python API (packaged via pip and also built as a Bazel target).
• Purpose: To quickly read ROS bag files (even without loading full message definitions) with good performance in both C++ and Python. It also includes components for decompression (e.g., lz4, bz2) and supports cross-platform building using Bazel.
File Structure
• Top-Level Build & Configuration: Files such as .bazelproject, WORKSPACE, .bazelversion, and BUILD files in various directories indicate Bazel is used as the primary build system.
• Core Library Code: The lib directory holds the main C++ implementation (embag.cc, message parsing logic, view classes, etc.) and associated headers. This is the “heart” of the reader.
• Auxiliary Tools: The embag_echo directory contains a demo binary using the core library to demonstrate functionality.
• Python API: There are two directories (pip_package and python) which contain code for the Python interface, setup scripts, and tests.
• Compression Components: Directories such as bz2 and lz4 indicate modules for handling possibly compressed data.
• Testing: The test directory includes both C++ tests (.cc files) and Python tests, along with sample bag files.
• Continuous Integration and Deployment configurations in .github and Dockerfile confirm automated build and deployment pipelines.
README for Additional Insights
• The README explains usage scenarios for both the C++ and Python APIs.
• It shows how to build different targets (e.g., embag_echo demo, shared libraries, Debian packages, Python wheels) using Bazel.
• Performance benchmarks and usage examples are provided, highlighting performance differences and integration points.
a. Main Components to Include:
• User Interfaces:
• Command-line interface (embag_echo demo) for C++ usage.
• Python API (pip_package and python modules) accessible via pip installation.
• Core Library:
• C++ core library (in lib) which implements parsing logic, message handling, and decompression.
• Compression Modules:
• Distinct modules for handling compressed data: bz2 and lz4.
• Build and Test Infrastructure:
• Bazel build system orchestrates building of C++ binaries, shared objects, Debian packages, wheel packages and running tests.
• Testing module (test) that validates both C++ and Python interfaces using sample ROS bag files.
• External Integration:
• Integration with ROS bag format and compatibility with ROS message structures.
• Deployment pipelines via Dockerfile and GitHub Actions for CI/CD.
b. Relationships and Interactions:
• The core library (lib) is central to Embag – both the embag_echo demo and the Python wrappers use it.
• The Python API wraps around the C++ library (possibly using extension modules or bindings) and is packaged separately.
• Compression components (bz2 and lz4) interact with the core library to decompress bag file content.
• The Bazel build system orchestrates the build process for all these components, ensuring that C++ binaries, shared libraries, and Python wheels are produced in a consistent manner.
• Test suite validates interactions between components by reading bag files and verifying message formats.
c. Important Architectural Patterns and Design Principles:
• Separation of Concerns: The design isolates core parsing logic (in lib) from client-specific interfaces (CLI demo, Python wrappers) and compression logic.
• Modularization: Separate directories for C++ implementation, Python API layers, demo binary, and tests promote maintainability and scalability.
• Build Automation and Consistency: Using Bazel for all build targets ensures reproducibility and cross-language integration.
• Performance Focus: Emphasis on efficient message processing and minimal dependencies indicates a strong focus on performance and low overhead.
d. Technologies, Frameworks, and Libraries:
• Bazel: Used as the build system (files like WORKSPACE, BUILD, .bazelproject).
• C++: The core functionality is implemented in C++ (lib, embag_echo).
• Python: Python API and packaging (pip_package, python directories).
• Compression Libraries: Support for lz4 (and bz2) indicates handling compressed data, either via custom or external libraries.
• CI/CD Tools: GitHub Actions workflows (in .github directory) and Docker for containerized builds hint at continuous integration best practices.