How ethz-asl/kalibr works
1. Repository shape
- This is a ROS1 catkin workspace, not a single application.
- The root contains several library stacks and application/tooling packages that are built together with CMake/catkin, plus Python wrappers and ROS-oriented Docker/CI files.
2. Primary purpose
- The repository is Kalibr: a calibration toolbox for multi-camera, camera-IMU, IMU-IMU, and rolling-shutter calibration.
- The architecture is centered on turning sensor observations into a nonlinear optimization problem, solving it, then exporting results and diagnostics.
3. Core optimization engine
- aslam_optimizer/aslam_backend is the main estimator core.
- It provides design variables, error terms, Jacobian construction, trust-region optimizers, and linear solvers.
- sparse_block_matrix under the same area supplies the sparse linear algebra substrate used by the backend, including CHOLMOD/CSparse/SPQR/PCG-style solvers and block-matrix structures.
4. Expression and geometry layer
- aslam_optimizer/aslam_backend_expressions adds typed expression graphs for scalars, vectors, rotations, transformations, and homogeneous points.
- These abstractions let calibration problems compose analytic expressions and feed them into backend error terms.
- This is the bridge between geometric model objects and optimization variables.
5. Camera and vision model stack
- aslam_cv/aslam_cameras contains camera geometries, projection models, distortions, shutters, frames, keypoints, landmarks, and triangulation.
- It supports multiple camera models: pinhole, omni, double sphere, extended unified, equidistant, radial-tangential, depth, and rolling/global shutter.
- aslam_imgproc provides undistortion and overlap-FOV helpers, while aslam_cv_serialization handles serialization of camera/frame types.
6. Calibration problem formulation
- aslam_cv_backend defines camera design variables, reprojection-related estimation containers, and calibration-specific backend glue.
- aslam_cv_error_terms defines the main reprojection and covariance error terms used by camera calibration problems.
- These modules sit directly on top of the optimizer backend and translate camera observations into residuals.
7. IMU, timing, and motion modeling
- aslam_time and sm_timing provide time/duration/timestamp utilities and timestamp correction helpers.
- aslam_splines and aslam_nonparametric_estimation provide continuous-time BSpline motion models used to represent camera/IMU trajectories smoothly over time.
- This matters for Kalibr because temporal alignment and continuous-time motion are central to the calibration problem.
8. Offline calibration applications
- aslam_offline_calibration/kalibr is the main user-facing calibration application package.
- Its Python code contains command-line tools and workflows for camera calibration, IMU-camera calibration, rolling-shutter calibration, validation, target extraction, plotting, and report generation.
- It uses the C++ libraries for heavy computation and Python for orchestration and visualization.
9. Target detection and input processing
- ethz_apriltag2 supplies AprilTag detection used as a calibration target frontend.
- The Python-side calibration tools read image datasets and IMU datasets, detect calibration targets, and build observation databases before optimization.
- This is the main input path: raw sensor logs → detections/feature observations → calibration graph.
10. Data flow through the system
- Sensor data enters through dataset readers and target extractors in Python.
- Observations become camera frames, keypoints, IMU constraints, and spline-based motion states.
- These are converted into design variables and error terms in the backend, then solved by sparse nonlinear optimization.
- Results are exported to files, plots, and tool-specific configuration formats.
11. Python bindings as the integration boundary
- Several packages expose C++ libraries to Python: sm_python, aslam_cv_python, aslam_cv_backend_python, aslam_splines_python, aslam_backend_python, and incremental_calibration_python.
- These bindings are important because most end-user calibration workflows, visualizations, and exporters are implemented in Python, while compute-heavy geometry/optimization stays in C++.
12. Incremental calibration subsystem
- aslam_incremental_calibration/incremental_calibration is a separate calibration framework focused on online/incremental optimization.
- It includes its own core optimization, linear algebra, statistics, threading, and data-structure layers, plus a Python wrapper.
- This is architecturally adjacent to Kalibr’s offline pipeline but serves a more streaming/incremental use case.
13. Shared utility libraries
- The Schweizer-Messer packages provide reusable foundations:
- sm_common: general utilities, IDs, assertions, serialization helpers
- sm_eigen: Eigen helpers, random utilities, matrix square roots
- sm_kinematics: rotations, transformations, homogeneous coordinates
- sm_logging: logging infrastructure
- sm_matrix_archive: matrix persistence
- sm_random: random utilities
- sm_boost: Boost wrappers and binary archive helpers
- sm_property_tree: config/property tree abstractions
- sm_python: Python-facing helpers and exports
- These are shared support layers rather than standalone apps.
14. Build, packaging, and deployment
- The repo is organized as ROS packages with package.xml and CMakeLists.txt per package.
- Dockerfiles target ROS1 on Ubuntu 16.04/18.04/20.04, and GitHub Actions validate those builds.
- The workspace is designed to be built from source in a ROS environment or run in containers, which affects how the whole stack is deployed.
15. How to draw the architecture
- At the center, place the optimizer backend and sparse linear algebra.
- Feed it from camera/IMU models, spline motion models, and calibration-specific error terms.
- Surround that with Python calibration tools for dataset ingestion, target detection, plotting, and export.
- Attach shared utility libraries underneath as foundational dependencies, and show Docker/CI as build/runtime infrastructure around the workspace.
Main entry points: aslam_backend Backend · kalibr Kalibr · aslam_splines Splines · aslam_cameras Cameras
Components 22
Shared libs
Reusable Schweizer-Messer foundations and timing/math utilities.
- sm_common · Common utilities, IDs, assertions, serialization helpers.
Schweizer-Messer/sm_common - sm_eigen · Eigen helpers, random utilities, matrix square roots, property-tree support.
Schweizer-Messer/sm_eigen - sm_kinematics · Rotations, transformations, homogeneous coordinates, uncertain geometry.
Schweizer-Messer/sm_kinematics - sm_logging · Logging infrastructure and macros.
Schweizer-Messer/sm_logging - sm_python · Python-facing helpers and exports for shared types.
Schweizer-Messer/sm_python - sm_timing · Duration, timestamp, timer, and timestamp correction utilities.
Schweizer-Messer/sm_timing
Optimization core
Nonlinear estimation backend and sparse linear algebra substrate.
- Backend · Design variables, error terms, Jacobians, trust-region optimization, and linear system solving.
aslam_optimizer/aslam_backend - Sparse blocks · Sparse block matrix structures and CHOLMOD/CSparse/SPQR/PCG-style solvers.
aslam_optimizer/sparse_block_matrix - Expressions · Typed scalar, vector, rotation, transformation, and homogeneous expression graphs.
aslam_optimizer/aslam_backend_expressions
Vision stack
Camera models, image processing, serialization, and backend glue.
- Cameras · Camera geometries, projections, distortions, shutters, frames, keypoints, landmarks, triangulation.
aslam_cv/aslam_cameras - Image proc · Undistortion helpers and overlap-FOV utilities.
aslam_cv/aslam_imgproc - Serialization · Serialization for frames and camera geometry types.
aslam_cv/aslam_cv_serialization - CV backend · Camera design variables and calibration-specific backend containers.
aslam_cv/aslam_cv_backend - CV errors · Reprojection and covariance error terms for camera calibration.
aslam_cv/aslam_cv_error_terms - CV Python · Python bindings and orchestration for camera frontend, calibration, and undistortion.
aslam_cv/aslam_cv_python
Motion models
Continuous-time spline representations used for trajectories and temporal alignment.
- Splines · BSpline motion models, design variables, and spline-specific errors.
aslam_nonparametric_estimation/aslam_splines - Spline Python · Python bindings for spline motion models and errors.
aslam_nonparametric_estimation/aslam_splines_python
Kalibr tools
Offline calibration workflows, target extraction, and exporters.
Incremental calib
Streaming calibration framework and its Python wrapper.
- Inc core · Online/incremental optimization, linear algebra, statistics, threading, and data structures.
aslam_incremental_calibration/incremental_calibration - Inc Python · Python bindings for incremental calibration and vision data association.
aslam_incremental_calibration/incremental_calibration_python
Build infra
Containers and CI for the ROS workspace.
- Containers/CI · Docker images and GitHub Actions for ROS1 build validation.
Connections 28
- sm_common → Backend: foundation · Shared utilities support optimizer core.
- sm_eigen → Backend: math · Eigen helpers underpin numerical routines.
- sm_kinematics → Expressions: geometry · Rotation and transformation primitives feed typed expressions.
- sm_kinematics → Cameras: geometry · Camera and frame math depends on transformations and rotations.
- sm_logging → Kalibr: logs · Calibration workflows use shared logging infrastructure.
- sm_python → CV Python: bridge · Shared Python export helpers support CV bindings.
- sm_python → Spline Python: bridge · Shared Python export helpers support spline bindings.
- sm_timing → CV Python: time · Time and timestamp utilities are exposed to Python workflows.
- sm_timing → Kalibr: timestamps · Dataset handling and temporal alignment rely on timing utilities.
- Sparse blocks → Backend: linear solve · Sparse block algebra is the solver substrate for optimization.
- Expressions → Backend: residuals · Typed expressions are converted into backend error terms and variables.
- Cameras → CV backend: models · Camera geometries are wrapped as design variables and estimation containers.
- Cameras → CV errors: reprojection · Camera models define the residuals used by calibration error terms.
- Image proc → CV Python: undistort · Undistortion helpers are surfaced to Python calibration tools.
- Serialization → Cameras: serialize · Camera and frame types are persisted and reconstructed through serializers.
- CV backend → Backend: backend glue · Calibration containers plug camera variables into the optimizer.
- CV errors → Backend: error terms · Calibration residuals are built on the backend estimator core.
- Splines → Backend: motion residuals · BSpline motion terms are optimization error terms over the backend.
- Splines → Expressions: spline expr · Spline state interpolation is represented with backend expressions.
- Spline Python → Splines: wrap · Python bindings expose spline models and errors.
- CV Python → Kalibr: orchestrate · Python camera tooling drives the offline calibration workflows.
- AprilTag2 → Kalibr: targets · AprilTag detection provides the calibration target frontend.
- Inc core → Inc Python: wrap · Incremental calibration core is exposed to Python.
- Backend → Kalibr: solve · Kalibr workflows invoke the nonlinear optimizer for final estimation.
- CV backend → Kalibr: problem setup · Camera calibration workflows assemble the estimation problem.
- Splines → Kalibr: trajectories · Continuous-time motion models support IMU-camera and rolling-shutter calibration.
- Containers/CI → Kalibr: builds · Container and CI setup validate the ROS workspace and calibration packages.
- Containers/CI → Backend: builds · Infrastructure also validates core optimizer packages.