# ethz-asl/kalibr architecture

> Architecture diagram of the GitHub repository ethz-asl/kalibr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ethz-asl/kalibr
- Repository: https://github.com/ethz-asl/kalibr
- Video tour (about a minute): https://gitdiagram.com/ethz-asl/kalibr/video

## Overview

1. **Repository shape**
   - This is a ROS1 catkin workspace, not a single application.
   - The root contains several library stacks and application/tooling packages that are built together with CMake/catkin, plus Python wrappers and ROS-oriented Docker/CI files.

2. **Primary purpose**
   - The repository is Kalibr: a calibration toolbox for multi-camera, camera-IMU, IMU-IMU, and rolling-shutter calibration.
   - The architecture is centered on turning sensor observations into a nonlinear optimization problem, solving it, then exporting results and diagnostics.

3. **Core optimization engine**
   - `aslam_optimizer/aslam_backend` is the main estimator core.
   - It provides design variables, error terms, Jacobian construction, trust-region optimizers, and linear solvers.
   - `sparse_block_matrix` under the same area supplies the sparse linear algebra substrate used by the backend, including CHOLMOD/CSparse/SPQR/PCG-style solvers and block-matrix structures.

4. **Expression and geometry layer**
   - `aslam_optimizer/aslam_backend_expressions` adds typed expression graphs for scalars, vectors, rotations, transformations, and homogeneous points.
   - These abstractions let calibration problems compose analytic expressions and feed them into backend error terms.
   - This is the bridge between geometric model objects and optimization variables.

5. **Camera and vision model stack**
   - `aslam_cv/aslam_cameras` contains camera geometries, projection models, distortions, shutters, frames, keypoints, landmarks, and triangulation.
   - It supports multiple camera models: pinhole, omni, double sphere, extended unified, equidistant, radial-tangential, depth, and rolling/global shutter.
   - `aslam_imgproc` provides undistortion and overlap-FOV helpers, while `aslam_cv_serialization` handles serialization of camera/frame types.

6. **Calibration problem formulation**
   - `aslam_cv_backend` defines camera design variables, reprojection-related estimation containers, and calibration-specific backend glue.
   - `aslam_cv_error_terms` defines the main reprojection and covariance error terms used by camera calibration problems.
   - These modules sit directly on top of the optimizer backend and translate camera observations into residuals.

7. **IMU, timing, and motion modeling**
   - `aslam_time` and `sm_timing` provide time/duration/timestamp utilities and timestamp correction helpers.
   - `aslam_splines` and `aslam_nonparametric_estimation` provide continuous-time BSpline motion models used to represent camera/IMU trajectories smoothly over time.
   - This matters for Kalibr because temporal alignment and continuous-time motion are central to the calibration problem.

8. **Offline calibration applications**
   - `aslam_offline_calibration/kalibr` is the main user-facing calibration application package.
   - Its Python code contains command-line tools and workflows for camera calibration, IMU-camera calibration, rolling-shutter calibration, validation, target extraction, plotting, and report generation.
   - It uses the C++ libraries for heavy computation and Python for orchestration and visualization.

9. **Target detection and input processing**
   - `ethz_apriltag2` supplies AprilTag detection used as a calibration target frontend.
   - The Python-side calibration tools read image datasets and IMU datasets, detect calibration targets, and build observation databases before optimization.
   - This is the main input path: raw sensor logs → detections/feature observations → calibration graph.

10. **Data flow through the system**
   - Sensor data enters through dataset readers and target extractors in Python.
   - Observations become camera frames, keypoints, IMU constraints, and spline-based motion states.
   - These are converted into design variables and error terms in the backend, then solved by sparse nonlinear optimization.
   - Results are exported to files, plots, and tool-specific configuration formats.

11. **Python bindings as the integration boundary**
   - Several packages expose C++ libraries to Python: `sm_python`, `aslam_cv_python`, `aslam_cv_backend_python`, `aslam_splines_python`, `aslam_backend_python`, and `incremental_calibration_python`.
   - These bindings are important because most end-user calibration workflows, visualizations, and exporters are implemented in Python, while compute-heavy geometry/optimization stays in C++.

12. **Incremental calibration subsystem**
   - `aslam_incremental_calibration/incremental_calibration` is a separate calibration framework focused on online/incremental optimization.
   - It includes its own core optimization, linear algebra, statistics, threading, and data-structure layers, plus a Python wrapper.
   - This is architecturally adjacent to Kalibr’s offline pipeline but serves a more streaming/incremental use case.

13. **Shared utility libraries**
   - The `Schweizer-Messer` packages provide reusable foundations:
     - `sm_common`: general utilities, IDs, assertions, serialization helpers
     - `sm_eigen`: Eigen helpers, random utilities, matrix square roots
     - `sm_kinematics`: rotations, transformations, homogeneous coordinates
     - `sm_logging`: logging infrastructure
     - `sm_matrix_archive`: matrix persistence
     - `sm_random`: random utilities
     - `sm_boost`: Boost wrappers and binary archive helpers
     - `sm_property_tree`: config/property tree abstractions
     - `sm_python`: Python-facing helpers and exports
   - These are shared support layers rather than standalone apps.

14. **Build, packaging, and deployment**
   - The repo is organized as ROS packages with `package.xml` and `CMakeLists.txt` per package.
   - Dockerfiles target ROS1 on Ubuntu 16.04/18.04/20.04, and GitHub Actions validate those builds.
   - The workspace is designed to be built from source in a ROS environment or run in containers, which affects how the whole stack is deployed.

15. **How to draw the architecture**
   - At the center, place the optimizer backend and sparse linear algebra.
   - Feed it from camera/IMU models, spline motion models, and calibration-specific error terms.
   - Surround that with Python calibration tools for dataset ingestion, target detection, plotting, and export.
   - Attach shared utility libraries underneath as foundational dependencies, and show Docker/CI as build/runtime infrastructure around the workspace.

Main entry points: [`aslam_optimizer/aslam_backend`](https://github.com/ethz-asl/kalibr/tree/master/aslam_optimizer/aslam_backend) (Backend), [`aslam_offline_calibration/kalibr`](https://github.com/ethz-asl/kalibr/tree/master/aslam_offline_calibration/kalibr) (Kalibr), [`aslam_nonparametric_estimation/aslam_splines`](https://github.com/ethz-asl/kalibr/tree/master/aslam_nonparametric_estimation/aslam_splines) (Splines), [`aslam_cv/aslam_cameras`](https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cameras) (Cameras)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_shared["Shared libs"]
  node_sm_common["sm_common<br/>utility core"]
  node_sm_eigen["sm_eigen<br/>eigen helpers"]
  node_sm_kinematics["sm_kinematics<br/>geometry core"]
  node_sm_logging["sm_logging<br/>logging core"]
  node_sm_python["sm_python<br/>python bridge"]
  node_sm_timing["sm_timing<br/>time utils"]
end

subgraph group_optimizer["Optimization core"]
  node_backend["Backend<br/>nonlinear optimizer"]
  node_sbm["Sparse blocks<br/>sparse linear algebra"]
  node_backend_expr["Expressions<br/>typed geometry expr"]
end

subgraph group_vision["Vision stack"]
  node_cameras["Cameras<br/>camera models"]
  node_imgproc["Image proc<br/>undistortion tools"]
  node_cv_ser["Serialization<br/>cv serializers"]
  node_cv_backend["CV backend<br/>calibration glue"]
  node_cv_errors["CV errors<br/>reprojection errors"]
  node_cv_py["CV Python<br/>python wrappers"]
end

subgraph group_motion["Motion models"]
  node_splines["Splines<br/>bspline motion"]
  node_splines_py["Spline Python<br/>python wrappers"]
end

subgraph group_kalibr["Kalibr tools"]
  node_kalibr_app["Kalibr<br/>offline calibration"]
  node_apriltag["AprilTag2<br/>target frontend"]
end

subgraph group_incremental["Incremental calib"]
  node_inc_core["Inc core<br/>incremental optimizer"]
  node_inc_py["Inc Python<br/>python wrapper"]
end

subgraph group_infra["Build infra"]
  node_infra["Containers/CI<br/>build infra"]
end

node_sm_common -->|"foundation"| node_backend
node_sm_eigen -->|"math"| node_backend
node_sm_kinematics -->|"geometry"| node_backend_expr
node_sm_kinematics -->|"geometry"| node_cameras
node_sm_logging -->|"logs"| node_kalibr_app
node_sm_python -->|"bridge"| node_cv_py
node_sm_python -->|"bridge"| node_splines_py
node_sm_timing -->|"time"| node_cv_py
node_sm_timing -->|"timestamps"| node_kalibr_app
node_sbm -->|"linear solve"| node_backend
node_backend_expr -->|"residuals"| node_backend
node_cameras -->|"models"| node_cv_backend
node_cameras -->|"reprojection"| node_cv_errors
node_imgproc -->|"undistort"| node_cv_py
node_cv_ser -->|"serialize"| node_cameras
node_cv_backend -->|"backend glue"| node_backend
node_cv_errors -->|"error terms"| node_backend
node_splines -->|"motion residuals"| node_backend
node_splines -->|"spline expr"| node_backend_expr
node_splines_py -->|"wrap"| node_splines
node_cv_py -->|"orchestrate"| node_kalibr_app
node_apriltag -->|"targets"| node_kalibr_app
node_inc_core -->|"wrap"| node_inc_py
node_backend -->|"solve"| node_kalibr_app
node_cv_backend -->|"problem setup"| node_kalibr_app
node_splines -->|"trajectories"| node_kalibr_app
node_infra -.->|"builds"| node_kalibr_app
node_infra -.->|"builds"| node_backend

click node_sm_common "https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_common"
click node_sm_eigen "https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_eigen"
click node_sm_kinematics "https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_kinematics"
click node_sm_logging "https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_logging"
click node_sm_python "https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_python"
click node_sm_timing "https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_timing"
click node_backend "https://github.com/ethz-asl/kalibr/tree/master/aslam_optimizer/aslam_backend"
click node_sbm "https://github.com/ethz-asl/kalibr/tree/master/aslam_optimizer/sparse_block_matrix"
click node_backend_expr "https://github.com/ethz-asl/kalibr/tree/master/aslam_optimizer/aslam_backend_expressions"
click node_cameras "https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cameras"
click node_imgproc "https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_imgproc"
click node_cv_ser "https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cv_serialization"
click node_cv_backend "https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cv_backend"
click node_cv_errors "https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cv_error_terms"
click node_cv_py "https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cv_python"
click node_splines "https://github.com/ethz-asl/kalibr/tree/master/aslam_nonparametric_estimation/aslam_splines"
click node_splines_py "https://github.com/ethz-asl/kalibr/tree/master/aslam_nonparametric_estimation/aslam_splines_python"
click node_kalibr_app "https://github.com/ethz-asl/kalibr/tree/master/aslam_offline_calibration/kalibr"
click node_apriltag "https://github.com/ethz-asl/kalibr/tree/master/aslam_offline_calibration/ethz_apriltag2"
click node_inc_core "https://github.com/ethz-asl/kalibr/tree/master/aslam_incremental_calibration/incremental_calibration"
click node_inc_py "https://github.com/ethz-asl/kalibr/tree/master/aslam_incremental_calibration/incremental_calibration_python"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sm_common,node_sm_eigen,node_sm_kinematics,node_sm_logging,node_sm_python,node_sm_timing,node_infra toneBlue
class node_backend,node_sbm,node_backend_expr toneAmber
class node_cameras,node_imgproc,node_cv_ser,node_cv_backend,node_cv_errors,node_cv_py toneMint
class node_splines,node_splines_py toneRose
class node_kalibr_app,node_apriltag toneIndigo
class node_inc_core,node_inc_py toneTeal
```

## Components

### Shared libs

Reusable Schweizer-Messer foundations and timing/math utilities.

- [sm_common](https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_common): utility core · Common utilities, IDs, assertions, serialization helpers. · `Schweizer-Messer/sm_common`
- [sm_eigen](https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_eigen): eigen helpers · Eigen helpers, random utilities, matrix square roots, property-tree support. · `Schweizer-Messer/sm_eigen`
- [sm_kinematics](https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_kinematics): geometry core · Rotations, transformations, homogeneous coordinates, uncertain geometry. · `Schweizer-Messer/sm_kinematics`
- [sm_logging](https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_logging): logging core · Logging infrastructure and macros. · `Schweizer-Messer/sm_logging`
- [sm_python](https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_python): python bridge · Python-facing helpers and exports for shared types. · `Schweizer-Messer/sm_python`
- [sm_timing](https://github.com/ethz-asl/kalibr/tree/master/Schweizer-Messer/sm_timing): time utils · Duration, timestamp, timer, and timestamp correction utilities. · `Schweizer-Messer/sm_timing`

### Optimization core

Nonlinear estimation backend and sparse linear algebra substrate.

- [Backend](https://github.com/ethz-asl/kalibr/tree/master/aslam_optimizer/aslam_backend): nonlinear optimizer · Design variables, error terms, Jacobians, trust-region optimization, and linear system solving. · `aslam_optimizer/aslam_backend`
- [Sparse blocks](https://github.com/ethz-asl/kalibr/tree/master/aslam_optimizer/sparse_block_matrix): sparse linear algebra · Sparse block matrix structures and CHOLMOD/CSparse/SPQR/PCG-style solvers. · `aslam_optimizer/sparse_block_matrix`
- [Expressions](https://github.com/ethz-asl/kalibr/tree/master/aslam_optimizer/aslam_backend_expressions): typed geometry expr · Typed scalar, vector, rotation, transformation, and homogeneous expression graphs. · `aslam_optimizer/aslam_backend_expressions`

### Vision stack

Camera models, image processing, serialization, and backend glue.

- [Cameras](https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cameras): camera models · Camera geometries, projections, distortions, shutters, frames, keypoints, landmarks, triangulation. · `aslam_cv/aslam_cameras`
- [Image proc](https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_imgproc): undistortion tools · Undistortion helpers and overlap-FOV utilities. · `aslam_cv/aslam_imgproc`
- [Serialization](https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cv_serialization): cv serializers · Serialization for frames and camera geometry types. · `aslam_cv/aslam_cv_serialization`
- [CV backend](https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cv_backend): calibration glue · Camera design variables and calibration-specific backend containers. · `aslam_cv/aslam_cv_backend`
- [CV errors](https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cv_error_terms): reprojection errors · Reprojection and covariance error terms for camera calibration. · `aslam_cv/aslam_cv_error_terms`
- [CV Python](https://github.com/ethz-asl/kalibr/tree/master/aslam_cv/aslam_cv_python): python wrappers · Python bindings and orchestration for camera frontend, calibration, and undistortion. · `aslam_cv/aslam_cv_python`

### Motion models

Continuous-time spline representations used for trajectories and temporal alignment.

- [Splines](https://github.com/ethz-asl/kalibr/tree/master/aslam_nonparametric_estimation/aslam_splines): bspline motion · BSpline motion models, design variables, and spline-specific errors. · `aslam_nonparametric_estimation/aslam_splines`
- [Spline Python](https://github.com/ethz-asl/kalibr/tree/master/aslam_nonparametric_estimation/aslam_splines_python): python wrappers · Python bindings for spline motion models and errors. · `aslam_nonparametric_estimation/aslam_splines_python`

### Kalibr tools

Offline calibration workflows, target extraction, and exporters.

- [Kalibr](https://github.com/ethz-asl/kalibr/tree/master/aslam_offline_calibration/kalibr): offline calibration · User-facing calibration workflows, validation, plotting, and reporting. · `aslam_offline_calibration/kalibr`
- [AprilTag2](https://github.com/ethz-asl/kalibr/tree/master/aslam_offline_calibration/ethz_apriltag2): target frontend · AprilTag detector and calibration target extraction frontend. · `aslam_offline_calibration/ethz_apriltag2`

### Incremental calib

Streaming calibration framework and its Python wrapper.

- [Inc core](https://github.com/ethz-asl/kalibr/tree/master/aslam_incremental_calibration/incremental_calibration): incremental optimizer · Online/incremental optimization, linear algebra, statistics, threading, and data structures. · `aslam_incremental_calibration/incremental_calibration`
- [Inc Python](https://github.com/ethz-asl/kalibr/tree/master/aslam_incremental_calibration/incremental_calibration_python): python wrapper · Python bindings for incremental calibration and vision data association. · `aslam_incremental_calibration/incremental_calibration_python`

### Build infra

Containers and CI for the ROS workspace.

- **Containers/CI**: build infra · Docker images and GitHub Actions for ROS1 build validation.

## Connections

- sm_common → Backend: foundation · Shared utilities support optimizer core.
- sm_eigen → Backend: math · Eigen helpers underpin numerical routines.
- sm_kinematics → Expressions: geometry · Rotation and transformation primitives feed typed expressions.
- sm_kinematics → Cameras: geometry · Camera and frame math depends on transformations and rotations.
- sm_logging → Kalibr: logs · Calibration workflows use shared logging infrastructure.
- sm_python → CV Python: bridge · Shared Python export helpers support CV bindings.
- sm_python → Spline Python: bridge · Shared Python export helpers support spline bindings.
- sm_timing → CV Python: time · Time and timestamp utilities are exposed to Python workflows.
- sm_timing → Kalibr: timestamps · Dataset handling and temporal alignment rely on timing utilities.
- Sparse blocks → Backend: linear solve · Sparse block algebra is the solver substrate for optimization.
- Expressions → Backend: residuals · Typed expressions are converted into backend error terms and variables.
- Cameras → CV backend: models · Camera geometries are wrapped as design variables and estimation containers.
- Cameras → CV errors: reprojection · Camera models define the residuals used by calibration error terms.
- Image proc → CV Python: undistort · Undistortion helpers are surfaced to Python calibration tools.
- Serialization → Cameras: serialize · Camera and frame types are persisted and reconstructed through serializers.
- CV backend → Backend: backend glue · Calibration containers plug camera variables into the optimizer.
- CV errors → Backend: error terms · Calibration residuals are built on the backend estimator core.
- Splines → Backend: motion residuals · BSpline motion terms are optimization error terms over the backend.
- Splines → Expressions: spline expr · Spline state interpolation is represented with backend expressions.
- Spline Python → Splines: wrap · Python bindings expose spline models and errors.
- CV Python → Kalibr: orchestrate · Python camera tooling drives the offline calibration workflows.
- AprilTag2 → Kalibr: targets · AprilTag detection provides the calibration target frontend.
- Inc core → Inc Python: wrap · Incremental calibration core is exposed to Python.
- Backend → Kalibr: solve · Kalibr workflows invoke the nonlinear optimizer for final estimation.
- CV backend → Kalibr: problem setup · Camera calibration workflows assemble the estimation problem.
- Splines → Kalibr: trajectories · Continuous-time motion models support IMU-camera and rolling-shutter calibration.
- Containers/CI → Kalibr: builds · Container and CI setup validate the ROS workspace and calibration packages.
- Containers/CI → Backend: builds · Infrastructure also validates core optimizer packages.
