# uoip/stereo_msckf architecture

> Architecture diagram of the GitHub repository uoip/stereo_msckf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uoip/stereo_msckf
- Repository: https://github.com/uoip/stereo_msckf

## Overview

##### Project type and purpose
- This is a research-grade, open-source tool: a Python re-implementation of the stereo version of the Multi-State Constraint Kalman Filter (S-MSCKF) for visual-inertial odometry. Its goal is to load stereo images and IMU data (from EuRoC MAV), tightly couple them in an EKF, and output a pose trajectory (optionally visualized).

##### File structure
- Top–level modules:
- config.py        – global parameters (camera intrinsics, noise models, EKF settings)
- dataset.py       – loader/publisher for the EuRoC MAV dataset (stereo images + IMU)
- image.py         – image preprocessing (undistortion, rectification)
- feature.py       – feature detection/tracking and stereo matching
- msckf.py         – core EKF state, propagation, update, residual building
- vio.py           – main script / CLI entrypoint (argument parsing, orchestrator)
- viewer.py        – optional trajectory visualization (pangolin)
- utils.py         – linear algebra helpers, quaternion math, timing, config parsing
- imgs/            – sample output plots
- test/            – unit tests for feature initialization and utility functions
- No separate “frontend/backend”—this is a single executable pipeline.

##### README for additional insights
- Confirms stereo-MSCKF algorithm, dependencies (numpy, scipy, OpenCV, optional pangolin), dataset type, and run commands.

##### Main components & their relationships
a. Configuration Module (config.py)
- Defines camera calibration, IMU noise, EKF parameters
- Consumed by dataset loader, feature extractor, and MSCKF core
b. Dataset Loader & Publisher (dataset.py)
- Reads stereo image streams + time-synced IMU data
- Publishes timestamped IMU measurements & image pairs downstream
c. Image Preprocessing (image.py)
- Undistorts / rectifies raw images based on config intrinsics
- Forwards rectified images to feature module
d. Feature Processing (feature.py)
- Detects keypoints, tracks them across frames, and performs stereo match
- Builds feature tracks & measurement lists for filter update
e. MSCKF Core Filter (msckf.py)
- Maintains EKF state: IMU pose, velocity, biases + cloned camera poses
- Propagation: integrates IMU measurements to predict state
- Update: constructs measurement Jacobians from feature tracks, applies EKF update
- Manages state augmentation / marginalization
f. Visualization Module (viewer.py)
- Subscribes to updated pose stream
- Renders trajectory & keyframe poses (using pangolin if available)
g. Orchestrator / CLI (vio.py)
- Parses command-line args (path, view flag)
- Instantiates Dataset, MSCKF, Viewer
- Drives the main loop: pull IMU until next image, propagate filter, receive image, detect features, update filter, send pose to viewer
h. Utilities (utils.py)
- Shared math routines: quaternion‐vector transforms, Jacobian computations, linear solvers

##### Architectural patterns & design principles
- Modular pipeline: clear separation of data ingestion, preprocessing, feature extraction, estimation core, visualization.
- Publisher/subscriber (dataset publishes, MSCKF & viewer subscribe) – can be drawn as event streams.
- Stateful filter with propagation/update loop.
- Config-driven behavior: all parameters centrally defined.
- Test harness for correctness on small units (feature init, utils).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Orchestrator"
        VIO["VIO CLI Orchestrator"]:::orchestrator
    end

    subgraph "Data Ingestion"
        CONFIG["Config Module"]:::data
        DATASET["Dataset Loader & Publisher"]:::data
    end

    subgraph "Preprocessing"
        IMAGE["Image Preprocessing"]:::processing
        FEATURE["Feature Processing"]:::processing
    end

    subgraph "Estimation"
        MSCKF["MSCKF Core Filter"]:::processing
    end

    subgraph "Visualization"
        VIEWER["Visualization Module"]:::visual
    end

    subgraph "Utilities" 
        UTILS["Utils Library"]:::utils
    end

    subgraph "Tests"
        F_TEST["Feature Init Tests"]:::test
        U_TEST["Utils Tests"]:::test
    end

    VIO --> CONFIG
    VIO --> DATASET
    VIO --> MSCKF
    VIO --> VIEWER

    CONFIG --> DATASET
    CONFIG --> IMAGE
    CONFIG --> FEATURE
    CONFIG --> MSCKF

    DATASET -->|"IMU measurements"| MSCKF
    DATASET -->|"Stereo images"| IMAGE

    IMAGE -->|"Rectified images"| FEATURE
    FEATURE -->|"Feature tracks"| MSCKF

    MSCKF -->|"Pose estimates"| VIEWER

    UTILS --> CONFIG
    UTILS --> FEATURE
    UTILS --> MSCKF

    F_TEST --> FEATURE
    U_TEST --> UTILS

    click CONFIG "https://github.com/uoip/stereo_msckf/blob/master/config.py"
    click DATASET "https://github.com/uoip/stereo_msckf/blob/master/dataset.py"
    click IMAGE "https://github.com/uoip/stereo_msckf/blob/master/image.py"
    click FEATURE "https://github.com/uoip/stereo_msckf/blob/master/feature.py"
    click MSCKF "https://github.com/uoip/stereo_msckf/blob/master/msckf.py"
    click VIEWER "https://github.com/uoip/stereo_msckf/blob/master/viewer.py"
    click VIO "https://github.com/uoip/stereo_msckf/blob/master/vio.py"
    click UTILS "https://github.com/uoip/stereo_msckf/blob/master/utils.py"
    click F_TEST "https://github.com/uoip/stereo_msckf/blob/master/test/feature_initialization_test.py"
    click U_TEST "https://github.com/uoip/stereo_msckf/blob/master/test/utils_test.py"

    classDef data fill:#BBDEFB,stroke:#2196F3,color:#0D47A1
    classDef processing fill:#C8E6C9,stroke:#388E3C,color:#1B5E20
    classDef visual fill:#FFE0B2,stroke:#FB8C00,color:#E65100
    classDef orchestrator fill:#E0E0E0,stroke:#9E9E9E,color:#424242
    classDef utils fill:#FFFFFF,stroke:#616161,color:#424242,stroke-dasharray: 5 5
    classDef test fill:#F8BBD0,stroke:#C2185B,color:#880E4F
```
