# bytedance/schurvins architecture

> Architecture diagram of the GitHub repository bytedance/schurvins, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance/schurvins
- Repository: https://github.com/bytedance/schurvins

## Overview

##### Project Type and Purpose
- SchurVINS is a robotics/perception software system built on top of SVO2.0. It is a full-fledged Visual‐Inertial Navigation System (VINS), structured as a ROS catkin workspace with multiple packages for frontend processing, backend optimization, loop closing, global mapping, benchmarking, visualization, and utility libraries.
- Its goal is to perform real‐time ego‐motion estimation and landmark mapping using an EKF framework with Schur complement for efficiency.

##### High‐Level Architectural Components
a. Sensor Interface & Data Flow
- ROS Topics: Camera images, IMU measurements (via svo_ros/csv_dataset_reader or live ROS topics)
- Launch files (svo_ros/launch/*) orchestrate three processes: SVO nodelet (svo_ros), RViz visualizer, and data playback.
b. Frontend Processing
- Frame Handler (svo/src/frame_handler_*.cpp) – handles mono, stereo, array cameras
- Direct & Feature Tracking Modules (svo_direct, svo_tracker) – compute image features or direct photometric alignments
- IMU Handler (svo/src/imu_handler.cpp) – preintegrates IMU data
- Initialization (svo/src/initialization.cpp) – bootstraps VIO
c. Backend Optimization
- SchurVINS Core (svo/include/svo/schur_vins.h, svo/src/schur_vins.cpp) – formulates full residual, splits via Schur complement into ego‐motion and landmark models, performs EKF updates
- Ceres‐Based Solvers (svo_ceres_backend) – provides nonlinear least squares for landmark refinement, margin­talization, parameter blocks, callbacks
- GTSAM‐Based Global Mapping (svo_global_map) – optimizes global map with smart factors
- Pose Graph Optimization (svo_pgo) – runs global PGO on keyframe graph
d. Loop Closing
- Online Loopclosing Module (svo_online_loopclosing) – bag‐of‐words retrieval, geometric verification, map alignment
e. ROS Integration & Visualization
- ROS Nodelets & Nodes (svo_ros/src/*.cpp) – wrap the SVO core into ROS, publish odometry, map, landmarks, and diagnostics
- RQT GUI (rqt_svo) – interactive widget for parameter tweaking, visualization of features, tracks
- RViz Configs & Visualizer (svo_ros/rviz_config*.rviz, svo_ros/src/visualizer.cpp)
f. Utility Libraries
- Common Types & Helpers (rpg_common, svo_common, svo_vio_common)
- Image Alignment GPU (svo_img_align), camera models (vikit/vikit_cameras), solver utilities (vikit/vikit_solver)
- Test & Benchmarking Tools (svo_benchmarking, svo_test_utils)

##### Relationships & Data Flow
1. Sensor Data → ROS Nodelet (svo_ros)
2. Nodelet feeds → Frontend Pipeline (frame handlers, tracker, IMU handler)
3. Measurements → Initialization / SchurVINS Core
4. SchurVINS splits into:
- Ego‐Motion Residual Model → EKF motion update
- Landmark Residual Model → EKF landmark update
5. Backend Optimization Calls Ceres (for solving nonlinear subproblems)
6. Keyframes & Tracks → Loop Closing → Map Alignment → Pose Graph
7. Global Map updates via GTSAM
8. Final State → ROS Topics → RViz / RQT

##### Key Architectural Patterns & Principles
- Modular ROS architecture: separation of concerns into nodelets, libraries, and launch files
- Pipeline pattern: sequential processing from sensor input to output topics
- Filter‐based estimation: EKF with explicit residual modeling
- Schur complement: reduces computational complexity by marginalizing landmarks
- Plugin‐like backends: choose between Ceres‐based solver or other optimizer via factory (svo_ros/backend_factory)
- Reusable libraries: common types in rpg_common/vikit, encourages code reuse

##### Essential Details to Capture
- Schur complement splitter inside EKF and two residual models
- Backend selection via factory (Ceres vs. default)
- Flow of keyframes into loop closing and pose graph
- Role of rqt_svo for runtime parameter adjustment
- Interaction with benchmarking scripts and datasets (as side utilities)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Sensor Interface & Data Flow
    subgraph "Sensors & Data" 
        Camera["Camera Images"] 
        IMU["IMU Measurements"] 
        Launch["ROS Launch Files"]:::ros
        CSVReader["CSV Dataset Reader"]:::ros
    end

    %% Frontend Processing
    subgraph "Frontend Processing" 
        FHMono["FrameHandler Mono"]:::frontend
        FHStereo["FrameHandler Stereo"]:::frontend
        FHArray["FrameHandler Array"]:::frontend
        DirectTrack["Direct Tracking Module"]:::frontend
        FeatureTrack["Feature Tracking Module"]:::frontend
        IMUHandler["IMU Handler"]:::frontend
        Init["Initialization"]:::frontend
    end

    %% Core Estimation
    subgraph "SchurVINS Core" 
        SchurVINS["EKF & Schur Complement"]:::core
        EgoRes["Ego‐Motion Residual"]:::core
        LandRes["Landmark Residual"]:::core
    end

    %% Backend/Optimization
    subgraph "Optimization Backends" 
        Ceres["Ceres‐Based Solver"]:::backend
        GTSAM["GTSAM Global Mapping"]:::backend
        PGO["Pose Graph Optimization"]:::backend
    end

    %% Loop Closing Module
    LoopClosing["Online Loop Closing"]:::backend

    %% ROS Integration & Visualization
    subgraph "ROS & Visualization" 
        ROSNodelets["ROS Nodelets & Nodes"]:::ros
        BackendFactory["Backend Factory Plugin"]:::ros
        RViz["RViz Configurations"]:::ros
        RQT["RQT GUI"]:::ros
    end

    %% Utility Libraries
    subgraph "Utility Libraries" 
        RPG["rpg_common"]:::util
        SVOCommon["svo_common"]:::util
        VIOCommon["svo_vio_common"]:::util
        GPUAlign["Image Alignment GPU"]:::util
        Cameras["Camera Models"]:::util
        SolverUtil["Solver Utilities"]:::util
    end

    %% Benchmarking & Testing
    Benchmark["Benchmarking Suite"]:::util

    %% Data Stores
    DBMap["Global Map"]:::backend
    PoseGraph["Pose Graph"]:::backend

    %% Connections - Sensor to Frontend
    Camera -->|"images"| ROSNodelets
    IMU -->|"measurements"| ROSNodelets
    CSVReader -->|"/csv_dataset_reader"| ROSNodelets
    Launch -->|"launches"| ROSNodelets

    ROSNodelets -->|"feeds frames & IMU"| FHMono
    ROSNodelets --> FHArray
    ROSNodelets --> FHStereo
    ROSNodelets --> DirectTrack
    ROSNodelets --> FeatureTrack
    ROSNodelets --> IMUHandler
    ROSNodelets --> Init

    %% Frontend to Core
    FHMono -->|"features/alignments"| SchurVINS
    FHStereo --> SchurVINS
    FHArray --> SchurVINS
    DirectTrack --> SchurVINS
    FeatureTrack --> SchurVINS
    IMUHandler --> SchurVINS
    Init --> SchurVINS

    %% Core internal splitter
    SchurVINS --> EgoRes
    SchurVINS --> LandRes

    %% Core to Backends
    EgoRes -->|"solve nonlinear"| Ceres
    LandRes --> Ceres
    Ceres --> SchurVINS

    %% Loop Closing and Global Mapping
    SchurVINS -->|"keyframes & tracks"| LoopClosing
    LoopClosing -.->|"map alignment"| DBMap
    LoopClosing -.->|"graph updates"| PoseGraph
    DBMap -->|"global optimize"| GTSAM
    PoseGraph -->|"PGO solve"| PGO

    %% Core output to ROS & Visualization
    SchurVINS -->|"state & landmarks"| ROSNodelets
    ROSNodelets --> RViz
    ROSNodelets --> RQT

    %% Utilities integration
    FHMono --> RPG
    DirectTrack --> GPUAlign
    Ceres --> SolverUtil
    GTSAM --> Cameras

    %% Benchmarking
    Benchmark --> Ceres
    Benchmark --> GTSAM

    %% Click Events
    click Launch "https://github.com/bytedance/schurvins/blob/master/svo_ros/launch/*.launch"
    click CSVReader "https://github.com/bytedance/schurvins/blob/master/svo_ros/src/csv_dataset_reader.cpp"
    click FHMono "https://github.com/bytedance/schurvins/blob/master/svo/src/frame_handler_mono.cpp"
    click FHStereo "https://github.com/bytedance/schurvins/blob/master/svo/src/frame_handler_stereo.cpp"
    click FHArray "https://github.com/bytedance/schurvins/blob/master/svo/src/frame_handler_array.cpp"
    click DirectTrack "https://github.com/bytedance/schurvins/blob/master/svo_direct/src/feature_alignment.cpp"
    click FeatureTrack "https://github.com/bytedance/schurvins/blob/master/svo_tracker/src/feature_tracker.cpp"
    click IMUHandler "https://github.com/bytedance/schurvins/blob/master/svo/src/imu_handler.cpp"
    click Init "https://github.com/bytedance/schurvins/blob/master/svo/src/initialization.cpp"
    click SchurVINS "https://github.com/bytedance/schurvins/blob/master/svo/include/svo/schur_vins.h"
    click Ceres "https://github.com/bytedance/schurvins/blob/master/svo_ceres_backend/include/svo/ceres_backend_interface.hpp"
    click GTSAM "https://github.com/bytedance/schurvins/blob/master/svo_global_map/include/svo/global_map.h"
    click PGO "https://github.com/bytedance/schurvins/blob/master/svo_pgo/include/svo/pgo/pgo.h"
    click LoopClosing "https://github.com/bytedance/schurvins/blob/master/svo_online_loopclosing/include/svo/online_loopclosing/loop_closing.h"
    click ROSNodelets "https://github.com/bytedance/schurvins/tree/master/svo_ros/src/"
    click BackendFactory "https://github.com/bytedance/schurvins/blob/master/svo_ros/src/backend_factory.cpp"
    click RViz "https://github.com/bytedance/schurvins/blob/master/svo_ros/rviz_config_vio.rviz"
    click RQT "https://github.com/bytedance/schurvins/blob/master/rqt_svo/src/rqt_svo/svo_widget.py"
    click RPG "https://github.com/bytedance/schurvins/tree/master/rpg_common/src"
    click SVOCommon "https://github.com/bytedance/schurvins/tree/master/svo_common/src"
    click VIOCommon "https://github.com/bytedance/schurvins/blob/master/svo_vio_common/src/vio_common.cpp"
    click GPUAlign "https://github.com/bytedance/schurvins/blob/master/svo_img_align/src/frame_gpu.cpp"
    click Cameras "https://github.com/bytedance/schurvins/tree/master/vikit/vikit_cameras/src"
    click SolverUtil "https://github.com/bytedance/schurvins/blob/master/vikit/vikit_solver/include/vikit/solver/mini_least_squares_solver.h"
    click Benchmark "https://github.com/bytedance/schurvins/tree/master/svo_benchmarking/src/svo_ceres_benchmarking"

    %% Styles
    classDef frontend fill:#D0E8FF,stroke:#1E70BF,color:#000
    classDef core fill:#DFF7D8,stroke:#2E8B57,color:#000
    classDef backend fill:#FFE7C6,stroke:#E87500,color:#000
    classDef ros fill:#E5D1F2,stroke:#8A2BE2,color:#000
    classDef util fill:#F0F0F0,stroke:#A0A0A0,color:#000
```
