# air-ufg/delora architecture

> Architecture diagram of the GitHub repository air-ufg/delora, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/air-ufg/delora
- Repository: https://github.com/air-ufg/delora

## Overview

##### Main Components
- Data Ingestion & Preprocessing:
– Directories: src/data, src/preprocessing, and datasets
– Role: This component is responsible for reading in LiDAR scan data from various sources (e.g., KITTI dataset and rosbag files) and preprocessing them (e.g., normal computation, kd-tree generation). The configuration files in the config/ folder (e.g., config_datasets.yaml) inform settings such as sensor specifications.

- Model & Loss Computation:
– Directories: src/models and src/losses
– Role: The core neural network (including the modified ResNet, model parts, etc.) is defined here along with the loss functions (e.g., ICP losses) for self-supervised training.

- Deployment & Training/Testing Management:
– Directories: src/deploy (with deployer.py, trainer.py, tester.py) and the various executable scripts in the bin/ and scripts/ folders
– Role: This module manages training, testing, and deployment routines. It integrates parameter configurations (from config/hyperparameters.yaml and deployment_options.yaml), manages experiment logging with MLFlow, and handles checkpoint usage.

- ROS Integration & Utilities:
– Directories: src/ros_utils
– Role: This part of the system is specific to robotic applications. The ROS nodes handle the conversion of data (e.g., converting scans to rosbag messages, publishing odometry as nav_msgs/Odometry) and integrate the pretrained models into live ROS pipelines.

- Utility & Helper Functions:
– Directories: src/utility
– Role: Implements general-purpose functions (e.g., geometry, linear algebra, plotting) that support other modules.

- Environment & Dependencies:
– Files/directories: conda/, pip/, Dockerfile under docker/
– Role: These files define the runtime, dependencies (PyTorch, CUDA, ROS), and containerization setup which are crucial for the reproducibility and deployment of the project in various environments.

- The Training/Testing module (src/deploy) interfaces with both the model and the loss module.
- An independent branch where the ROS Integration module (src/ros_utils) uses a pretrained model for inference and publishes results through ROS topics.

##### Architectural Patterns and Design Principles
– Separation of Concerns: Each directory encapsulates a specific responsibility (data, model, deployment, ROS, utilities).
– Modularity: The design is modular; for example, the preprocessing and ROS utilities can be modified or extended without impacting core model training.
– Config-driven Behavior: Multiple configuration files allow customization of datasets, hyperparameters, and deployment options.
– Environment Management: Use of Conda (and optionally Docker) ensures consistency in dependencies for both training and inference.

##### Relevant Technologies, Frameworks, and Libraries
– PyTorch: Provided in the conda environment and used for the model.
– ROS/ROS Noetic: Specifically in src/ros_utils for integrating with robotic systems.
– MLFlow: For experiment and parameter logging.
– CUDA/cuDNN: For GPU acceleration (handled through the conda environment).
– Docker: For containerized deployment (see docker/Dockerfile).

##### Summarizing the Process
– Start by listing front-line external inputs (raw LiDAR data, ground truth poses) from the datasets.
– Map their journey through preprocessing (normal computation, kd-tree generation).
– Show the flow into the core deep learning model that predicts odometry, with loss functions guiding training.
– Illustrate how the system supports iterative training with checkpoints and logging, and how it eventually deploys via a ROS node for real-time inference.
– Also indicate that support scripts (for training continuation, model conversion, and time benchmarking) offer additional capabilities that might be drawn as auxiliary components linked to the main flow.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Preprocessing Layer
    subgraph "Data & Preprocessing Layer"
        DataIngest["Data Ingestion (src/data)"]:::data
        Preproc["Preprocessing (src/preprocessing)"]:::data
        Datasets["Datasets Folder"]:::data
        CfgData["Config: config/config_datasets.yaml"]:::data
    end

    %% Model & Loss Computation
    subgraph "Model & Loss Computation"
        Model["Neural Network Model (src/models)"]:::model
        Loss["Loss Computation (src/losses)"]:::model
    end

    %% Deployment & Training/Testing Management
    subgraph "Deployment & Training/Testing"
        Deploy["Training/Testing Manager (src/deploy)"]:::deploy
        BinTrain["Training Script (bin/run_training.py)"]:::deploy
        BinTest["Testing Script (bin/run_testing.py)"]:::deploy
        Scripts["Extra Scripts (scripts)"]:::deploy
        CfgDeploy["Deploy Options (config/deployment_options.yaml)"]:::deploy
        CfgHyper["Hyperparameters (config/hyperparameters.yaml)"]:::deploy
        MLFlow["MLFlow Logging"]:::deploy
    end

    %% ROS Integration & Inference
    subgraph "ROS Integration & Inference"
        ROSUtils["ROS Integration (src/ros_utils)"]:::ros
        ROSNode["ROS Node (bin/run_rosnode.py)"]:::ros
    end

    %% Utility & Helper Functions
    subgraph "Utility Functions"
        Utility["Utility Functions (src/utility)"]:::utility
    end

    %% Environment & Dependencies
    subgraph "Environment & Dependencies"
        Conda["Conda Environment (conda/DeLORA-py3.9.yml)"]:::env
        Pip["Pip Requirements (pip/requirements.txt)"]:::env
        Docker["Docker Setup (docker/Dockerfile)"]:::env
    end

    %% Connections
    Datasets -->|"LiDARScans"| DataIngest
    CfgData -->|"SensorConfig"| DataIngest
    DataIngest -->|"PreprocessedData"| Preproc
    Preproc -->|"Features"| Model
    Model -->|"LossFeedback"| Loss
    Model -->|"TrainedModel"| Deploy
    Loss -->|"OptimizationSignals"| Deploy
    CfgDeploy --> Deploy
    CfgHyper --> Deploy
    Deploy -->|"StartTraining"| BinTrain
    Deploy -->|"StartTesting"| BinTest
    Deploy -->|"TriggerLogging"| MLFlow
    Deploy -->|"RunScripts"| Scripts
    Model -->|"PretrainedModel"| ROSUtils
    ROSUtils -->|"PublishResults"| ROSNode

    Utility ---|"Support"| DataIngest
    Utility ---|"Support"| Preproc
    Utility ---|"Support"| Model
    Utility ---|"Support"| Deploy

    Conda ---|"Runtime"| Deploy
    Pip ---|"Dependencies"| Deploy
    Docker ---|"Containerization"| Deploy

    %% Class Definitions
    classDef data fill:#ace7ef,stroke:#1f77b4,color:#000;
    classDef model fill:#c7f9cc,stroke:#2ca02c,color:#000;
    classDef deploy fill:#fff7b2,stroke:#ff7f0e,color:#000;
    classDef ros fill:#ffebb2,stroke:#d62728,color:#000;
    classDef utility fill:#d9b3ff,stroke:#9467bd,color:#000;
    classDef env fill:#e2e2e2,stroke:#7f7f7f,color:#000;

    %% Click Events
    click DataIngest "https://github.com/air-ufg/delora/tree/main/src/data"
    click Preproc "https://github.com/air-ufg/delora/tree/main/src/preprocessing"
    click Datasets "https://github.com/air-ufg/delora/tree/main/datasets"
    click CfgData "https://github.com/air-ufg/delora/blob/main/config/config_datasets.yaml"
    click Model "https://github.com/air-ufg/delora/tree/main/src/models"
    click Loss "https://github.com/air-ufg/delora/tree/main/src/losses"
    click Deploy "https://github.com/air-ufg/delora/tree/main/src/deploy"
    click BinTrain "https://github.com/air-ufg/delora/blob/main/bin/run_training.py"
    click BinTest "https://github.com/air-ufg/delora/blob/main/bin/run_testing.py"
    click Scripts "https://github.com/air-ufg/delora/tree/main/scripts"
    click CfgDeploy "https://github.com/air-ufg/delora/blob/main/config/deployment_options.yaml"
    click CfgHyper "https://github.com/air-ufg/delora/blob/main/config/hyperparameters.yaml"
    click ROSUtils "https://github.com/air-ufg/delora/tree/main/src/ros_utils"
    click ROSNode "https://github.com/air-ufg/delora/blob/main/bin/run_rosnode.py"
    click Utility "https://github.com/air-ufg/delora/tree/main/src/utility"
    click Conda "https://github.com/air-ufg/delora/blob/main/conda/DeLORA-py3.9.yml"
    click Pip "https://github.com/air-ufg/delora/blob/main/pip/requirements.txt"
    click Docker "https://github.com/air-ufg/delora/tree/main/docker/Dockerfile"
```
