# nepfaff/scalable-real2sim architecture

> Architecture diagram of the GitHub repository nepfaff/scalable-real2sim, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nepfaff/scalable-real2sim
- Repository: https://github.com/nepfaff/scalable-real2sim

## Overview

##### Main Components
- Pipeline Entrypoints:
- Data Collection (run_data_collection.py)
- Asset Generation (run_asset_generation.py)
- Data Processing Module: (scalable_real2sim/data_processing); includes routines for handling images, masks, COLMAP data, neural network inputs, etc.
- Submodules & External Components:
- BundleSDF (object tracking and reconstruction)
- Frosting (surface reconstruction using Gaussian Splatting)
- Neuralangelo (neural surface reconstruction)
- nerfstudio (NeRF-based processing)
- robot_payload_id (robot and payload identification)
- Segmentation: (scalable_real2sim/segmentation) for detect_object and segment_moving_object_data routines.
- Additional Scripts: (scripts directory) for post-processing and metric computations, such as geometric metrics and mask subtraction.

##### Relationships and Data Flow
- Data Collection Phase:
- Data is acquired via run_data_collection.py, which interacts with the robot setup (possibly invoking pickplace_data_collection and robot_payload_id) and writes data to a shared output directory.
- This phase involves capturing sensor or joint data as well as object and scene information.
- Pre-Processing & Processing:
- Data from the collection phase is then sent into data_processing routines, where segmentation (object detection, masks) and transformation (colmap, neuralangelo dataset preparation) are applied.
- Asset Generation Phase:
- run_asset_generation.py consumes the processed data and, possibly in a distributed or asynchronous manner, sends it through several reconstruction pipelines.
- Each submodule (BundleSDF, Frosting, neuralangelo, nerfstudio) works as a distinct process (often with its own virtual environment/setup) to perform high-end object reconstruction and texture mapping.
- External Tools & Dependencies:

##### Architectural Patterns and Technologies
- Modularity: The system is highly modular (different directories hinting at a microservices-like approach, even within a single repository) with clear separation between data collection and asset generation.
- Distributed Execution Possibility: The separation of the asset generation script from the data collection script implies that these can run in different environments (e.g., local robot versus cloud).
- Robotics and System Identification: Including the robot_payload_id module and associated scripts for trajectory optimization and joint data collection is crucial to understanding how the physical robot setup integrates with the digital pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Collection Pipeline
    subgraph "Data Collection Pipeline"
        A["Robot Data Collection (run_data_collection.py)"]:::core
        B["Robot Payload Identification"]:::core
        A -->|"sends_payload_data"| B
    end

    %% Data Processing Pipeline
    subgraph "Data Processing Pipeline"
        C["Data Processing Module (scalable_real2sim/data_processing)"]:::core
        D["Segmentation (scalable_real2sim/segmentation)"]:::utility
        K["COLMAP (External)"]:::external
        B -->|"data_for_processing"| C
        C -->|"apply_segmentation"| D
        C -->|"uses"| K
    end

    %% Asset Generation Pipeline
    subgraph "Asset Generation Pipeline"
        E["Asset Generation (run_asset_generation.py)"]:::core
        C -->|"processed_data"| E
        E -->|"to_reconstruct"| F["BundleSDF (scalable_real2sim/BundleSDF)"]:::submodule
        E -->|"to_reconstruct"| G["Frosting (scalable_real2sim/Frosting)"]:::submodule
        E -->|"to_reconstruct"| H["Neuralangelo (scalable_real2sim/neuralangelo)"]:::submodule
        E -->|"to_reconstruct"| I["nerfstudio (scalable_real2sim/data_processing/nerfstudio.py)"]:::submodule
    end

    %% Additional Utilities
    J["Post-Processing & Metrics (scripts)"]:::utility
    E -->|"post_processing"| J

    %% Click Events
    click A "https://github.com/nepfaff/scalable-real2sim/blob/main/run_data_collection.py"
    click B "https://github.com/nepfaff/scalable-real2sim/tree/main/scalable_real2sim/robot_payload_id"
    click C "https://github.com/nepfaff/scalable-real2sim/tree/main/scalable_real2sim/data_processing"
    click E "https://github.com/nepfaff/scalable-real2sim/blob/main/run_asset_generation.py"
    click F "https://github.com/nepfaff/scalable-real2sim/tree/main/scalable_real2sim/BundleSDF"
    click G "https://github.com/nepfaff/scalable-real2sim/tree/main/scalable_real2sim/Frosting"
    click G "https://github.com/nepfaff/scalable-real2sim/blob/main/scalable_real2sim/data_processing/frosting.py"
    click H "https://github.com/nepfaff/scalable-real2sim/tree/main/scalable_real2sim/neuralangelo"
    click H "https://github.com/nepfaff/scalable-real2sim/blob/main/scalable_real2sim/data_processing/neuralangelo.py"
    click I "https://github.com/nepfaff/scalable-real2sim/blob/main/scalable_real2sim/data_processing/nerfstudio.py"
    click D "https://github.com/nepfaff/scalable-real2sim/tree/main/scalable_real2sim/segmentation"
    click J "https://github.com/nepfaff/scalable-real2sim/tree/main/scripts"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef submodule fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef utility fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef external fill:#FFD700,stroke:#000,stroke-width:2px;
```
