# h-freax/thinkgrasp architecture

> Architecture diagram of the GitHub repository h-freax/thinkgrasp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/h-freax/thinkgrasp
- Repository: https://github.com/h-freax/thinkgrasp

## Overview

##### Main Components
- Input & Data Sources:
– Sensor Input Module: This includes camera inputs (RGB and depth) and text goal descriptions used in real-world tests (as seen in the API payload for realarm.py).
– Simulation Data: Test cases (testing_cases, heavy_unseen, assets of objects/URDF models) and external datasets from assets.
- Vision-Language Pipeline:
– Vision Processing: The “VLP” directory contains modules for GLIP, segment-anything, and even VLPart. These modules leverage maskrcnn_benchmark and segmentation neural networks to extract object and part features.
– Language Processing: Language backbones are found in the glip directory and in “langsam” (which contains language and actor modules). BERT models and tokenizers are used (as shown by the bert_model.py files) to process language inputs.
- Grasp Inference Module:
– GraspNet Models: In the “models” folder you have two variations (FGC_graspnet and graspnet) that include dataset handling, training scripts (train.py), inference modules, and optimized KNN/PointNet2 implementations.
– Grasp Detector: The file “grasp_detetor.py” also indicates part of the inference/control pipeline.
- Environment & Simulation:
– Simulation Engine: Files like simulation_main.py and environment_sim.py build a simulated environment (using PyBullet) to test negotiations with obstacles and clutter.
– ROS/URDF Assets: The assets folder provides URDFs and mesh files that are used for simulating robot hardware and the environment.
- Real-World Interface:
– Flask API Server: The realarm.py script runs a Flask application that exposes an API endpoint (/grasp_pose) for controlling or triggering the grasp process in real-world scenarios.
- Utility & Infrastructure Modules:
– Core Utilities: Files like constants.py, logger.py, engine.py, and utils.py support configuration, logging, and miscellaneous functions.
– Model Build & Setup: Setup scripts (setup.py, requirements.txt, langsam.txt, vlp_requirements.txt) handle package and dependency management for various subcomponents.
- External Dependencies:
– Hardware & Accelerator Support: PyTorch, CUDA (note CUDA installation instructions and GPU recommendations), PyBullet for simulation, and WandB for experiment tracking.
– Compilation & Custom Ops: The multiple build directories (e.g., for knn and pointnet2 inside models) indicate custom CUDA/C++ kernels integration.

##### Representing Relationships and Data Flow
a. Real-World Input (from sensors) that flows to the Real-World Interface (Flask server “realarm.py”).
b. Simulation Input (using pre-saved test cases and assets) that flows to the Simulation Engine (“simulation_main.py”).
– From these, create a central processing pipeline where sensor data or images enters the Vision-Language Pipeline. In this zone:
- Visual Data is processed by pre-trained vision networks (GLIP, segment-anything from VLP/glip/ and VLP/segment-anything) to extract features, semantic segmentation, and object proposals.
- Language Input (goal/task text) is encoded using BERT/language backbones (files in glip and langsam).
- The two modalities are fused (via modules in “maskrcnn_benchmark” and “vlpart”) to generate context-aware representations.
– The fused representation is sent into the Grasp Inference Module, which comprises of:
- The FGC_graspnet and graspnet models (including their dataset, training, and inference pipelines).
- Custom KNN and PointNet2 operations for geometric reasoning.
– The output (grasp pose predictions) is then processed by a control/decision block that routes commands to:
- The Simulator (for testing and visualization in simulation_main.py).
- The real robot controller (invoked via the Flask API for realarm.py) to execute physical grasps.
– Add supporting utility and logging interactions that monitor each stage (often connected to WandB for experiment tracking and logging events).

##### Architectural Patterns & Design Principles
– Modular & Layered Architecture: Each major functionality (vision-language processing, grasp inference, simulation/real-world execution) is isolated into its own directory and module.
– Separation of Concerns: The project cleanly separates simulation versus real-world pipelines and decouples core inference logic from environment integration.
– Hardware-Accelerated Processing: Emphasize use of CUDA-enabled modules, custom C++ extensions, and multi-GPU support (noting the two GTX 3090 recommendation).
– Extensibility: The code structure (e.g., “setup.py”, multiple requirements files) supports rapid prototyping and addition of new modules.

##### Instructing the Principal Software Engineer
– Map out the Vision-Language Pipeline with its two branches (visual and language) that converge in a fusion module.
– Next, illustrate the Grasp Inference Module block showing the integration of FGC_graspnet, graspnet, and supporting KNN/PointNet2 custom kernels.
– Finally, list external dependencies (PyTorch, CUDA, PyBullet, Flask) and utility components (logger, configuration, experiment tracking via WandB) as ancillary elements that interface with the main processing pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input & Data Sources
    subgraph "Input & Data Sources"
        sensor["Sensor Input Module (cameras)"]:::input
        subgraph "Simulation Data"
            tc["Test Cases"]:::input
            hu["Heavy Unseen"]:::input
            aset["Assets"]:::input
        end
    end

    %% Vision-Language Pipeline
    subgraph "Vision-Language Pipeline"
        %% Vision Processing
        subgraph "Vision Processing"
            glip["GLIP (VLP/glip, bert_model.py)"]:::process
            seg["Segment-Anything (VLP/segment-anything)"]:::process
            vlp_module["VLPart (VLP/vlpart)"]:::process
        end
        %% Language Processing
        subgraph "Language Processing"
            language_proc["Language Processing (langsam)"]:::process
        end
        fusion["Feature Fusion"]:::process
    end

    %% Grasp Inference Module
    subgraph "Grasp Inference Module"
        fgc["FGC_graspnet"]:::process
        graspnet["graspnet"]:::process
        detector["Grasp Detector (grasp_detetor.py)"]:::process
        fusion_out["Grasp Pose Estimation"]:::process
    end

    %% Control & Actuation
    control["Control & Actuation"]:::control
    simEng["Simulation Engine (simulation_main.py, environment_sim.py)"]:::control
    flaskAPI["Flask API Server (realarm.py)"]:::control

    %% Utility & Infrastructure Modules
    subgraph "Utility & Infrastructure"
        util1["Constants (constants.py)"]:::utility
        util2["Engine (engine.py)"]:::utility
        util3["Logger (logger.py)"]:::utility
        util4["Utils (utils.py)"]:::utility
        util5["Setup (setup.py)"]:::utility
        util6["Requirements (requirements.txt)"]:::utility
        util7["Langsam Requirements (langsam.txt)"]:::utility
        util8["VLP Requirements (vlp_requirements.txt)"]:::utility
    end

    %% External Dependencies
    ext["External Dependencies (PyTorch, CUDA, PyBullet, Flask, WandB)"]:::external

    %% Data Flow Connections
    sensor -- "ImageInput" --> glip
    sensor -- "TextInput" --> language_proc
    tc -- "SimData" --> glip
    hu -- "SimData" --> glip
    aset -- "SimData" --> glip

    glip -- "ExtractedVisual" --> fusion
    seg -- "Segmentation" --> fusion
    vlp_module -- "ObjectParts" --> fusion
    language_proc -- "LinguisticFeatures" --> fusion

    fusion -- "FusedRepresentation" --> fgc
    fusion -- "FusedRepresentation" --> graspnet
    fusion -- "FusedRepresentation" --> detector

    fgc -- "PoseEst" --> fusion_out
    graspnet -- "PoseEst" --> fusion_out
    detector -- "PoseEst" --> fusion_out

    fusion_out -- "GraspPose" --> control

    control -- "SimulatedCommand" --> simEng
    control -- "RealWorldCommand" --> flaskAPI

    %% Utility & External Dependencies Connections
    ext -- "Support" --> fusion

    %% Classes / Styling
    classDef input fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef process fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef control fill:#FFDAB9,stroke:#000,stroke-width:2px;
    classDef utility fill:#D3D3D3,stroke:#000,stroke-width:2px;
    classDef external fill:#D8BFD8,stroke:#000,stroke-width:2px;

    %% Click Events
    click sensor "https://github.com/h-freax/thinkgrasp/blob/main/cameras.py"
    click tc "https://github.com/h-freax/thinkgrasp/tree/main/testing_cases"
    click hu "https://github.com/h-freax/thinkgrasp/tree/main/heavy_unseen"
    click aset "https://github.com/h-freax/thinkgrasp/tree/main/assets"
    click glip "https://github.com/h-freax/thinkgrasp/tree/main/VLP/glip"
    click seg "https://github.com/h-freax/thinkgrasp/tree/main/VLP/segment-anything"
    click vlp_module "https://github.com/h-freax/thinkgrasp/tree/main/VLP/vlpart"
    click language_proc "https://github.com/h-freax/thinkgrasp/tree/main/langsam"
    click fgc "https://github.com/h-freax/thinkgrasp/tree/main/models/FGC_graspnet"
    click graspnet "https://github.com/h-freax/thinkgrasp/tree/main/models/graspnet"
    click detector "https://github.com/h-freax/thinkgrasp/blob/main/grasp_detetor.py"
    click simEng "https://github.com/h-freax/thinkgrasp/blob/main/simulation_main.py"
    click flaskAPI "https://github.com/h-freax/thinkgrasp/blob/main/realarm.py"
    click util1 "https://github.com/h-freax/thinkgrasp/blob/main/constants.py"
    click util2 "https://github.com/h-freax/thinkgrasp/blob/main/engine.py"
    click util3 "https://github.com/h-freax/thinkgrasp/blob/main/logger.py"
    click util4 "https://github.com/h-freax/thinkgrasp/blob/main/utils.py"
    click util5 "https://github.com/h-freax/thinkgrasp/blob/main/setup.py"
    click util6 "https://github.com/h-freax/thinkgrasp/blob/main/requirements.txt"
    click util7 "https://github.com/h-freax/thinkgrasp/blob/main/langsam.txt"
    click util8 "https://github.com/h-freax/thinkgrasp/blob/main/vlp_requirements.txt"
```
