# skumra/robotic-grasping architecture

> Architecture diagram of the GitHub repository skumra/robotic-grasping, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skumra/robotic-grasping
- Repository: https://github.com/skumra/robotic-grasping

## Overview

##### Project Type and Purpose
- This is a full-stack research prototype for antipodal robotic grasping. It combines a deep learning training pipeline (GR-ConvNet models), data preprocessing utilities, real-time inference, and hardware interface components for camera calibration and robot execution.

##### File Structure Analysis
- Top-level scripts:
- train_network.py, evaluate.py – model training and evaluation
- run_*.py – task runners (calibration, grasp generation, offline, realtime)
- inference/ – model definitions (grconvnet, grconvnet2–4), grasp generator, post-processing
- hardware/ – camera calibration, device wrappers, camera interface
- utils/ – data loaders (Cornell, Jacquard), dataset processing (depth generation, evaluation), visualization, timing
- trained-models/ – model artifact store
- requirements.txt, config (_config.yml), cleanup scripts

##### README Insights
- Core architecture: Generative Residual Convolutional Neural Network for predicting antipodal grasps
- Supports two datasets (Cornell, Jacquard) with preprocessing steps
- Real-time inference using pyrealsense2 camera feed or offline evaluation
- ROS integration via separate Baxter PnP repository

##### Main Components
a. Data Layer
- Raw datasets (Cornell, Jacquard)
- Preprocessing modules (utils.dataset_processing: generate depths, evaluation)
b. Training Pipeline
- train_network.py
- inference.models (GR-ConvNet variants)
- TensorBoardX for logging
- Model artifacts (trained-models/)
c. Inference Pipeline
- run_grasp_generator.py, run_offline.py, run_realtime.py
- inference.grasp_generator (loads model, runs forward pass)
- inference.post_process (non-max suppression, grasp filtering)
d. Hardware Interface
- hardware.camera (pyrealsense2 wrapper)
- hardware.calibrate_camera (intrinsic/extrinsic calibration)
- hardware.device (robot or simulator interface)
e. Visualization & Utilities
- utils.visualisation (gridshow, plotting)
- utils.timeit, utils.data (camera_data, grasp_data)

##### Component Relationships and Data Flow
- Data Layer → Training Pipeline: raw images → depth generation → batch loader → training script → saved model
- Trained Model → Inference Pipeline: model file → grasp_generator → grasp proposals
- Inference Pipeline ↔ Hardware Interface: camera feed → inference → device commands
- Visualization modules tap into both training (TensorBoard plots) and inference (gridshow) for diagnostics

##### Architectural Patterns and Design Principles
- Modular separation: data utilities, training, inference, hardware drivers
- Pipeline orchestration: clear data → model → hardware flow
- Extensibility: swapping model variants (grconvnet2,3,4) via common interface
- Script-based entry points for tasks (run_*.py)

##### Technologies and Frameworks
- Python 3.6+, PyTorch, TorchVision, TensorBoardX
- OpenCV, scikit-image, imageio for preprocessing
- pyrealsense2 for camera interface
- Optional ROS integration for robot control

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        direction TB
        Datasets["Datasets"]:::storage
        DL1["Cornell Dataset Loader"]:::data
        DL2["Jacquard Dataset Loader"]:::data
        CData["Camera Data Utilities"]:::data
        GData["Grasp Annotation Utilities"]:::data
        Pre1["Cornell Depth Generation"]:::data
        Pre2["Dataset Evaluation Metrics"]:::data
        Pre3["Generic Grasp Processing"]:::data
        Pre4["Image Utilities"]:::data
    end

    subgraph "Training Pipeline"
        direction TB
        Train["train_network.py"]:::train
        ModelsIF["grasp_model.py"]:::train
        Model1["grconvnet.py"]:::train
        Model2["grconvnet2.py"]:::train
        Model3["grconvnet3.py"]:::train
        Model4["grconvnet4.py"]:::train
        ModelStore["Model Artifacts\n(trained-models/)"]:::storage
    end

    subgraph "Inference Pipeline"
        direction TB
        RunGen["run_grasp_generator.py"]:::infer
        RunOff["run_offline.py"]:::infer
        RunRT["run_realtime.py"]:::infer
        Generator["Grasp Generator\n(inference/grasp_generator.py)"]:::infer
        Post["Post-processing\n(inference/post_process.py)"]:::infer
    end

    subgraph "Hardware Interface"
        direction TB
        Calib["run_calibration.py"]:::hw
        CalibMod["Camera Calibration\n(hardware/calibrate_camera.py)"]:::hw
        Camera["Camera Wrapper\n(hardware/camera.py)"]:::hw
        Device["Robot/Device Abstraction\n(hardware/device.py)"]:::hw
    end

    subgraph "Visualization & Utilities"
        direction TB
        Grid["Grid Display\n(utils/visualisation/gridshow.py)"]:::util
        Plot["Plotting Utilities\n(utils/visualisation/plot.py)"]:::util
        Time["Timing Helpers\n(utils/timeit.py)"]:::util
    end

    %% Data Flow
    Datasets -->|RGB-D frames| Pre1
    Datasets -->|Raw images| Pre2
    Datasets -->|Annotations| Pre3
    Datasets -->|Images| Pre4
    Datasets -->|Camera streams| CData
    Datasets -->|Grasp labels| GData

    Pre1 -->|Depth images| Train
    Pre2 -->|Metrics| Train
    Pre3 -->|Processed grasps| Train
    Pre4 -->|Augmented data| Train

    Train -->|Tensor batches| ModelsIF
    ModelsIF --> Model1
    ModelsIF --> Model2
    ModelsIF --> Model3
    ModelsIF --> Model4
    Model1 --> ModelStore
    Model2 --> ModelStore
    Model3 --> ModelStore
    Model4 --> ModelStore

    ModelStore -->|Saved model| Generator
    Camera -->|Live frames| Generator
    Generator -->|Grasp proposals| Post
    Post -->|Filtered grasps| Device
    Calib --> Camera
    Calib --> CalibMod

    %% Entry Points
    Train --> Grid
    Train --> Plot
    Generator --> Grid
    Generator --> Time
    Train --> Time

    %% Click Events
    click DL1 "https://github.com/skumra/robotic-grasping/blob/master/utils/data/cornell_data.py"
    click DL2 "https://github.com/skumra/robotic-grasping/blob/master/utils/data/jacquard_data.py"
    click CData "https://github.com/skumra/robotic-grasping/blob/master/utils/data/camera_data.py"
    click GData "https://github.com/skumra/robotic-grasping/blob/master/utils/data/grasp_data.py"
    click Pre1 "https://github.com/skumra/robotic-grasping/blob/master/utils/dataset_processing/generate_cornell_depth.py"
    click Pre2 "https://github.com/skumra/robotic-grasping/blob/master/utils/dataset_processing/evaluation.py"
    click Pre3 "https://github.com/skumra/robotic-grasping/blob/master/utils/dataset_processing/grasp.py"
    click Pre4 "https://github.com/skumra/robotic-grasping/blob/master/utils/dataset_processing/image.py"
    click Train "https://github.com/skumra/robotic-grasping/blob/master/train_network.py"
    click ModelsIF "https://github.com/skumra/robotic-grasping/blob/master/inference/models/grasp_model.py"
    click Model1 "https://github.com/skumra/robotic-grasping/blob/master/inference/models/grconvnet.py"
    click Model2 "https://github.com/skumra/robotic-grasping/blob/master/inference/models/grconvnet2.py"
    click Model3 "https://github.com/skumra/robotic-grasping/blob/master/inference/models/grconvnet3.py"
    click Model4 "https://github.com/skumra/robotic-grasping/blob/master/inference/models/grconvnet4.py"
    click ModelStore "https://github.com/skumra/robotic-grasping/tree/master/trained-models/"
    click RunGen "https://github.com/skumra/robotic-grasping/blob/master/run_grasp_generator.py"
    click RunOff "https://github.com/skumra/robotic-grasping/blob/master/run_offline.py"
    click RunRT "https://github.com/skumra/robotic-grasping/blob/master/run_realtime.py"
    click Generator "https://github.com/skumra/robotic-grasping/blob/master/inference/grasp_generator.py"
    click Post "https://github.com/skumra/robotic-grasping/blob/master/inference/post_process.py"
    click Calib "https://github.com/skumra/robotic-grasping/blob/master/run_calibration.py"
    click CalibMod "https://github.com/skumra/robotic-grasping/blob/master/hardware/calibrate_camera.py"
    click Camera "https://github.com/skumra/robotic-grasping/blob/master/hardware/camera.py"
    click Device "https://github.com/skumra/robotic-grasping/blob/master/hardware/device.py"
    click Grid "https://github.com/skumra/robotic-grasping/blob/master/utils/visualisation/gridshow.py"
    click Plot "https://github.com/skumra/robotic-grasping/blob/master/utils/visualisation/plot.py"
    click Time "https://github.com/skumra/robotic-grasping/blob/master/utils/timeit.py"

    %% Styles
    classDef data fill:#CCE5FF,stroke:#333;
    classDef train fill:#C6ECC6,stroke:#333;
    classDef infer fill:#FFDEAD,stroke:#333;
    classDef hw fill:#DCC6FF,stroke:#333;
    classDef util fill:#E0E0E0,stroke:#333;
    classDef storage fill:#CCCCCC,stroke:#333,stroke-dasharray: 5 5;
    class Datasets,ModelStore storage
    classDef cylinder shape:cylinder;
    class Datasets,ModelStore cylinder
```
