# ihbkaiser/trackingtooltipslab architecture

> Architecture diagram of the GitHub repository ihbkaiser/trackingtooltipslab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ihbkaiser/trackingtooltipslab
- Repository: https://github.com/ihbkaiser/trackingtooltipslab

## Overview

Based on our analysis of the provided file structure and README, the project is an algorithmic research prototype aimed at improving instance segmentation to perform surgical instrument tip detection. It is a computer vision project that leverages deep learning models (including YOLO variants and custom network modules) to identify tool tips on surgical instruments from images and videos.

##### Main Components and Their Roles
- Data Acquisition and Preprocessing:
- “assets/” contains images and figures used for demo outputs.
- Although the ideal folder structure (as shown in the README) expects a “data/” directory with the Endovis15 dataset, your project’s file tree shows that data is expected to be externally organized. Emphasize an external data input source.
- Preprocessing might be handled either in notebooks (“notebook/train.ipynb” and “notebook/predict.ipynb”) or within scripts (e.g., “src/LoadEndovis15.py”, “src/Image.py”, “src/Mask.py”).

- Core Algorithm Modules:
- The “ihbnet/” directory provides the core segmentation components such as:
· data_loader.py – responsible for loading the data
· ihb_net.py – likely combines sub-modules such as “modules.py”, “graph_attn.py”, and “pe_layer.py” signifying the use of attention mechanisms and positional encoding.
- “src/Models/” contains specific model definitions (e.g., LabModel, YOLOModel, and Endovis15GT) that define the detection and segmentation architectures.

- Training, Inference, and Evaluation:
- “src/Eval.py”, “src/Mobile_train.py”, and similar scripts serve as the driver code for both evaluating the model and training (including Mobile/YOLO based configurations).
- The “notebook/” scripts serve as interactive demos for training and prediction.

- External Model Integration and Configuration:
- The extensive “src/mobilev4_ultralytics/” directory represents an integration of an external deep learning library (i.e., Ultralytics YOLO codebase) for object detection. This contains submodules covering datasets, configuration files (for models, trackers, etc.), engine (trainer, validator, predictor, etc.), and utilities.
- This component is responsible for the implementation and customization of various YOLO-based models (from YOLOv3 all the way to YOLOv9 variants) to support the detection pipeline.

##### Relationships and Data Flow Among Components
- Data Flow:
- Input: Raw images/videos from the Endovis15 dataset (as specified in the README).
- Preprocessing: Images are loaded and preprocessed via the corresponding modules (data_loader, Image, Mask).
- Core Processing: The core algorithm (ihbnet) processes the data using segmentation and tip detection routines, with help from attention and positional encoding layers.
- Model Integration: In parallel or integrated, the YOLO-based modules from “src/mobilev4_ultralytics/” (and possibly “src/Models/YOLOModel.py”) perform detection tasks. These models might also integrate tracking routines available under “src/mobilev4_ultralytics/ultralytics/models/yolo/” and tracker modules.
- Output: Evaluated using “src/Eval.py” (which likely compares detected tool tip positions with ground truth), and demo outputs created in the “assets/” folder are used for visualization.

- Interactions:
- The primary algorithm in ihb_net utilizes outputs from the supporting modules (e.g., feature extraction, graph attention). Their results feed into the evaluation stage.
- The YOLO and mobile modules communicate through a consistent API defined in the “Models/” folder and may update parameters during training (guided by scripts like “Mobile_train.py”).

##### Architectural Patterns and Key Technologies
- Modular Architecture: The project is split into clear modules – core algorithm (ihbnet), model definitions (src/Models), and an external detection/tracking framework (mobilev4_ultralytics). This supports separation of concerns.
- Deep Learning & Computer Vision: Leveraging segmentation and attention-based methods alongside YOLO detection indicates a layered deep learning architecture.
- Extensible Pipeline: The existence of notebooks for training/predicting and multiple model configurations indicates design for experimentation and rapid prototyping.
- Integration of External Libraries: The ultralytics folder suggests utilization of third-party libraries to manage model configurations, training loops, evaluation, and inference processes.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Data Source
  DS["Endovis15 Dataset (External)"]:::ds

  %% Data Acquisition & Preprocessing
  subgraph "Data Acquisition & Preprocessing"
    L15["LoadEndovis15.py"]:::preproc
    IMG["Image.py"]:::preproc
    MASK["Mask.py"]:::preproc
    TrainNB["Train Notebook"]:::preproc
    PredictNB["Predict Notebook"]:::preproc
  end

  %% Core Segmentation (ihbnet)
  subgraph "Core Segmentation (ihbnet)"
    IHB["ihb_net.py"]:::core
    DATA_LOADER["data_loader.py"]:::core
    MODULES["modules.py"]:::core
    GRAPH_ATTN["graph_attn.py"]:::core
    PE_LAYER["pe_layer.py"]:::core
  end

  %% Model Definitions
  subgraph "Model Definitions"
    ENDOVIS["Endovis15GT.py"]:::model
    LAB["LabModel.py"]:::model
    YOLO_MODEL["YOLOModel.py"]:::model
  end

  %% External YOLO Framework Integration
  subgraph "External YOLO Framework"
    MOBILEV4["Mobilev4 Ultralytics"]:::external
    ULTRA["Ultralytics Modules"]:::external
  end

  %% Evaluation & Visualization
  subgraph "Evaluation & Visualization"
    EVAL["Eval.py"]:::eval
    MOBILE_TRAIN["Mobile_train.py"]:::eval
    VIDEO["Video.py"]:::eval
    ROI["ROI.py"]:::eval
    ASSETS["Assets Demo Outputs"]:::eval
  end

  %% Data Flow Connections
  DS --> L15
  DS --> TrainNB
  DS --> PredictNB

  L15 --> IMG
  IMG --> MASK

  %% Preprocessing flows into Core Segmentation and External YOLO
  MASK --> IHB
  TrainNB --> IHB
  PredictNB --> IHB
  MASK --> MOBILEV4

  %% Core Segmentation internal flow
  IHB --> DATA_LOADER
  DATA_LOADER --> MODULES
  MODULES --> GRAPH_ATTN
  GRAPH_ATTN --> PE_LAYER

  %% Integration with Model Definitions
  PE_LAYER --> ENDOVIS

  %% Chain in Model Definitions
  ENDOVIS --> LAB
  LAB --> YOLO_MODEL

  %% External YOLO internal flow
  MOBILEV4 --> ULTRA

  %% Convergence into Evaluation & Visualization
  ENDOVIS --> EVAL
  YOLO_MODEL --> EVAL
  ULTRA --> EVAL

  %% Evaluation processing chain
  EVAL --> MOBILE_TRAIN
  MOBILE_TRAIN --> VIDEO
  VIDEO --> ROI
  ROI --> ASSETS

  %% Click Events for Data Acquisition & Preprocessing
  click L15 "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/src/LoadEndovis15.py"
  click IMG "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/src/Image.py"
  click MASK "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/src/Mask.py"
  click TrainNB "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/notebook/train.ipynb"
  click PredictNB "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/notebook/predict.ipynb"

  %% Click Events for Core Segmentation
  click IHB "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/ihbnet/ihb_net.py"
  click DATA_LOADER "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/ihbnet/data_loader.py"
  click MODULES "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/ihbnet/modules.py"
  click GRAPH_ATTN "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/ihbnet/graph_attn.py"
  click PE_LAYER "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/ihbnet/pe_layer.py"

  %% Click Events for Model Definitions
  click ENDOVIS "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/src/Models/Endovis15GT.py"
  click LAB "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/src/Models/LabModel.py"
  click YOLO_MODEL "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/src/Models/YOLOModel.py"

  %% Click Events for External YOLO Framework
  click MOBILEV4 "https://github.com/ihbkaiser/trackingtooltipslab/tree/main/src/mobilev4_ultralytics"
  click ULTRA "https://github.com/ihbkaiser/trackingtooltipslab/tree/main/src/mobilev4_ultralytics/ultralytics"

  %% Click Events for Evaluation & Visualization
  click EVAL "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/src/Eval.py"
  click MOBILE_TRAIN "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/src/Mobile_train.py"
  click VIDEO "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/src/Video.py"
  click ROI "https://github.com/ihbkaiser/trackingtooltipslab/blob/main/src/ROI.py"

  %% Styles
  classDef ds fill:#f6d55c,stroke:#333,stroke-width:2px,color:#000
  classDef preproc fill:#AED6F1,stroke:#333,stroke-width:2px,color:#000
  classDef core fill:#F9E79F,stroke:#333,stroke-width:2px,color:#000
  classDef model fill:#D7BDE2,stroke:#333,stroke-width:2px,color:#000
  classDef external fill:#A9DFBF,stroke:#333,stroke-width:2px,color:#000
  classDef eval fill:#F5B7B1,stroke:#333,stroke-width:2px,color:#000
```
