# rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation architecture

> Architecture diagram of the GitHub repository rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation
- Repository: https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation

## Overview

1. **What this repo is**
   SurgiKneeVision is a research-oriented Python project for **knee surface capture and 3D registration** during total knee arthroplasty. The live pipeline uses **Microsoft Azure Kinect** for RGB/depth acquisition, **YOLO** for object detection, **SAM** for segmentation, and then evaluates the segmented knee depth/point cloud against registration methods.

2. **Top-level architecture split**
   The repository is split into two clear halves:
   - **Runtime inference scripts** that run on camera input and produce detections, masks, depth/3D overlays, and registration results.
   - **Offline experimentation/training artifacts** in notebooks and training folders for model development, evaluation, visualization, and dataset inspection.

3. **Primary live pipeline**
   The main data flow is:
   **Azure Kinect frame stream → YOLO knee detection → SAM mask refinement → segmented depth extraction → 3D visualization / registration**.
   The scripts expose toggles for showing the YOLO box, SAM mask, and 3D bounding-box overlays, which suggests the system is designed as an interactive live demo rather than a batch-only pipeline.

4. **Main runtime entry points**
   There are several focused CLI scripts, each wrapping a slightly different end-to-end flow:
   - `surgikneevision.py`: the full integrated application.
   - `yolo_inference_az.py`: YOLO-only inference on Azure Kinect input.
   - `o3d_sam_inference_az.py`: segmentation plus 3D/point-cloud-focused output.
   - `yolosam_inference_az.py`: combined YOLO + SAM inference with visualization options.
   These form the operational boundary of the repo and are the scripts most likely to appear in a deployment or demo diagram.

5. **Detection and segmentation subsystem**
   YOLO and SAM are treated as separate model stages:
   - YOLO proposes the knee region via bounding boxes.
   - SAM uses those boxes as prompts to generate a more precise mask.
   This separation is important architecturally because detection narrows the search space and SAM turns a coarse localization into a pixel-level region that can be projected into depth/3D.

6. **3D registration subsystem**
   The repo contains multiple registration approaches, indicating a benchmarking-style architecture:
   - `ICP_reg.py`
   - `RANSAC_reg.py`
   - `global_reg.py`
   - `Robust_ICP.py`
   - `DCP_model.py`
   - `DCP_reg.ipynb`, `DCP_ICP.ipynb`
   This shows a comparison between **classical point-cloud registration** and **deep learning-based registration**. The segmented knee point cloud is the common input boundary for these algorithms.

7. **Offline research and analysis layer**
   Many notebooks are used for exploratory work rather than production runtime:
   - `dataset_inspection.ipynb`
   - `plytoxyz.ipynb`
   - `CT_to_pc.ipynb`
   - `SICP_vis.ipynb`
   - `SAM_*training*.ipynb`, `SAM_*testing*.ipynb`
   These notebooks support dataset QA, format conversion, registration experiments, and model training/validation. They are a separate analysis layer, not part of the real-time inference path.

8. **Model training and artifact management**
   YOLO training is isolated under `yolo_train/`, including:
   - `train_yolo.py`
   - `runs/detect/train/...`
   - saved weights such as `best.pt` and `last.pt`
   This indicates the repo keeps both training code and experiment outputs. The README also says SAM weights are loaded from an external Google Drive folder, so the runtime depends on local model artifacts copied into a `models/` directory.

9. **Shared utilities**
   `utils.py` likely contains helper functions shared across inference and registration scripts: image/point-cloud conversion, drawing, coordinate handling, or model-loading helpers. In architecture terms, it acts as the glue layer between camera input, model inference, and 3D processing.

10. **Data and external dependencies**
   The project is centered on **proprietary research data** and includes a sample point cloud file (`scan_knee.ply`) plus model outputs. It also depends on external model files that are not committed in the repo. The presence of separate `requirements_pip.txt` and `requirements_conda.txt` suggests environment setup is handled explicitly, which matters because camera SDKs, YOLO/SAM, and 3D processing stacks often require careful version alignment.

11. **How to draw the architecture**
   A good diagram would show:
   - **Input layer:** Azure Kinect RGB + depth stream
   - **Perception layer:** YOLO detector → SAM segmenter
   - **Geometry layer:** masked depth → point cloud / 3D overlay
   - **Registration layer:** ICP / RANSAC / global / DCP variants
   - **Offline layer:** training notebooks, evaluation notebooks, dataset conversion, and saved weights
   - **Artifacts:** trained YOLO weights, SAM weights, and research point clouds
   The key boundary is between **live inference** and **offline experimentation/training**.

Main entry points: [`utils.py`](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/utils.py) (Utils), [`surgikneevision.py`](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/surgikneevision.py) (Main app), [`yolosam_inference_az.py`](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/yolosam_inference_az.py) (YOLO + SAM), [`o3d_sam_inference_az.py`](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/o3d_sam_inference_az.py) (3D SAM)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Live inference"]
  node_azure_kinect[("Azure Kinect<br/>sensor stream")]
  node_surgikneevision["Main app<br/>live pipeline<br/>[surgikneevision.py]"]
  node_yolo_inference_az["YOLO infer<br/>live detector"]
  node_yolosam_inference_az["YOLO + SAM<br/>live perception"]
  node_o3d_sam_inference_az["3D SAM<br/>point-cloud demo"]
  node_yolo_detector{{"YOLO<br/>object detector"}}
  node_sam_segmenter{{"SAM<br/>mask segmenter"}}
  node_seg_depth["Masked depth<br/>geometry extraction"]
  node_visual_3d(("3D view<br/>visualization"))
end

subgraph group_registration["Registration methods"]
  node_icp_reg["ICP<br/>classical registration<br/>[ICP_reg.py]"]
  node_ransac_reg["RANSAC<br/>classical registration<br/>[RANSAC_reg.py]"]
  node_global_reg["Global reg<br/>classical registration<br/>[global_reg.py]"]
  node_robust_icp["Robust ICP<br/>classical registration<br/>[Robust_ICP.py]"]
  node_dcp_model{{"DCP<br/>deep registration<br/>[DCP_model.py]"}}
end

subgraph group_research["Offline research"]
  node_registration_notebooks["Reg notebooks<br/>experiments"]
  node_dataset_notebooks["Data notebooks<br/>analysis"]
  node_sam_notebooks["SAM notebooks<br/>training/testing"]
end

subgraph group_training["Model artifacts"]
  node_yolo_train_script["Train YOLO<br/>training script<br/>[train_yolo.py]"]
  node_yolo_weights[("YOLO weights<br/>trained artifacts<br/>[best.pt]")]
end

node_scan_knee["Scan knee<br/>sample point cloud<br/>[scan_knee.ply]"]
node_utils{{"Utils<br/>shared helpers<br/>[utils.py]"}}

node_azure_kinect -->|"frames"| node_surgikneevision
node_azure_kinect -->|"frames"| node_yolo_inference_az
node_azure_kinect -->|"frames"| node_yolosam_inference_az
node_azure_kinect -->|"frames"| node_o3d_sam_inference_az
node_surgikneevision -->|"detect"| node_yolo_detector
node_yolo_inference_az -->|"detect"| node_yolo_detector
node_yolosam_inference_az -->|"detect"| node_yolo_detector
node_yolo_detector -->|"prompt"| node_sam_segmenter
node_yolosam_inference_az -->|"segment"| node_sam_segmenter
node_o3d_sam_inference_az -->|"segment"| node_sam_segmenter
node_sam_segmenter -->|"mask depth"| node_seg_depth
node_seg_depth -->|"render"| node_visual_3d
node_surgikneevision -->|"overlay"| node_visual_3d
node_seg_depth -->|"register"| node_icp_reg
node_seg_depth -->|"register"| node_ransac_reg
node_seg_depth -->|"register"| node_global_reg
node_seg_depth -->|"register"| node_robust_icp
node_seg_depth -->|"register"| node_dcp_model
node_registration_notebooks -->|"study"| node_icp_reg
node_registration_notebooks -->|"study"| node_ransac_reg
node_registration_notebooks -->|"study"| node_global_reg
node_registration_notebooks -->|"study"| node_robust_icp
node_registration_notebooks -->|"study"| node_dcp_model
node_dataset_notebooks -->|"inspect"| node_scan_knee
node_sam_notebooks -->|"train/test"| node_sam_segmenter
node_yolo_train_script -->|"produce"| node_yolo_weights
node_yolo_weights -->|"load"| node_yolo_detector
node_utils -->|"shared"| node_surgikneevision
node_utils -->|"shared"| node_yolosam_inference_az
node_utils -->|"shared"| node_o3d_sam_inference_az
node_utils -->|"shared"| node_icp_reg

click node_surgikneevision "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/surgikneevision.py"
click node_yolo_inference_az "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/yolo_inference_az.py"
click node_yolosam_inference_az "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/yolosam_inference_az.py"
click node_o3d_sam_inference_az "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/o3d_sam_inference_az.py"
click node_icp_reg "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/ICP_reg.py"
click node_ransac_reg "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/RANSAC_reg.py"
click node_global_reg "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/global_reg.py"
click node_robust_icp "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/Robust_ICP.py"
click node_dcp_model "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/DCP_model.py"
click node_yolo_train_script "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/yolo_train/train_yolo.py"
click node_yolo_weights "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/yolo_train/runs/detect/train/weights/best.pt"
click node_scan_knee "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/scan_knee.ply"
click node_utils "https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_azure_kinect,node_surgikneevision,node_yolo_inference_az,node_yolosam_inference_az,node_o3d_sam_inference_az,node_yolo_detector,node_sam_segmenter,node_seg_depth,node_visual_3d toneBlue
class node_icp_reg,node_ransac_reg,node_global_reg,node_robust_icp,node_dcp_model toneAmber
class node_registration_notebooks,node_dataset_notebooks,node_sam_notebooks toneMint
class node_yolo_train_script,node_yolo_weights toneRose
class node_scan_knee,node_utils toneNeutral
```

## Components

### Live inference

Interactive Azure Kinect pipelines for detection, segmentation, depth extraction, and visualization.

- **Azure Kinect**: sensor stream · RGB and depth input source for live capture.
- [Main app](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/surgikneevision.py): live pipeline · Integrated end-to-end knee capture, inference, and 3D registration demo. · `surgikneevision.py`
- [YOLO infer](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/yolo_inference_az.py): live detector · YOLO-only Azure Kinect inference entry point. · `yolo_inference_az.py`
- [YOLO + SAM](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/yolosam_inference_az.py): live perception · Combined detection and segmentation pipeline with visualization toggles. · `yolosam_inference_az.py`
- [3D SAM](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/o3d_sam_inference_az.py): point-cloud demo · Segmentation-focused live pipeline emphasizing Open3D and 3D output. · `o3d_sam_inference_az.py`
- **YOLO**: object detector · Finds the knee region with bounding boxes.
- **SAM**: mask segmenter · Refines the YOLO box into a pixel-accurate knee mask.
- **Masked depth**: geometry extraction · Projects the segmentation onto depth data to isolate knee surface points.
- **3D view**: visualization · Overlay and point-cloud visualization for live inspection.

### Registration methods

Classical and deep registration variants used to align segmented knee point clouds.

- [ICP](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/ICP_reg.py): classical registration · Iterative closest point registration baseline. · `ICP_reg.py`
- [RANSAC](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/RANSAC_reg.py): classical registration · Robust feature-based registration approach. · `RANSAC_reg.py`
- [Global reg](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/global_reg.py): classical registration · Global alignment method for point clouds. · `global_reg.py`
- [Robust ICP](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/Robust_ICP.py): classical registration · ICP variant designed for noisy or partial scans. · `Robust_ICP.py`
- [DCP](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/DCP_model.py): deep registration · Deep Closest Point model for learned point-cloud registration. · `DCP_model.py`

### Offline research

Notebooks for dataset inspection, conversion, training, testing, and visualization.

- **Reg notebooks**: experiments · Notebook-based registration experiments and comparisons.
- **Data notebooks**: analysis · Dataset inspection and format conversion notebooks.
- **SAM notebooks**: training/testing · Notebook workflows for SAM training and prompt-based evaluation.

### Model artifacts

YOLO training code and saved experiment outputs; external SAM weights are referenced at runtime.

- [Train YOLO](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/yolo_train/train_yolo.py): training script · YOLO training entry point for the custom knee dataset. · `yolo_train/train_yolo.py`
- [YOLO weights](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/yolo_train/runs/detect/train/weights/best.pt): trained artifacts · Saved detector checkpoints from training runs. · `yolo_train/runs/detect/train/weights/best.pt`

### Other components

- [Scan knee](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/scan_knee.ply): sample point cloud · Example knee surface point-cloud artifact used for research and validation. · `scan_knee.ply`
- [Utils](https://github.com/rafat2205/surgikneevision-3d-reconstruction-and-registration-of-knee-for-surgical-navigation/blob/main/utils.py): shared helpers · Common helper layer used by inference and registration scripts. · `utils.py`

## Connections

- Azure Kinect → Main app: frames · Live RGB/depth stream enters the integrated demo.
- Azure Kinect → YOLO infer: frames · Camera input feeds the detector-only script.
- Azure Kinect → YOLO + SAM: frames · Camera input feeds the combined perception pipeline.
- Azure Kinect → 3D SAM: frames · Camera input feeds the 3D-focused segmentation pipeline.
- Main app → YOLO: detect · Main app invokes YOLO for localization.
- YOLO infer → YOLO: detect · Detector-only runtime stage.
- YOLO + SAM → YOLO: detect · YOLO proposes the knee bounding box.
- YOLO → SAM: prompt · Detected box is used to prompt SAM.
- YOLO + SAM → SAM: segment · Combined pipeline runs mask refinement.
- 3D SAM → SAM: segment · 3D pipeline uses SAM for knee isolation.
- SAM → Masked depth: mask depth · Mask is projected into depth space to extract knee surface points.
- Masked depth → 3D view: render · Segmented geometry is visualized as 3D output.
- Main app → 3D view: overlay · Integrated demo shows live overlays and 3D bounding boxes.
- Masked depth → ICP: register · Masked point cloud is aligned with classical registration baselines.
- Masked depth → RANSAC: register · Masked point cloud is aligned with RANSAC-based registration.
- Masked depth → Global reg: register · Masked point cloud is aligned with global registration.
- Masked depth → Robust ICP: register · Masked point cloud is aligned with a robust ICP variant.
- Masked depth → DCP: register · Masked point cloud is also evaluated with learned registration.
- Reg notebooks → ICP: study · Notebooks compare registration approaches.
- Reg notebooks → RANSAC: study · Notebooks compare registration approaches.
- Reg notebooks → Global reg: study · Notebooks compare registration approaches.
- Reg notebooks → Robust ICP: study · Notebooks compare registration approaches.
- Reg notebooks → DCP: study · Notebooks compare registration approaches.
- Data notebooks → Scan knee: inspect · Dataset and point-cloud notebooks use sample scans and conversions.
- SAM notebooks → SAM: train/test · SAM notebook workflows develop and validate segmentation behavior.
- Train YOLO → YOLO weights: produce · Training script outputs detector checkpoints.
- YOLO weights → YOLO: load · Live inference consumes trained YOLO weights.
- Utils → Main app: shared · Common helper functions support the integrated app.
- Utils → YOLO + SAM: shared · Common helper functions support the combined pipeline.
- Utils → 3D SAM: shared · Common helper functions support the 3D pipeline.
- Utils → ICP: shared · Common helper functions support registration scripts.