# isl-org/open3d-ml architecture

> Architecture diagram of the GitHub repository isl-org/open3d-ml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/isl-org/open3d-ml
- Repository: https://github.com/isl-org/open3d-ml

## Overview

##### Project type and purpose
- This is an open‐source 3D machine-learning library and framework extension for Open3D.
- It provides generic utilities, dataset readers, config‐driven builders, framework adapters (TensorFlow and PyTorch), model implementations, training/inference pipelines, visualization tools, example scripts, and CLI automation.

##### File structure
- Top‐level directories:
– ml3d (core package)
– scripts (CLI and demos)
– examples (user-facing code samples)
– docs (manuals & tutorials)
– data (demo assets)
– tests, CI, GitHub workflows
- Inside ml3d:
– configs: YAML files for models, datasets, pipelines
– datasets: framework-agnostic dataset reader code
– metrics: evaluation modules (mAP, IoU)
– utils: core utilities (Config loader, registry, builder, logging)
– tf: TensorFlow-specific subpackage (dataloaders, models, modules, pipelines, utils)
– torch: PyTorch-specific subpackage (mirror structure of tf)
- Scripts folder:
– run_pipeline.py: unified CLI
– download_datasets: shell scripts
– demo & preprocessing scripts
- Build/config/deployment files: Makefile, setup.py, CI scripts, requirements-*.txt

##### Extract architecture insights from README
- Config-driven instantiation: utilities.Config + utils.get_module build datasets, models, pipelines
- Dual‐framework support via parallel subpackages (ml3d.tf vs ml3d.torch)
- Core library (ml3d.*) integrated into open3d.ml namespace
- Example & script layers use the same core API

##### Main components & interactions
1. Core Layer (blue):
– ml3d.utils (Config loader, registry, builder)
– ml3d.configs (YAML definitions)
– ml3d.datasets (generic readers)
– ml3d.metrics, ml3d.vis
2. Framework Adapters (green):
– ml3d.tf / ml3d.tf.dataloaders → wrap generic datasets into tf.data pipelines
– ml3d.tf.models / modules (losses, metrics, schedulers)
– ml3d.tf.pipelines (SemanticSegmentation, ObjectDetection)
– ml3d.torch (same structure for PyTorch)
3. User Interfaces (orange):
– Python API (import open3d.ml.torch as ml3d)
– CLI scripts (scripts/run_pipeline.py, demo scripts)
– Notebooks & examples
4. External Systems (gray):
– File system (configs, datasets on disk, downloaded weights)
– Remote storage (model-zoo on GCS)
– Open3D core C++ library
– TensorBoard (for logs & metrics)
1. User → CLI / Python API → Core Layer (Config loader reads YAML)
2. Core Layer → Builder → Instantiates Dataset / Model / Pipeline classes in framework adapter
3. Pipeline → Dataloader → Generic Dataset → Disk (point clouds, labels)
4. Pipeline → Model (forward/backward passes) → Modules (loss, scheduler)
5. Pipeline → Metrics → ml3d.metrics → logs to console / TensorBoard
6. Pipeline → ml3d.vis → Visualization in UI
7. Scripts → Remote storage download → model weights → Disk
c. Architectural patterns & design principles:
– Plugin/Registry pattern: dynamic module loading via registry in utils
– Config‐driven architecture: separates specification (YAML) from implementation
– Adapter pattern: uniform API over TensorFlow and PyTorch backends
– Modular layering: core utility layer, framework‐specific layer, interface layer
d. Key technologies & libraries:
– Python 3.x, Open3D core
– TensorFlow 2.x, PyTorch 2.0
– YAML (config files)
– Shell scripts for dataset download & preprocessing
– CI pipelines (GitHub Actions)
– TensorBoard for monitoring

##### Elements
– Orange clouds for user entrypoints

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interfaces Layer
    subgraph "User Interfaces" 
        direction TB
        CLI["CLI Scripts"]:::ui
        Examples["Example Scripts"]:::ui
        API["Python API"]:::ui
    end

    %% Core Layer
    subgraph "Core Layer"
        direction TB
        subgraph "Utilities"
            direction TB
            ConfigLoader["Config Loader"]:::core
            Registry["Registry"]:::core
            Builder["Builder"]:::core
            Log["Logger"]:::core
        end
        subgraph "Configurations"
            direction TB
            Configs["YAML Config Files"]:::core
        end
        subgraph "Datasets"
            direction TB
            BaseDS["Base Dataset"]:::core
            CustomDS["Custom Dataset"]:::core
            OtherDS["Other Datasets"]:::core
        end
        subgraph "Metrics"
            direction TB
            mAP["mAP Module"]:::core
        end
        subgraph "Visualization"
            direction TB
            Visualizer["Visualizer"]:::core
            BoundingBox["Bounding Box"]:::core
            ColorMap["Color Map"]:::core
            LabelLUT["Label LUT"]:::core
        end
    end

    %% Framework Adapters
    subgraph "TensorFlow Adapter"
        direction TB
        TFData["TF Dataloader"]:::fw
        TFModels["TF Models"]:::fw
        TFModules["TF Modules"]:::fw
        TFPipelines["TF Pipelines"]:::fw
        TFUtils["TF Utils"]:::fw
    end

    subgraph "PyTorch Adapter"
        direction TB
        TorchData["Torch Dataloader"]:::fw
        TorchModels["Torch Models"]:::fw
        TorchModules["Torch Modules"]:::fw
        TorchPipelines["Torch Pipelines"]:::fw
        TorchUtils["Torch Utils"]:::fw
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        FS["File System"]:::ext
        Remote["Remote Storage"]:::ext
        Open3D["Open3D Core Lib"]:::ext
        TB["TensorBoard"]:::ext
    end

    %% Relationships
    CLI -->|"invokes"| ConfigLoader
    API -->|"imports"| Builder
    ConfigLoader -->|"reads"| Configs
    Builder -->|"builds"| BaseDS
    Builder -->|"builds"| CustomDS
    Builder -->|"builds"| TFData
    Builder -->|"builds"| TorchData

    BaseDS -->|"reads from"| FS
    CustomDS -->|"reads from"| FS
    TFData -->|"wraps"| BaseDS
    TorchData -->|"wraps"| BaseDS

    TFPipelines -->|"uses"| TFData
    TFPipelines -->|"uses"| TFModels
    TFPipelines -->|"logs to"| TB
    TFPipelines -->|"evaluates with"| mAP
    TFPipelines -->|"visualizes via"| Visualizer

    TorchPipelines -->|"uses"| TorchData
    TorchPipelines -->|"uses"| TorchModels
    TorchPipelines -->|"logs to"| TB
    TorchPipelines -->|"evaluates with"| mAP
    TorchPipelines -->|"visualizes via"| Visualizer

    TFModels -->|"uses modules"| TFModules
    TorchModels -->|"uses modules"| TorchModules

    CLI -->|"downloads"| Remote
    Remote -->|"stores"| FS

    %% Click Events - Core Utils
    click ConfigLoader "https://github.com/isl-org/open3d-ml/blob/main/ml3d/utils/config.py"
    click Registry "https://github.com/isl-org/open3d-ml/blob/main/ml3d/utils/registry.py"
    click Builder "https://github.com/isl-org/open3d-ml/blob/main/ml3d/utils/builder.py"
    click Log "https://github.com/isl-org/open3d-ml/blob/main/ml3d/utils/log.py"

    %% Click Events - Configs
    click Configs "https://github.com/isl-org/open3d-ml/tree/main/ml3d/configs/"
    click Configs "https://github.com/isl-org/open3d-ml/tree/main/ml3d/configs/default_cfgs/"

    %% Click Events - Datasets
    click BaseDS "https://github.com/isl-org/open3d-ml/blob/main/ml3d/datasets/base_dataset.py"
    click CustomDS "https://github.com/isl-org/open3d-ml/blob/main/ml3d/datasets/customdataset.py"
    click OtherDS "https://github.com/isl-org/open3d-ml/tree/main/ml3d/datasets/"

    %% Click Events - Metrics
    click mAP "https://github.com/isl-org/open3d-ml/blob/main/ml3d/metrics/mAP.py"

    %% Click Events - Visualization
    click Visualizer "https://github.com/isl-org/open3d-ml/blob/main/ml3d/vis/visualizer.py"
    click BoundingBox "https://github.com/isl-org/open3d-ml/blob/main/ml3d/vis/boundingbox.py"
    click ColorMap "https://github.com/isl-org/open3d-ml/blob/main/ml3d/vis/colormap.py"
    click LabelLUT "https://github.com/isl-org/open3d-ml/blob/main/ml3d/vis/labellut.py"

    %% Click Events - TensorFlow Adapter
    click TFData "https://github.com/isl-org/open3d-ml/blob/main/ml3d/tf/dataloaders/tf_dataloader.py"
    click TFModels "https://github.com/isl-org/open3d-ml/tree/main/ml3d/tf/models/"
    click TFModules "https://github.com/isl-org/open3d-ml/tree/main/ml3d/tf/modules/"
    click TFPipelines "https://github.com/isl-org/open3d-ml/blob/main/ml3d/tf/pipelines/semantic_segmentation.py"
    click TFPipelines "https://github.com/isl-org/open3d-ml/blob/main/ml3d/tf/pipelines/object_detection.py"
    click TFUtils "https://github.com/isl-org/open3d-ml/tree/main/ml3d/tf/utils/"

    %% Click Events - PyTorch Adapter
    click TorchData "https://github.com/isl-org/open3d-ml/blob/main/ml3d/torch/dataloaders/torch_dataloader.py"
    click TorchModels "https://github.com/isl-org/open3d-ml/tree/main/ml3d/torch/models/"
    click TorchModules "https://github.com/isl-org/open3d-ml/tree/main/ml3d/torch/modules/"
    click TorchPipelines "https://github.com/isl-org/open3d-ml/blob/main/ml3d/torch/pipelines/semantic_segmentation.py"
    click TorchPipelines "https://github.com/isl-org/open3d-ml/blob/main/ml3d/torch/pipelines/object_detection.py"
    click TorchUtils "https://github.com/isl-org/open3d-ml/tree/main/ml3d/torch/utils/"

    %% Click Events - User Interfaces
    click CLI "https://github.com/isl-org/open3d-ml/blob/main/scripts/run_pipeline.py"
    click CLI "https://github.com/isl-org/open3d-ml/tree/main/scripts/download_datasets/"
    click CLI "https://github.com/isl-org/open3d-ml/blob/main/scripts/preprocess_*.py"
    click CLI "https://github.com/isl-org/open3d-ml/tree/main/scripts/train_scripts/"
    click Examples "https://github.com/isl-org/open3d-ml/tree/main/examples/"

    %% Click Events - Deployment
    click FS "https://github.com/isl-org/open3d-ml/blob/main/setup.py"
    click FS "https://github.com/isl-org/open3d-ml/tree/main/Makefile"
    click FS "https://github.com/isl-org/open3d-ml/blob/main/requirements*.txt"
    click FS "https://github.com/isl-org/open3d-ml/tree/main/.github/workflows/"

    %% Styles
    classDef core fill:#cde8ff,stroke:#4b89dc,stroke-width:2px
    classDef fw fill:#d0f0c0,stroke:#3c763d,stroke-width:2px
    classDef ui fill:#ffe5b4,stroke:#d99600,stroke-width:2px
    classDef ext fill:#e0e0e0,stroke:#6e6e6e,stroke-width:2px
```
