How isl-org/open3d-ml works
Project type and purpose
• This is an open‐source 3D machine-learning library and framework extension for Open3D.
• It provides generic utilities, dataset readers, config‐driven builders, framework adapters (TensorFlow and PyTorch), model implementations, training/inference pipelines, visualization tools, example scripts, and CLI automation.
File structure
• Top‐level directories:
– ml3d (core package)
– scripts (CLI and demos)
– examples (user-facing code samples)
– docs (manuals & tutorials)
– data (demo assets)
– tests, CI, GitHub workflows
• Inside ml3d:
– configs: YAML files for models, datasets, pipelines
– datasets: framework-agnostic dataset reader code
– metrics: evaluation modules (mAP, IoU)
– utils: core utilities (Config loader, registry, builder, logging)
– tf: TensorFlow-specific subpackage (dataloaders, models, modules, pipelines, utils)
– torch: PyTorch-specific subpackage (mirror structure of tf)
• Scripts folder:
– run_pipeline.py: unified CLI
– download_datasets: shell scripts
– demo & preprocessing scripts
• Build/config/deployment files: Makefile, setup.py, CI scripts, requirements-*.txt
Extract architecture insights from README
• Config-driven instantiation: utilities.Config + utils.get_module build datasets, models, pipelines
• Dual‐framework support via parallel subpackages (ml3d.tf vs ml3d.torch)
• Core library (ml3d.*) integrated into open3d.ml namespace
• Example & script layers use the same core API
Main components & interactions
1. Core Layer (blue):
– ml3d.utils (Config loader, registry, builder)
– ml3d.configs (YAML definitions)
– ml3d.datasets (generic readers)
– ml3d.metrics, ml3d.vis
2. Framework Adapters (green):
– ml3d.tf / ml3d.tf.dataloaders → wrap generic datasets into tf.data pipelines
– ml3d.tf.models / modules (losses, metrics, schedulers)
– ml3d.tf.pipelines (SemanticSegmentation, ObjectDetection)
– ml3d.torch (same structure for PyTorch)
3. User Interfaces (orange):
– Python API (import open3d.ml.torch as ml3d)
– CLI scripts (scripts/run_pipeline.py, demo scripts)
– Notebooks & examples
4. External Systems (gray):
– File system (configs, datasets on disk, downloaded weights)
– Remote storage (model-zoo on GCS)
– Open3D core C++ library
– TensorBoard (for logs & metrics)
1. User → CLI / Python API → Core Layer (Config loader reads YAML)
2. Core Layer → Builder → Instantiates Dataset / Model / Pipeline classes in framework adapter
3. Pipeline → Dataloader → Generic Dataset → Disk (point clouds, labels)
4. Pipeline → Model (forward/backward passes) → Modules (loss, scheduler)
5. Pipeline → Metrics → ml3d.metrics → logs to console / TensorBoard
6. Pipeline → ml3d.vis → Visualization in UI
7. Scripts → Remote storage download → model weights → Disk
c. Architectural patterns & design principles:
– Plugin/Registry pattern: dynamic module loading via registry in utils
– Config‐driven architecture: separates specification (YAML) from implementation
– Adapter pattern: uniform API over TensorFlow and PyTorch backends
– Modular layering: core utility layer, framework‐specific layer, interface layer
d. Key technologies & libraries:
– Python 3.x, Open3D core
– TensorFlow 2.x, PyTorch 2.0
– YAML (config files)
– Shell scripts for dataset download & preprocessing
– CI pipelines (GitHub Actions)
– TensorBoard for monitoring
Elements
– Orange clouds for user entrypoints