# mit-spark/buffer-x architecture

> Architecture diagram of the GitHub repository mit-spark/buffer-x, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mit-spark/buffer-x
- Repository: https://github.com/mit-spark/buffer-x

## Overview

1. **Overall purpose**
   BUFFER-X is a Python-based point cloud registration system for zero-shot evaluation across many datasets. The repository is organized around a training/inference pipeline, dataset adapters and preprocessing, learned descriptor/patch models, and a final pose-estimation stage. It supports both standard and heterogeneous benchmark settings.

2. **Top-level execution flow**
   The main runtime entry points are `train.py` and `test.py`, with shared orchestration in `trainer.py`. Inference takes a dataset name plus an experiment ID, loads the corresponding pretrained snapshot, runs multi-scale point-cloud feature extraction, matches local patches, and then estimates a rigid transform. Training uses the same model stack but is mainly supported for `3DMatch` and `KITTI`.

3. **Configuration layer**
   The `config/` package is the architecture’s dataset- and scenario-specific control plane. Files like `indoor_config.py`, `kitti_config.py`, `oxford_config.py`, `tiers_hetero_config.py`, etc. encode per-dataset hyperparameters, matching settings, scale/patch parameters, and evaluation options. `config/splits/` contains benchmark split definitions that determine which scenes belong to train/val/test partitions.

4. **Dataset subsystem**
   `dataset/` contains a common base abstraction (`base_dataset.py`) plus dataset-specific loaders for indoor, outdoor, automotive, and special benchmarks such as `threedmatch`, `kitti`, `oxford`, `eth`, `mit`, `kaist`, `wod`, `tiers`, and `modelnet40`. The loaders are responsible for reading raw/preprocessed point clouds, generating paired samples, and exposing them through `dataloader.py`. This layer is the main boundary between benchmark data formats and the model pipeline.

5. **ScanNet++ preprocessing branch**
   ScanNet++ is treated as a heavier preprocessing pipeline rather than a simple loader. The `dataset/scannetpp/` subtree contains separate preparation flows for iPhone and FARO data, including scripts for decomposing point clouds, pairing frames, and compiling GPU/CUDA components. This indicates that some datasets require offline preprocessing into the repository’s internal pair format before they can be consumed by the standard dataset loaders.

6. **Model stack**
   The core learnable components live in `models/`: `patch_embedder.py`, `patchnet.py`, `pose_estimator.py`, and `BUFFERX.py`. Architecturally, this suggests a staged pipeline: local point neighborhoods are embedded into descriptors, patch-level features are aggregated/matched, and a pose-estimation module converts correspondences into a rigid alignment. `BUFFERX.py` likely serves as the top-level model wrapper that composes these pieces.

7. **Loss and optimization**
   The `loss/desc_loss.py` module defines the descriptor-training objective. This is a narrow subsystem focused on metric learning for patch/feature quality, separate from pose estimation. That separation matters architecturally: the learned representation is optimized independently of the final geometric solver.

8. **Geometry and SE(3) utilities**
   The `utils/` package provides the geometric and runtime support code: `SE3.py` for rigid transforms, timing/profiling helpers, progress formatting, result serialization, and miscellaneous tools. These utilities underpin both training and evaluation, especially for reporting registration metrics such as translation/rotation error and inlier statistics.

9. **Native acceleration layer**
   `cpp_wrappers/` contains compiled extensions for performance-critical point-cloud operations: neighborhood queries and grid subsampling. These are not optional implementation details; they are the high-throughput geometry primitives that make multi-scale local patch extraction practical on large point clouds. There is also a KISS-Matcher-related utility subtree, reflecting an alternate pose-estimation backend that can be swapped in during testing.

10. **Evaluation and result reporting**
   `test.py` runs the benchmark evaluation flow and writes outputs through utilities in `utils/result_io.py`. Results are emitted both per sample and as aggregated CSV summaries, which makes the evaluation pipeline reproducible and easy to audit. The scripts `scripts/eval_all.sh` and `scripts/eval_all_hetero.sh` are convenience wrappers for running the same evaluator across many datasets or heterogeneous sensor pairings.

11. **Data and model artifacts**
   The repo assumes two external artifact stores: preprocessed datasets and pretrained weights under `snapshot/`. `scripts/download_all_data.sh` and `scripts/download_pretrained_models.py` populate these assets, which are then consumed by training/testing without further tuning. This separation cleanly isolates large, copyright-sensitive data from the source tree.

12. **Packaging and deployment**
   The project is packaged as a Python distribution (`pyproject.toml`) with dependency sets in `requirements/`. There are explicit install paths for different CUDA/PyTorch combinations, plus optional extras for Hugging Face and ScanNet++ support. The `huggingface/` subtree is a deployment boundary: it contains a model-card template and a lightweight CPU-only Space helper for publishing and validating released snapshots.

13. **Architectural summary**
   In diagram terms, the repository is best viewed as: dataset adapters and preprocessing on the left, a Python model pipeline in the center, C++ geometry kernels as an acceleration layer beneath it, and evaluation/export tooling on the right. The primary data flow is raw/preprocessed point clouds → dataset loader → multi-scale patch extraction and descriptor inference → pose solver → metrics/results, with pretrained snapshots and optional Hugging Face distribution as external artifact dependencies.

Main entry points: [`trainer.py`](https://github.com/mit-spark/buffer-x/blob/main/trainer.py) (trainer.py), [`models/BUFFERX.py`](https://github.com/mit-spark/buffer-x/blob/main/models/BUFFERX.py) (BUFFERX), [`test.py`](https://github.com/mit-spark/buffer-x/blob/main/test.py) (test.py), [`dataset/dataloader.py`](https://github.com/mit-spark/buffer-x/blob/main/dataset/dataloader.py) (Dataloader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime & orchestration"]
  node_train_py["train.py<br/>entrypoint<br/>[train.py]"]
  node_test_py["test.py<br/>entrypoint<br/>[test.py]"]
  node_trainer_py["trainer.py<br/>orchestrator<br/>[trainer.py]"]
  node_result_io["Results IO<br/>reporting<br/>[result_io.py]"]
end

subgraph group_data["Data & config"]
  node_config_pkg["Config"]
  node_splits["Splits"]
  node_dataset_base["Base dataset<br/>dataset abstraction<br/>[base_dataset.py]"]
  node_dataset_loader["Dataloader<br/>batching<br/>[dataloader.py]"]
  node_benchmark_loaders["Benchmark loaders<br/>adapters"]
  node_scannetpp_preproc["ScanNet++ prep<br/>offline preprocessing"]
end

subgraph group_model["Model & loss"]
  node_bufferx_model["BUFFERX<br/>model wrapper<br/>[BUFFERX.py]"]
  node_patch_embedder["Patch embedder<br/>descriptor encoder<br/>[patch_embedder.py]"]
  node_patchnet["PatchNet<br/>patch matcher<br/>[patchnet.py]"]
  node_pose_estimator["Pose estimator<br/>SE3 solver<br/>[pose_estimator.py]"]
  node_desc_loss["Desc loss<br/>metric learning loss<br/>[desc_loss.py]"]
end

subgraph group_infra["Acceleration & packaging"]
  node_cpp_wrappers["CPP wrappers<br/>native geometry"]
  node_kiss_utils["KISS utils<br/>native backend"]
  node_utils_pkg["Utilities<br/>support code"]
  node_huggingface_pkg["Hugging Face<br/>distribution"]
  node_snapshot_store[("Snapshots<br/>external artifacts")]
end

node_train_py -->|"uses"| node_trainer_py
node_test_py -->|"uses"| node_trainer_py
node_trainer_py -->|"loads"| node_config_pkg
node_trainer_py -->|"reads batches"| node_dataset_loader
node_dataset_loader -->|"extends"| node_dataset_base
node_dataset_loader -->|"adapts"| node_benchmark_loaders
node_benchmark_loaders -->|"uses"| node_splits
node_benchmark_loaders -.->|"feeds"| node_scannetpp_preproc
node_trainer_py -->|"runs"| node_bufferx_model
node_bufferx_model -->|"composes"| node_patch_embedder
node_bufferx_model -->|"composes"| node_patchnet
node_bufferx_model -->|"composes"| node_pose_estimator
node_trainer_py -->|"optimizes"| node_desc_loss
node_patch_embedder -->|"accelerated by"| node_cpp_wrappers
node_patchnet -->|"accelerated by"| node_cpp_wrappers
node_pose_estimator -->|"uses"| node_utils_pkg
node_trainer_py -->|"writes"| node_result_io
node_test_py -->|"reports"| node_result_io
node_train_py -->|"loads"| node_snapshot_store
node_test_py -->|"loads"| node_snapshot_store
node_huggingface_pkg -.->|"publishes"| node_snapshot_store

click node_train_py "https://github.com/mit-spark/buffer-x/blob/main/train.py"
click node_test_py "https://github.com/mit-spark/buffer-x/blob/main/test.py"
click node_trainer_py "https://github.com/mit-spark/buffer-x/blob/main/trainer.py"
click node_result_io "https://github.com/mit-spark/buffer-x/blob/main/utils/result_io.py"
click node_config_pkg "https://github.com/mit-spark/buffer-x/tree/main/config"
click node_splits "https://github.com/mit-spark/buffer-x/tree/main/config/splits"
click node_dataset_base "https://github.com/mit-spark/buffer-x/blob/main/dataset/base_dataset.py"
click node_dataset_loader "https://github.com/mit-spark/buffer-x/blob/main/dataset/dataloader.py"
click node_benchmark_loaders "https://github.com/mit-spark/buffer-x/tree/main/dataset"
click node_scannetpp_preproc "https://github.com/mit-spark/buffer-x/tree/main/dataset/scannetpp"
click node_bufferx_model "https://github.com/mit-spark/buffer-x/blob/main/models/BUFFERX.py"
click node_patch_embedder "https://github.com/mit-spark/buffer-x/blob/main/models/patch_embedder.py"
click node_patchnet "https://github.com/mit-spark/buffer-x/blob/main/models/patchnet.py"
click node_pose_estimator "https://github.com/mit-spark/buffer-x/blob/main/models/pose_estimator.py"
click node_desc_loss "https://github.com/mit-spark/buffer-x/blob/main/loss/desc_loss.py"
click node_cpp_wrappers "https://github.com/mit-spark/buffer-x/tree/main/cpp_wrappers"
click node_kiss_utils "https://github.com/mit-spark/buffer-x/tree/main/cpp_wrappers/cpp_utils/kiss_matcher"
click node_utils_pkg "https://github.com/mit-spark/buffer-x/tree/main/utils"
click node_huggingface_pkg "https://github.com/mit-spark/buffer-x/tree/main/huggingface"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_train_py,node_test_py,node_trainer_py,node_result_io toneBlue
class node_config_pkg,node_splits,node_dataset_base,node_dataset_loader,node_benchmark_loaders,node_scannetpp_preproc toneAmber
class node_bufferx_model,node_patch_embedder,node_patchnet,node_pose_estimator,node_desc_loss toneMint
class node_cpp_wrappers,node_kiss_utils,node_utils_pkg,node_huggingface_pkg,node_snapshot_store toneRose
```

## Components

### Runtime & orchestration

Entry points, shared training loop, evaluation, and reporting.

- [train.py](https://github.com/mit-spark/buffer-x/blob/main/train.py): entrypoint · Training entry script. · `train.py`
- [test.py](https://github.com/mit-spark/buffer-x/blob/main/test.py): entrypoint · Benchmark evaluation entry script. · `test.py`
- [trainer.py](https://github.com/mit-spark/buffer-x/blob/main/trainer.py): orchestrator · Shared training and inference coordination. · `trainer.py`
- [Results IO](https://github.com/mit-spark/buffer-x/blob/main/utils/result_io.py): reporting · Result writing and aggregation utilities. · `utils/result_io.py`

### Data & config

Dataset adapters, benchmark splits, and dataset-specific control files.

- [Config](https://github.com/mit-spark/buffer-x/tree/main/config): scenario config · Dataset and benchmark-specific hyperparameter control plane. · `config`
- [Splits](https://github.com/mit-spark/buffer-x/tree/main/config/splits): benchmark splits · Train/val/test scene partitions. · `config/splits`
- [Base dataset](https://github.com/mit-spark/buffer-x/blob/main/dataset/base_dataset.py): dataset abstraction · Common dataset interface. · `dataset/base_dataset.py`
- [Dataloader](https://github.com/mit-spark/buffer-x/blob/main/dataset/dataloader.py): batching · Dataset sampling and batching interface. · `dataset/dataloader.py`
- [Benchmark loaders](https://github.com/mit-spark/buffer-x/tree/main/dataset): adapters · Dataset-specific loaders for indoor, outdoor, automotive, and special benchmarks. · `dataset`
- [ScanNet++ prep](https://github.com/mit-spark/buffer-x/tree/main/dataset/scannetpp): offline preprocessing · Heavier preprocessing pipeline for iPhone and FARO ScanNet++ data. · `dataset/scannetpp`

### Model & loss

Learned patch descriptors, matching, pose estimation, and training objective.

- [BUFFERX](https://github.com/mit-spark/buffer-x/blob/main/models/BUFFERX.py): model wrapper · Top-level model composition for descriptor extraction and pose estimation. · `models/BUFFERX.py`
- [Patch embedder](https://github.com/mit-spark/buffer-x/blob/main/models/patch_embedder.py): descriptor encoder · Embeds local point neighborhoods into features. · `models/patch_embedder.py`
- [PatchNet](https://github.com/mit-spark/buffer-x/blob/main/models/patchnet.py): patch matcher · Aggregates and matches patch-level features. · `models/patchnet.py`
- [Pose estimator](https://github.com/mit-spark/buffer-x/blob/main/models/pose_estimator.py): SE3 solver · Converts correspondences into rigid alignment. · `models/pose_estimator.py`
- [Desc loss](https://github.com/mit-spark/buffer-x/blob/main/loss/desc_loss.py): metric learning loss · Descriptor training objective. · `loss/desc_loss.py`

### Acceleration & packaging

Native geometry kernels, utility layer, and distribution artifacts.

- [CPP wrappers](https://github.com/mit-spark/buffer-x/tree/main/cpp_wrappers): native geometry · Accelerated neighborhood query and grid subsampling kernels. · `cpp_wrappers`
- [KISS utils](https://github.com/mit-spark/buffer-x/tree/main/cpp_wrappers/cpp_utils/kiss_matcher): native backend · Alternate geometry and matcher utilities used by the C++ support layer. · `cpp_wrappers/cpp_utils/kiss_matcher`
- [Utilities](https://github.com/mit-spark/buffer-x/tree/main/utils): support code · Transforms, timing, progress, and helper functions. · `utils`
- [Hugging Face](https://github.com/mit-spark/buffer-x/tree/main/huggingface): distribution · Model card and Space deployment artifacts. · `huggingface`
- **Snapshots**: external artifacts · Pretrained weights and benchmark data consumed at runtime.

## Connections

- train.py → trainer.py: uses · Training delegates orchestration to the shared trainer.
- test.py → trainer.py: uses · Evaluation reuses the shared execution flow.
- trainer.py → Config: loads · Selects dataset- and scenario-specific settings.
- trainer.py → Dataloader: reads batches · Consumes point-cloud pairs through the dataloader.
- Dataloader → Base dataset: extends · Uses the shared dataset abstraction.
- Dataloader → Benchmark loaders: adapts · Routes to dataset-specific readers and pair builders.
- Benchmark loaders → Splits: uses · Applies benchmark partitions for train/val/test.
- Benchmark loaders → ScanNet++ prep: feeds · ScanNet++ loaders depend on offline preprocessing.
- trainer.py → BUFFERX: runs · Executes the main registration model.
- BUFFERX → Patch embedder: composes · Builds local descriptors from neighborhoods.
- BUFFERX → PatchNet: composes · Aggregates and matches patch-level features.
- BUFFERX → Pose estimator: composes · Solves rigid alignment from correspondences.
- trainer.py → Desc loss: optimizes · Uses descriptor loss during training.
- Patch embedder → CPP wrappers: accelerated by · Relies on fast neighborhood queries and subsampling.
- PatchNet → CPP wrappers: accelerated by · Uses native geometry primitives for large point clouds.
- Pose estimator → Utilities: uses · Relies on SE(3) and runtime helpers.
- trainer.py → Results IO: writes · Emits evaluation outputs and aggregates metrics.
- test.py → Results IO: reports · Serializes per-sample and summary results.
- train.py → Snapshots: loads · Consumes pretrained weights and saved artifacts when available.
- test.py → Snapshots: loads · Loads experiment snapshots for evaluation.
- Hugging Face → Snapshots: publishes · Packages released snapshots and model card assets.
