How microsoft/singleshotpose works
1. What this repository is
This is a research codebase for single-shot 6D object pose estimation. The main idea is to use a YOLO-style CNN to predict 2D image locations of projected 3D bounding-box keypoints, then recover the 6D pose with a PnP step. The repo contains both the single-object LINEMOD pipeline and a multi-object OCCLUSION variant.
2. Top-level architecture split
The repo is organized around three layers:
- Core single-object code at the root
- Multi-object extension in multi_obj_pose_estimation/
- A full Python 2 compatibility fork in py2/
The duplicated structure shows that the project supports the same pipeline across two runtimes / PyTorch-era implementations.
3. Core execution flow
The primary runtime path is:
- train.py for training
- valid.py for evaluation / inference
- cfg.py for loading dataset and model configuration
- darknet.py for building the YOLO-like network
- dataset.py and image.py for data loading / augmentation / image handling
- region_loss.py for the custom loss used during training
- utils.py for shared math, geometry, and post-processing helpers
This is a classic config-driven training stack rather than an app with services or APIs.
4. Model boundary: Darknet-style network definition
The network architecture is not hardcoded in Python; it is primarily defined by Darknet config files under cfg/, such as:
- yolo-pose.cfg
- yolo-pose-pre.cfg
These configs define the CNN layers, detection head, and training schedule. darknet.py translates that configuration into a PyTorch model. This boundary is important: the model topology is externalized from the training code.
5. Data boundary: .data files and dataset assets
Object-specific dataset metadata lives in cfg/*.data files, such as ape.data, duck.data, etc. These files encode the paths and camera/object parameters needed for training and testing. The repo also expects:
- train/test split text files
- label files
- .ply 3D object models
- optional mask/background assets
So the data pipeline is split between file-based dataset descriptors and Python loading code.
6. Single-object pose estimation pipeline
For a single object, the pipeline is:
- read image + label + camera/object metadata
- augment and normalize samples in dataset.py
- feed batches into the YOLO-like model from darknet.py
- compute detection/keypoint regression loss in region_loss.py
- save learned weights to a model file
- run valid.py to decode predictions and evaluate pose quality
The key output is not a bounding box alone, but the projected keypoints used to reconstruct pose.
7. Multi-object extension
multi_obj_pose_estimation/ mirrors the core stack with separate modules:
- darknet_multi.py
- dataset_multi.py
- image_multi.py
- region_loss_multi.py
- train_multi.py
- valid_multi.py
- utils_multi.py
This is a distinct architecture variant for the OCCLUSION dataset where the model handles multiple object classes and uses multi-class training configs in multi_obj_pose_estimation/cfg/.
8. How multi-object differs architecturally
The multi-object version adds an anchor-based assignment and class-aware output structure. The README notes that each label contains 21 values: class + 9 projected points + x/y range. This means the multi-object head has to predict both:
- pose-related keypoints
- class / object identity
- size or anchor-related terms
So the main difference is not the backbone, but the output encoding and training target assignment.
9. Shared geometry and 3D assets
The pose logic depends on 3D object model files (.ply) and camera intrinsics from the config files. MeshPly.py likely handles parsing mesh files or related 3D geometry utilities. This geometry layer is critical because pose is recovered from predicted 2D locations against known 3D reference points.
10. Training vs inference responsibilities
Training scripts are responsible for:
- loading dataset definitions
- initializing weights
- running optimization
- checkpointing model weights
Validation scripts are responsible for:
- loading trained weights
- running forward inference
- post-processing predictions into pose estimates
The repo keeps these paths separate, which matters when drawing the architecture: training is not embedded in inference code.
11. Configuration and reproducibility
The repo relies heavily on external config files for reproducibility:
- model config: layer structure, loss settings, learning rate schedule
- data config: paths, camera intrinsics, object diameter, train/test splits
This makes the codebase highly parameterized and dataset-dependent, which is central to how the architecture operates across LINEMOD, OCCLUSION, and custom datasets.
12. Legacy compatibility branch
The py2/ directory is not a separate product; it is a legacy implementation snapshot for older Python 2.7 / PyTorch 0.3.1 environments. It mirrors the same high-level modules as the root code, so it should be viewed as an alternate runtime branch rather than an independent subsystem.
13. External dependencies and runtime context
The architecture assumes a GPU training environment with PyTorch 0.4.1, CUDA, and common CV libraries like NumPy, SciPy, PIL, and OpenCV. Pretrained weights and dataset tarballs are external inputs expected at setup time, so the repository is only one part of the end-to-end pose estimation system.
14. Diagram takeaway
If you draw this repo, the central flow is:
config + dataset assets → dataset loader/augmentation → Darknet-style CNN → custom region loss during training / PnP-based pose decoding during validation
The main architectural boundaries are between:
- configuration vs code
- data/geometry vs model
- training vs inference
- single-object vs multi-object variants
- modern code vs Python 2 legacy fork
Main entry points: darknet.py darknet.py · valid.py valid.py · darknet_multi.py darknet_multi.py · train.py train.py
Components 19
Core pipeline
Single-object LINEMOD training and validation stack.
- cfg.py · Loads dataset and model configuration for the single-object pipeline.
cfg.py - cfg files · Darknet model definitions and per-object .data descriptors.
cfg - train.py · Runs optimization, checkpointing, and model export for single-object pose estimation.
train.py - valid.py · Loads trained weights and decodes predictions into pose estimates.
valid.py - darknet.py · Builds the YOLO-style network from external Darknet config files.
darknet.py - dataset.py · Reads images, labels, and metadata; applies augmentation and batching.
dataset.py - image.py · Image loading, preprocessing, and augmentation helpers.
image.py - region_loss.py · Pose-specific detection and keypoint regression loss used during training.
region_loss.py - utils.py · Shared math, decoding, and post-processing helpers.
utils.py - MeshPly.py · Parses or supports .ply mesh assets used for pose recovery.
MeshPly.py
Multi-object pipeline
OCCLUSION multi-class variant with mirrored modules.
- multi cfg · OCCLUSION dataset descriptors and multi-object Darknet configs.
multi_obj_pose_estimation/cfg - train_multi.py · Multi-object training loop for class-aware pose estimation.
multi_obj_pose_estimation/train_multi.py - valid_multi.py · Multi-object inference and pose evaluation pipeline.
multi_obj_pose_estimation/valid_multi.py - darknet_multi.py · Builds the multi-object YOLO-style network variant.
multi_obj_pose_estimation/darknet_multi.py - dataset_multi.py · Loads multi-object samples and target encodings for OCCLUSION.
multi_obj_pose_estimation/dataset_multi.py - image_multi.py · Multi-object image preprocessing and augmentation helpers.
multi_obj_pose_estimation/image_multi.py - region_loss_multi.py · Multi-class pose loss with anchor-aware target assignment.
multi_obj_pose_estimation/region_loss_multi.py - utils_multi.py · Shared decoding and post-processing helpers for the multi-object branch.
multi_obj_pose_estimation/utils_multi.py
Legacy fork
Python 2 compatibility branch that mirrors the core architecture.
- py2/ · Python 2 compatibility snapshot mirroring the core and multi-object pipelines.
py2
Connections 22
- cfg.py → train.py: configures · Provides training configuration and dataset parameters.
- cfg.py → valid.py: configures · Provides evaluation and dataset parameters.
- cfg files → darknet.py: defines · Darknet config files define the core network topology.
- cfg files → dataset.py: describes · Data files and split metadata feed the dataset loader.
- train.py → dataset.py: loads · Training consumes augmented batches from the dataset pipeline.
- dataset.py → image.py: uses · Dataset loading relies on image preprocessing helpers.
- train.py → darknet.py: builds · Training constructs the model from the Darknet wrapper.
- darknet.py → region_loss.py: optimizes · Model outputs are trained against the custom region loss.
- darknet.py → utils.py: decodes · Utilities support geometry and post-processing inside the model flow.
- region_loss.py → utils.py: uses · Loss computation shares geometric helper logic.
- valid.py → darknet.py: runs · Validation loads the trained model for inference.
- valid.py → utils.py: post-processes · Validation decodes predictions into pose estimates.
- valid.py → MeshPly.py: poses · Pose recovery depends on 3D mesh geometry.
- multi cfg → darknet_multi.py: defines · Multi-object configs define the branch-specific topology and data descriptors.
- train_multi.py → dataset_multi.py: loads · Multi-object training consumes OCCLUSION batches.
- train_multi.py → darknet_multi.py: builds · Training constructs the multi-object model.
- dataset_multi.py → image_multi.py: uses · Dataset loading relies on image preprocessing helpers.
- darknet_multi.py → region_loss_multi.py: optimizes · Multi-object outputs are trained with the branch-specific loss.
- valid_multi.py → darknet_multi.py: runs · Validation loads the multi-object model for inference.
- valid_multi.py → utils_multi.py: post-processes · Utilities decode predictions into multi-object pose outputs.
- py2/ → cfg.py: mirrors · Legacy branch mirrors the core configuration and runtime structure.
- py2/ → train_multi.py: mirrors · Legacy branch also contains the multi-object training variant.