How sanweiliti/rohm works
Project Type and Purpose
• The project is a research/experimental full-stack implementation for human motion reconstruction, specifically using diffusion models to recover complete and plausible motion sequences from noisy and occluded inputs.
• Key components include two separate sub-models (PoseNet for local motion and TrajNet for global trajectory) and a diffusion module that underpins both. The README highlights training, evaluation, and testing on multiple datasets (AMASS, PROX, EgoBody), which all bring unique input preparation and configuration steps.
Analysis of the File Structure
• The top-level directories clearly separate configuration (cfg_files) from core functionalities:
• cfg_files contains YAML configuration files for evaluation, testing, and training. These files dictate how the modules are set up in different experiment scenarios.
• data_loaders holds data ingestion and preprocessing logic, critical for converting raw input (AMASS, PROX, EgoBody) into the desired format.
• diffusion includes modules for different aspects of the diffusion process (e.g., gaussian_diffusion for both PoseNet and TrajNet, resample and respace utilities, logging, and fp16 support).
• model contains implementations of the core networks (posenet, trajnet) and components like heads and configuration samplers.
• train includes training loops for both PoseNet and TrajNet, showing the progression (curriculum training) from initial training stages to fine-tuning.
• Utilities (in utils) support various tasks such as seed fixing, occlusion mask generation, rendering, and general model utilities.
• There is also clear segregation for test and evaluation scripts that direct the flow for different datasets and scenarios.
High-Level System Components to Capture
a. Input/Preprocessing:
• Data Preparation: Preprocessing scripts (preprocessing_amass.py) used for preparing AMASS data.
• Data Loaders: Modules in the data_loaders folder that import and process data (common routines, quaternion handling, etc.).
• External Data: Downloaded checkpoints, SMPL-X model data, and dataset folders (data/ and datasets/ directories) mentioned in the README.
b. Core Model Components:
• PoseNet: Located in model/posenet.py and trained via train_posenet.py; responsible for local motion reconstruction.
• TrajNet: Found in model/trajnet.py and trained with train_trajnet.py; handles the global trajectory reconstruction.
• Conditioning Module: As described in the README (and implied by the coupling between PoseNet and TrajNet models), this module integrates the outputs of both branches to ensure consistency.
• Diffusion Module: In diffusion/ (including gaussian_diffusion_posenet.py and gaussian_diffusion_trajnet.py) which encapsulates the iterative inference method central to the reconstruction.
c. Training, Testing, and Evaluation Pipelines:
• Training Loops: Scripts in the train directory that outline the training stages; curriculum learning is emphasized with stage-specific behavior.
• Testing Framework: Test scripts (e.g., test_amass_full.py, test_prox_egobody.py) that run inference experiments on various datasets.
• Evaluation Pipeline: Evaluation scripts (e.g., eval_amass_full.py, eval_prox_egobody.py) that compute metrics and generate visuals/rendered outputs.
d. Utility and Support Modules:
• Utility Components: Helper functions in utils (e.g., model_util.py, render_util.py, vis_util.py) that support debugging, configuration, and rendering.
• Diffusion Support: Additional utilities for FP16 computation and logging within the diffusion directory.
Relationships and Data Flow
▸ The TrajNet branch (global trajectory reconstruction) which includes its diffusion process and training loop.
▸ The PoseNet branch (local motion reconstruction) following a similar pattern.
• Both branches then converge at the Conditioning Module, which ensures the two sub-tasks are coherent.
• From the Conditioning Module, flow forward to Output Generation (reconstructed motion sequences) and final Evaluation (metrics, visualization, rendering).
Detailed Element Inclusions
• Two Processing Branches:
▸ Left branch: Global Trajectory Reconstruction (TrajNet, its associated diffusion module, and training loops).
▸ Right branch: Local Pose Reconstruction (PoseNet, its associated diffusion module, and training loops).
• Utility Libraries and External Tool Dependencies: Clearly denote where tools from external repositories support parts of the process (e.g., CLIFF for initialization).