How arthur151/romp works
Project Type and Purpose
• This is a research-grade, Python-based deep learning library for monocular multi-person 3D human mesh recovery and tracking. It bundles three related CVPR/ICCV papers—ROMP (real-time mesh regression), BEV (depth relationships), and TRACE (temporal tracking with dynamic cameras)—into one repo. It supports training, evaluation, inference, export, and visualization, and exposes a simple cross-platform API via the simple_romp package.
High-Level File Structure
• Top-Level Directories
• configs/ – YAML files for various training/evaluation settings
• demo/ – example images and scripts
• docs/ – markdown guides on installation, training, evaluation, Docker, etc.
• romp/ – core implementation of ROMP & BEV (model definitions, dataset loaders, loss functions, training, evaluation, prediction, exports, visualization)
• trace/ – core implementation of TRACE (extensions for temporal regression under dynamic cameras)
• simple_romp/ – pip-installable wrapper for easy inference & export (fbx/glb/bvh) of ROMP, BEV, TRACE
• scripts/ – shell scripts to launch training, testing, exporting, Docker setup
• trace2/ under simple_romp – another wrapper for TRACE inference & evaluation
• Key Files
• train.py, eval.py, predict/ – entry points for training, evaluation, inference
• lib/models/, lib/dataset/, lib/loss_funcs/, lib/evaluation/, lib/visualization/ – modular subpackages implementing core stages
• Dockerfile, docs/docker.md – containerized environment setup
Main Architectural Components
a. Data Sources & Preprocessing
• Input modalities: images, video files, webcam streams
• Dataset modules under */lib/dataset/* for handling COCO, 3DPW, MPII, Panoptic, video, custom challenge datasets
• Augmentations & preprocessing pipelines in lib/utils/augments.py, lib/utils/*_utils.py
b. Model Engine
• PyTorch-based backbones (ResNet50, HRNet32) and custom heads in lib/models/
• SMPL parameter regression modules in lib/smpl_family/
• BEV depth refinement and TRACE temporal modules (GRU/optical-flow/trackers)
c. Training Pipeline
• Entry point: train.py, trace/train_video.py under trace/
• Loss computation in lib/loss_funcs/ (keypoints, parameters, prior, temporal)
• Multi-GPU support via parallel/* scripts
• Configuration via YAML files in configs/ and lib/config.py
d. Evaluation Pipeline
• Single/multi-dataset evaluation scripts in eval.py, romp/eval.py, lib/evaluation/
• Metrics for mesh accuracy (PCKh, MPJPE), 3D pose, tracking performance (HOTA, CLEAR)
e. Inference & Export
• Lightweight API wrapper in simple_romp/ and romp/predict/—predict on image/video/webcam
• Exporters to FBX/GLB/BVH in romp/exports/ and helper scripts (export_fbx.sh)
f. Visualization & Serving
• Open3D-based real-time visualizer with client/server in lib/visualization/vis_platform/
• Static renderers (vedo, matplotlib, Plotly)
g. Tracking (TRACE)
• Tracker modules in trace/lib/tracker/, romp/lib/tracking/ with Kalman filters, data association
• Temporal regressors (GRU, Transformer) in trace/lib/models/
Relationships and Data Flow
• Data (images/video/webcam) → Dataset Loader → Preprocessing (resize, augment) → Model Forward → Predicted SMPL params + 3D mesh → (optional) Temporal Smoother/Tracker → Export/Save results → Visualization
• Training: add Loss → Backward → Optimizer update → Checkpoint
• Evaluation: Model checkpoint → Inference on test sets → Metric computation
Architectural Patterns & Design Principles
• Layered modular design: clear separation among Data, Model, Loss, Eval, Inference, Visualization
• Config-driven behavior: YAML files control all hyperparameters and dataset choices
• Wrapper API pattern: simple_romp hides internal complexity for end users
• Plug-in evaluators & exporters: easy to add new dataset or export format
• Client-server visualization: decouples rendering from inference
Detailed Elements
• Clear component names matching folder names (e.g., romp/lib/models, trace/lib/datasets)
• Notes on how YAML config files steer the pipelines
• Optional inset showing how simple_romp orchestrates the underlying romp/ and trace/ modules for end-user inference