How nvlabs/genmo works
GEM is a unified human-motion estimation and generation system, accepting video, 2D keypoints, text, and audio. Its principal demo workflow combines video and text inputs, extracts visual cues, conditions a motion denoiser, decodes and post-processes SMPL motion, then optionally renders results. Training is a separate branch that assembles motion datasets and trains the same model family. README documents a webcam path through YOLOX, ViTPose-H, HMR2, and GEM; exact wiring beyond those documented stages is omitted where source excerpts are incomplete.
Main entry points: gem.py GEM model · demo_webcam.py Webcam demo · gem_pipeline.py Inference pipeline · demo_smpl.py Multimodal demo
Components 23
Inference and demos
- Multimodal demo
scripts/demo/demo_smpl.py - Video-only demo
scripts/demo/demo_smpl_hpe.py - Webcam demo
scripts/demo/demo_webcam.py - YOLOX detection
gem/utils/yolox_detector.py - ViTPose keypoints
gem/utils/vitpose_model.py - HMR2 features
gem/network/hmr2/hmr2.py - Motion rendering
gem/utils/vis/renderer.py - SMPL and video outputs
Motion modeling
- GEM model
gem/gem.py - Inference pipeline
gem/pipeline/gem_pipeline.py - Motion denoiser
gem/network/gem_denoiser.py - Diffusion sampler
gem/network/gem_diffusion.py - Gaussian diffusion
gem/diffusion_utils/gaussian_diffusion.py
Training data
- Data assembly
gem/datamodule/mocap_trainX_testY.py - Motion datasets
gem/datasets
Training and evaluation
- Training entry
scripts/train.py - Metrics and callbacks
gem/callbacks - Model checkpoint
Motion representation
- Motion encoding
gem/network/endecoder.py - Motion postprocess
gem/utils/postprocess.py - SMPL body model
gem/utils/body_model/smplx_lite.py
Other components
- User / video source
- Hugging Face
Connections 28
- User / video source → Multimodal demo: provides video/text
- User / video source → Video-only demo: provides video
- User / video source → Webcam demo: provides stream
- Multimodal demo → GEM model: runs inference
- Video-only demo → GEM model: runs inference
- Webcam demo → YOLOX detection: detects people
- Webcam demo → ViTPose keypoints: estimates keypoints
- Webcam demo → HMR2 features: extracts features
- Webcam demo → GEM model: streams motion input
- GEM model → HMR2 features: extracts image features
- GEM model → Inference pipeline: invokes
- Inference pipeline → Motion denoiser: predicts motion
- Inference pipeline → Diffusion sampler: samples motion
- Diffusion sampler → Gaussian diffusion: uses diffusion steps
- Inference pipeline → Motion encoding: decodes prediction
- Inference pipeline → Motion postprocess: post-processes motion
- Motion postprocess → SMPL body model: uses body geometry
- Multimodal demo → Motion rendering: renders results
- Video-only demo → Motion rendering: renders results
- Webcam demo → Motion rendering: renders stream
- Motion rendering → SMPL and video outputs: writes visualization
- Multimodal demo → SMPL and video outputs: saves parameters
- Training entry → Data assembly: configures loaders
- Data assembly → Motion datasets: instantiates datasets
- Training entry → GEM model: trains model
- GEM model → Metrics and callbacks: reports metrics
- GEM model → Model checkpoint: saves weights
- Hugging Face → Multimodal demo: provides checkpoint