# nvlabs/genmo architecture

> Architecture diagram of the GitHub repository nvlabs/genmo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvlabs/genmo
- Repository: https://github.com/nvlabs/genmo

## Overview

GEM is a unified human-motion estimation and generation system, accepting video, 2D keypoints, text, and audio. Its principal demo workflow combines video and text inputs, extracts visual cues, conditions a motion denoiser, decodes and post-processes SMPL motion, then optionally renders results. Training is a separate branch that assembles motion datasets and trains the same model family. README documents a webcam path through YOLOX, ViTPose-H, HMR2, and GEM; exact wiring beyond those documented stages is omitted where source excerpts are incomplete.

Main entry points: [`gem/gem.py`](https://github.com/nvlabs/genmo/blob/main/gem/gem.py) (GEM model), [`scripts/demo/demo_webcam.py`](https://github.com/nvlabs/genmo/blob/main/scripts/demo/demo_webcam.py) (Webcam demo), [`gem/pipeline/gem_pipeline.py`](https://github.com/nvlabs/genmo/blob/main/gem/pipeline/gem_pipeline.py) (Inference pipeline), [`scripts/demo/demo_smpl.py`](https://github.com/nvlabs/genmo/blob/main/scripts/demo/demo_smpl.py) (Multimodal demo)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_demo["Inference and demos"]
  node_demo_smpl["Multimodal demo<br/>[demo_smpl.py]"]
  node_demo_hpe["Video-only demo<br/>[demo_smpl_hpe.py]"]
  node_webcam["Webcam demo<br/>[demo_webcam.py]"]
  node_yolox["YOLOX detection<br/>[yolox_detector.py]"]
  node_vitpose["ViTPose keypoints<br/>[vitpose_model.py]"]
  node_hmr2["HMR2 features<br/>[hmr2.py]"]
  node_renderer["Motion rendering<br/>[renderer.py]"]
  node_outputs["SMPL and video outputs"]
end

subgraph group_model["Motion modeling"]
  node_gem_model["GEM model<br/>[gem.py]"]
  node_pipeline["Inference pipeline<br/>[gem_pipeline.py]"]
  node_denoiser["Motion denoiser<br/>[gem_denoiser.py]"]
  node_diffusion["Diffusion sampler<br/>[gem_diffusion.py]"]
  node_diffusion_utils["Gaussian diffusion"]
end

subgraph group_data["Training data"]
  node_data_module["Data assembly"]
  node_datasets["Motion datasets"]
end

subgraph group_train["Training and evaluation"]
  node_train_entry["Training entry<br/>[train.py]"]
  node_callbacks["Metrics and callbacks"]
  node_checkpoint["Model checkpoint"]
end

subgraph group_geometry["Motion representation"]
  node_endecoder["Motion encoding<br/>[endecoder.py]"]
  node_postprocess["Motion postprocess<br/>[postprocess.py]"]
  node_body_model["SMPL body model<br/>[smplx_lite.py]"]
end

node_user(("User / video source"))
node_hf["Hugging Face"]

node_user -->|"provides video/text"| node_demo_smpl
node_user -->|"provides video"| node_demo_hpe
node_user -->|"provides stream"| node_webcam
node_demo_smpl -->|"runs inference"| node_gem_model
node_demo_hpe -->|"runs inference"| node_gem_model
node_webcam -->|"detects people"| node_yolox
node_webcam -->|"estimates keypoints"| node_vitpose
node_webcam -.->|"extracts features"| node_hmr2
node_webcam -->|"streams motion input"| node_gem_model
node_gem_model -.->|"extracts image features"| node_hmr2
node_gem_model -->|"invokes"| node_pipeline
node_pipeline -->|"predicts motion"| node_denoiser
node_pipeline -->|"samples motion"| node_diffusion
node_diffusion -->|"uses diffusion steps"| node_diffusion_utils
node_pipeline -->|"decodes prediction"| node_endecoder
node_pipeline -->|"post-processes motion"| node_postprocess
node_postprocess -->|"uses body geometry"| node_body_model
node_demo_smpl -->|"renders results"| node_renderer
node_demo_hpe -->|"renders results"| node_renderer
node_webcam -->|"renders stream"| node_renderer
node_renderer -->|"writes visualization"| node_outputs
node_demo_smpl -->|"saves parameters"| node_outputs
node_train_entry -->|"configures loaders"| node_data_module
node_data_module -->|"instantiates datasets"| node_datasets
node_train_entry -->|"trains model"| node_gem_model
node_gem_model -->|"reports metrics"| node_callbacks
node_gem_model -->|"saves weights"| node_checkpoint
node_hf -.->|"provides checkpoint"| node_demo_smpl

click node_demo_smpl "https://github.com/nvlabs/genmo/blob/main/scripts/demo/demo_smpl.py"
click node_demo_hpe "https://github.com/nvlabs/genmo/blob/main/scripts/demo/demo_smpl_hpe.py"
click node_webcam "https://github.com/nvlabs/genmo/blob/main/scripts/demo/demo_webcam.py"
click node_yolox "https://github.com/nvlabs/genmo/blob/main/gem/utils/yolox_detector.py"
click node_vitpose "https://github.com/nvlabs/genmo/blob/main/gem/utils/vitpose_model.py"
click node_hmr2 "https://github.com/nvlabs/genmo/blob/main/gem/network/hmr2/hmr2.py"
click node_gem_model "https://github.com/nvlabs/genmo/blob/main/gem/gem.py"
click node_pipeline "https://github.com/nvlabs/genmo/blob/main/gem/pipeline/gem_pipeline.py"
click node_denoiser "https://github.com/nvlabs/genmo/blob/main/gem/network/gem_denoiser.py"
click node_diffusion "https://github.com/nvlabs/genmo/blob/main/gem/network/gem_diffusion.py"
click node_diffusion_utils "https://github.com/nvlabs/genmo/blob/main/gem/diffusion_utils/gaussian_diffusion.py"
click node_endecoder "https://github.com/nvlabs/genmo/blob/main/gem/network/endecoder.py"
click node_postprocess "https://github.com/nvlabs/genmo/blob/main/gem/utils/postprocess.py"
click node_body_model "https://github.com/nvlabs/genmo/blob/main/gem/utils/body_model/smplx_lite.py"
click node_renderer "https://github.com/nvlabs/genmo/blob/main/gem/utils/vis/renderer.py"
click node_data_module "https://github.com/nvlabs/genmo/blob/main/gem/datamodule/mocap_trainX_testY.py"
click node_datasets "https://github.com/nvlabs/genmo/tree/main/gem/datasets"
click node_train_entry "https://github.com/nvlabs/genmo/blob/main/scripts/train.py"
click node_callbacks "https://github.com/nvlabs/genmo/tree/main/gem/callbacks"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_demo_smpl,node_demo_hpe,node_webcam,node_yolox,node_vitpose,node_hmr2,node_renderer,node_outputs,node_user toneBlue
class node_gem_model,node_pipeline,node_denoiser,node_diffusion,node_diffusion_utils toneAmber
class node_data_module,node_datasets toneMint
class node_train_entry,node_callbacks,node_checkpoint toneRose
class node_endecoder,node_postprocess,node_body_model,node_hf toneIndigo
```

## Components

### Inference and demos

- [Multimodal demo](https://github.com/nvlabs/genmo/blob/main/scripts/demo/demo_smpl.py): `scripts/demo/demo_smpl.py`
- [Video-only demo](https://github.com/nvlabs/genmo/blob/main/scripts/demo/demo_smpl_hpe.py): `scripts/demo/demo_smpl_hpe.py`
- [Webcam demo](https://github.com/nvlabs/genmo/blob/main/scripts/demo/demo_webcam.py): `scripts/demo/demo_webcam.py`
- [YOLOX detection](https://github.com/nvlabs/genmo/blob/main/gem/utils/yolox_detector.py): `gem/utils/yolox_detector.py`
- [ViTPose keypoints](https://github.com/nvlabs/genmo/blob/main/gem/utils/vitpose_model.py): `gem/utils/vitpose_model.py`
- [HMR2 features](https://github.com/nvlabs/genmo/blob/main/gem/network/hmr2/hmr2.py): `gem/network/hmr2/hmr2.py`
- [Motion rendering](https://github.com/nvlabs/genmo/blob/main/gem/utils/vis/renderer.py): `gem/utils/vis/renderer.py`
- **SMPL and video outputs**

### Motion modeling

- [GEM model](https://github.com/nvlabs/genmo/blob/main/gem/gem.py): `gem/gem.py`
- [Inference pipeline](https://github.com/nvlabs/genmo/blob/main/gem/pipeline/gem_pipeline.py): `gem/pipeline/gem_pipeline.py`
- [Motion denoiser](https://github.com/nvlabs/genmo/blob/main/gem/network/gem_denoiser.py): `gem/network/gem_denoiser.py`
- [Diffusion sampler](https://github.com/nvlabs/genmo/blob/main/gem/network/gem_diffusion.py): `gem/network/gem_diffusion.py`
- [Gaussian diffusion](https://github.com/nvlabs/genmo/blob/main/gem/diffusion_utils/gaussian_diffusion.py): `gem/diffusion_utils/gaussian_diffusion.py`

### Training data

- [Data assembly](https://github.com/nvlabs/genmo/blob/main/gem/datamodule/mocap_trainX_testY.py): `gem/datamodule/mocap_trainX_testY.py`
- [Motion datasets](https://github.com/nvlabs/genmo/tree/main/gem/datasets): `gem/datasets`

### Training and evaluation

- [Training entry](https://github.com/nvlabs/genmo/blob/main/scripts/train.py): `scripts/train.py`
- [Metrics and callbacks](https://github.com/nvlabs/genmo/tree/main/gem/callbacks): `gem/callbacks`
- **Model checkpoint**

### Motion representation

- [Motion encoding](https://github.com/nvlabs/genmo/blob/main/gem/network/endecoder.py): `gem/network/endecoder.py`
- [Motion postprocess](https://github.com/nvlabs/genmo/blob/main/gem/utils/postprocess.py): `gem/utils/postprocess.py`
- [SMPL body model](https://github.com/nvlabs/genmo/blob/main/gem/utils/body_model/smplx_lite.py): `gem/utils/body_model/smplx_lite.py`

### Other components

- **User / video source**
- **Hugging Face**

## Connections

- User / video source → Multimodal demo: provides video/text
- User / video source → Video-only demo: provides video
- User / video source → Webcam demo: provides stream
- Multimodal demo → GEM model: runs inference
- Video-only demo → GEM model: runs inference
- Webcam demo → YOLOX detection: detects people
- Webcam demo → ViTPose keypoints: estimates keypoints
- Webcam demo → HMR2 features: extracts features
- Webcam demo → GEM model: streams motion input
- GEM model → HMR2 features: extracts image features
- GEM model → Inference pipeline: invokes
- Inference pipeline → Motion denoiser: predicts motion
- Inference pipeline → Diffusion sampler: samples motion
- Diffusion sampler → Gaussian diffusion: uses diffusion steps
- Inference pipeline → Motion encoding: decodes prediction
- Inference pipeline → Motion postprocess: post-processes motion
- Motion postprocess → SMPL body model: uses body geometry
- Multimodal demo → Motion rendering: renders results
- Video-only demo → Motion rendering: renders results
- Webcam demo → Motion rendering: renders stream
- Motion rendering → SMPL and video outputs: writes visualization
- Multimodal demo → SMPL and video outputs: saves parameters
- Training entry → Data assembly: configures loaders
- Data assembly → Motion datasets: instantiates datasets
- Training entry → GEM model: trains model
- GEM model → Metrics and callbacks: reports metrics
- GEM model → Model checkpoint: saves weights
- Hugging Face → Multimodal demo: provides checkpoint