How linghaochan/humanmac works
This repository is a PyTorch-based research codebase for HumanMAC, a human motion prediction system built around a diffusion model. Architecturally, it is organized into four main subsystems:
1. experiment orchestration
2. data loading and motion representation
3. model definition and sampling
4. training, evaluation, and visualization utilities
A separate preprocessing toolchain handles AMASS-to-Human3.6M retargeting for zero-shot experiments.
1) Top-level execution flow
The central entry point is main.py. It acts as the mode switch for the whole repo:
• train runs model optimization
• eval runs benchmark evaluation
• pred runs motion prediction visualization
• switch runs motion-switch visualizations
• control runs controllable prediction visualizations
• zero_shot runs AMASS zero-shot inference
config.py and the YAML files in cfg/ define dataset-specific experiment settings for:
• h36m
• humaneva
These config files are the boundary between the command line and the rest of the system. They determine dataset paths, sequence lengths, evaluation settings, and likely model/training hyperparameters.
The repository assumes a Python + GPU runtime, with install.sh setting up dependencies. The implementation is clearly centered on deep learning training/inference rather than serving or deployment.
2) Data subsystem
The data_loader/ package is the dataset boundary. It translates raw benchmark files into model-ready tensors and hides dataset-specific conventions.
Dataset modules
• dataset.py provides shared dataset abstractions
• dataset_h36m.py and dataset_humaneva.py implement standard benchmark loading
• dataset_h36m_multimodal.py and dataset_humaneva_multimodal.py handle multi-modal candidate data used for controllable or diverse prediction settings
Skeleton representation
• skeleton.py encodes the joint hierarchy / kinematic structure and motion layout
This is important because the model operates on human pose sequences, not generic vectors. Skeleton conventions affect:
• normalization and coordinate transforms
• joint indexing
• visualization
• retargeting and evaluation
External data sources
The repo consumes benchmark data prepared outside the codebase:
• Human3.6M
• HumanEva-I
• AMASS for zero-shot experiments
The README indicates the expected data lives under ./data, including standard .npz benchmark files and a retargeted AMASS .npy file.
Data flow
Raw dataset files → dataset loaders in data_loader/ → batched motion sequences → model input
For multimodal and zero-shot cases, the data loaders also supply candidate futures or retargeted motion sequences used during inference and evaluation.
3) Model subsystem
The models/ directory contains the core predictive model.
Core components
• diffusion.py is the main motion generation model
• transformer.py likely provides the sequence modeling backbone used by the diffusion process
The architectural idea, based on the README, is:
• training: learn to denoise motion sequences from noise
• inference: condition on observed past motion and sample plausible futures
This is the main conceptual boundary of the repo: unlike classic encoder-decoder motion prediction, HumanMAC is diffusion-based and supports diverse outputs, motion switching, and controllable completion.
Model behavior in the system
The model is used in several regimes:
• standard prediction on Human3.6M / HumanEva-I
• multi-sample prediction for diversity metrics and visualization
• motion switching, where sampled futures can change category
• controllable prediction, where certain body parts are constrained
• zero-shot transfer to AMASS-retargeted motion
So the model layer is not just a single forward pass; it includes iterative sampling/denoising logic that is central to the architecture.
4) Training, evaluation, logging, and visualization
The utils/ package contains most of the orchestration logic around the model.
Training pipeline
• training.py likely contains the training loop, optimization steps, checkpointing, and epoch management
• logger.py handles experiment logs
• script.py and util.py likely provide shared helpers for file management, config handling, seeding, and experiment setup
• torch.py likely wraps PyTorch-specific utilities
Training produces:
• checkpoints in a run-specific models/ directory
• logs in log/
• generated GIFs in out/
• metrics in results/
The README’s run directory structure shows that each experiment is isolated under a unique results/inference folder, which is an important architectural boundary for reproducibility.
Evaluation
• evaluation.py and metrics.py compute motion prediction metrics
The README specifically mentions:
• APD
• ADE
• FDE
• MMADE
• MMFDE
This indicates the evaluation subsystem is multi-output aware and likely handles both single best prediction quality and sample diversity/coverage. The README also notes this evaluation is parallelized and GPU-dependent.
Visualization
• visualization.py
• demo_visualize.py
These modules generate the animated GIF outputs shown in the README. Visualization is a first-class subsystem because the repo is built not only for metric benchmarking but also for qualitative inspection of:
• normal prediction
• motion switching
• controllable motion completion
• zero-shot AMASS transfer
5) Motion retargeting subsystem
motion-retargeting/ is a distinct preprocessing/analysis area, not part of the main training loop.
It contains scripts for:
• retargeting AMASS skeletons to the Human3.6M skeleton space
• visualizing AMASS and retargeted motions
• validating the retargeting results
Files:
• trans.py
• vis-amass.py
• vis-h36.py
• vis-retar-h36.py
This boundary matters because zero-shot experiments depend on this preprocessing step. In the architecture diagram, this should be shown as an offline data preparation pipeline feeding the main data directory, not as part of online inference.
6) Major runtime/data boundaries
A useful architecture diagram should show these external boundaries:
External datasets
• Human3.6M
• HumanEva-I
• AMASS
External artifacts
• pretrained checkpoints under ./checkpoints
• prepared benchmark data under ./data
Outputs
• training checkpoints
• logs
• evaluation metrics
• GIF visualizations
• inference outputs under ./inference
7) End-to-end data flow
A good way to think about the repo is:
1. User starts main.py with a mode and config
2. Config selects dataset, sequence lengths, and experiment settings
3. Data loaders read benchmark files or retargeted AMASS data
4. Motion sequences are normalized into skeleton-based tensors
5. The diffusion model, backed by transformer components, trains on observed/future motion pairs
6. During inference, the model denoises from noise conditioned on observed motion
7. Outputs go to:
- metrics for evaluation
- checkpoints for reuse
- GIFs for visualization
- inference folders for prediction/switch/control/zero-shot modes
8) What is architecturally most important
If you are drawing the repo, the key components are:
• main.py as the control hub
• config.py + cfg/*.yml as experiment configuration
• data_loader/ as dataset/skeleton abstraction
• models/ as the diffusion-based motion generator
• utils/ as training/eval/visualization infrastructure
• motion-retargeting/ as an offline preprocessing pipeline for zero-shot data
The core architectural idea is that the repo is not a generic ML framework; it is a motion prediction system built around a diffusion sampler conditioned on historical human motion, with special support for multimodal prediction, controllability, and dataset retargeting.
Main entry points: dataset.py Dataset base · motion-retargeting Retargeting tools · main.py main.py · diffusion.py Diffusion model
Components 22
Runtime & config
Entry points and experiment settings
- main.py · Mode switch for train, eval, prediction, switching, control, and zero-shot runs
main.py - config.py · CLI and experiment configuration boundary
config.py - H36M config · Dataset-specific settings for Human3.6M experiments
cfg/h36m.yml - HumanEva config · Dataset-specific settings for HumanEva-I experiments
cfg/humaneva.yml
Data layer
Dataset loading, motion layout, and benchmark boundaries
- Dataset base · Shared dataset interface for motion samples
data_loader/dataset.py - H36M loader · Human3.6M dataset implementation
data_loader/dataset_h36m.py - HumanEva loader · HumanEva-I dataset implementation
data_loader/dataset_humaneva.py - H36M multimodal · Human3.6M loader for diverse or controllable futures
data_loader/dataset_h36m_multimodal.py - HumanEva multimodal · HumanEva-I loader for diverse or controllable futures
data_loader/dataset_humaneva_multimodal.py - Skeleton · Joint hierarchy and motion representation conventions
data_loader/skeleton.py
Model layer
Diffusion-based motion prediction components
- Diffusion model · Main denoising-based human motion predictor
models/diffusion.py - Transformer · Temporal feature backbone used by the diffusion model
models/transformer.py
Training & inference
Optimization, evaluation, and visualization flows
- Training loop · Epoch management, checkpointing, and optimization steps
utils/training.py - Evaluation · Benchmark evaluation over sampled motion predictions
utils/evaluation.py - Metrics · APD, ADE, FDE, MMADE, and MMFDE computations
utils/metrics.py - Visualization · GIF generation and qualitative motion inspection
utils/visualization.py - Demo visualizer · Helper entry for visual demos and generated outputs
utils/demo_visualize.py
Retargeting
Offline AMASS to Human3.6M preprocessing
- Retargeting tools · AMASS to Human3.6M retargeting and inspection scripts
motion-retargeting - Retarget transform · Skeleton transfer from AMASS space to Human3.6M space
motion-retargeting/trans.py - AMASS viz · Visualizes source AMASS motions
motion-retargeting/vis-amass.py - H36M viz · Visualizes Human3.6M target motions
motion-retargeting/vis-h36.py - Retarget viz · Visualizes retargeted Human3.6M motions
motion-retargeting/vis-retar-h36.py
Connections 25
- main.py → config.py: loads · Selects runtime settings and experiment mode
- config.py → H36M config: selects · Human3.6M experiment configuration
- config.py → HumanEva config: selects · HumanEva-I experiment configuration
- main.py → Training loop: train · Starts optimization workflow
- main.py → Evaluation: eval · Runs benchmark evaluation workflow
- main.py → Visualization: pred/switch/control · Runs qualitative generation and rendering modes
- main.py → Retargeting tools: zero-shot · Uses retargeted data path for AMASS-based inference
- Training loop → Diffusion model: optimizes · Trains the generative motion model
- Evaluation → Diffusion model: samples · Evaluates model outputs under benchmark settings
- Visualization → Diffusion model: renders · Uses sampled predictions for visual output
- Diffusion model → Transformer: uses backbone · Temporal representation and conditioning support
- Training loop → Dataset base: consumes · Reads motion batches through shared dataset abstractions
- Dataset base → H36M loader: implements · Human3.6M standard loader
- Dataset base → HumanEva loader: implements · HumanEva-I standard loader
- Dataset base → H36M multimodal: implements · Human3.6M multimodal loader
- Dataset base → HumanEva multimodal: implements · HumanEva-I multimodal loader
- Dataset base → Skeleton: uses · Joint indexing and motion layout
- Evaluation → Metrics: computes · Derives APD, ADE, FDE, MMADE, MMFDE
- Visualization → Demo visualizer: supports · Shared demo rendering entrypoints
- Retargeting tools → Retarget transform: contains · Offline retargeting implementation
- Retargeting tools → AMASS viz: contains · Source motion inspection
- Retargeting tools → H36M viz: contains · Target skeleton inspection
- Retargeting tools → Retarget viz: contains · Retargeted motion inspection
- Retarget transform → Skeleton: aligns with · Maps motion into Human3.6M skeleton conventions
- Retargeting tools → Dataset base: feeds · Produces preprocessing outputs consumed by zero-shot loaders