# linghaochan/humanmac architecture

> Architecture diagram of the GitHub repository linghaochan/humanmac, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/linghaochan/humanmac
- Repository: https://github.com/linghaochan/humanmac

## Overview

This repository is a PyTorch-based research codebase for HumanMAC, a human motion prediction system built around a diffusion model. Architecturally, it is organized into four main subsystems:

1. experiment orchestration
2. data loading and motion representation
3. model definition and sampling
4. training, evaluation, and visualization utilities

A separate preprocessing toolchain handles AMASS-to-Human3.6M retargeting for zero-shot experiments.

#### 1) Top-level execution flow

The central entry point is `main.py`. It acts as the mode switch for the whole repo:

- `train` runs model optimization
- `eval` runs benchmark evaluation
- `pred` runs motion prediction visualization
- `switch` runs motion-switch visualizations
- `control` runs controllable prediction visualizations
- `zero_shot` runs AMASS zero-shot inference

`config.py` and the YAML files in `cfg/` define dataset-specific experiment settings for:

- `h36m`
- `humaneva`

These config files are the boundary between the command line and the rest of the system. They determine dataset paths, sequence lengths, evaluation settings, and likely model/training hyperparameters.

The repository assumes a Python + GPU runtime, with `install.sh` setting up dependencies. The implementation is clearly centered on deep learning training/inference rather than serving or deployment.

#### 2) Data subsystem

The `data_loader/` package is the dataset boundary. It translates raw benchmark files into model-ready tensors and hides dataset-specific conventions.

##### Dataset modules
- `dataset.py` provides shared dataset abstractions
- `dataset_h36m.py` and `dataset_humaneva.py` implement standard benchmark loading
- `dataset_h36m_multimodal.py` and `dataset_humaneva_multimodal.py` handle multi-modal candidate data used for controllable or diverse prediction settings

##### Skeleton representation
- `skeleton.py` encodes the joint hierarchy / kinematic structure and motion layout

This is important because the model operates on human pose sequences, not generic vectors. Skeleton conventions affect:

- normalization and coordinate transforms
- joint indexing
- visualization
- retargeting and evaluation

##### External data sources
The repo consumes benchmark data prepared outside the codebase:

- Human3.6M
- HumanEva-I
- AMASS for zero-shot experiments

The README indicates the expected data lives under `./data`, including standard `.npz` benchmark files and a retargeted AMASS `.npy` file.

##### Data flow
Raw dataset files → dataset loaders in `data_loader/` → batched motion sequences → model input

For multimodal and zero-shot cases, the data loaders also supply candidate futures or retargeted motion sequences used during inference and evaluation.

#### 3) Model subsystem

The `models/` directory contains the core predictive model.

##### Core components
- `diffusion.py` is the main motion generation model
- `transformer.py` likely provides the sequence modeling backbone used by the diffusion process

The architectural idea, based on the README, is:

- training: learn to denoise motion sequences from noise
- inference: condition on observed past motion and sample plausible futures

This is the main conceptual boundary of the repo: unlike classic encoder-decoder motion prediction, HumanMAC is diffusion-based and supports diverse outputs, motion switching, and controllable completion.

##### Model behavior in the system
The model is used in several regimes:

- standard prediction on Human3.6M / HumanEva-I
- multi-sample prediction for diversity metrics and visualization
- motion switching, where sampled futures can change category
- controllable prediction, where certain body parts are constrained
- zero-shot transfer to AMASS-retargeted motion

So the model layer is not just a single forward pass; it includes iterative sampling/denoising logic that is central to the architecture.

#### 4) Training, evaluation, logging, and visualization

The `utils/` package contains most of the orchestration logic around the model.

##### Training pipeline
- `training.py` likely contains the training loop, optimization steps, checkpointing, and epoch management
- `logger.py` handles experiment logs
- `script.py` and `util.py` likely provide shared helpers for file management, config handling, seeding, and experiment setup
- `torch.py` likely wraps PyTorch-specific utilities

Training produces:
- checkpoints in a run-specific `models/` directory
- logs in `log/`
- generated GIFs in `out/`
- metrics in `results/`

The README’s run directory structure shows that each experiment is isolated under a unique results/inference folder, which is an important architectural boundary for reproducibility.

##### Evaluation
- `evaluation.py` and `metrics.py` compute motion prediction metrics

The README specifically mentions:
- APD
- ADE
- FDE
- MMADE
- MMFDE

This indicates the evaluation subsystem is multi-output aware and likely handles both single best prediction quality and sample diversity/coverage. The README also notes this evaluation is parallelized and GPU-dependent.

##### Visualization
- `visualization.py`
- `demo_visualize.py`

These modules generate the animated GIF outputs shown in the README. Visualization is a first-class subsystem because the repo is built not only for metric benchmarking but also for qualitative inspection of:
- normal prediction
- motion switching
- controllable motion completion
- zero-shot AMASS transfer

#### 5) Motion retargeting subsystem

`motion-retargeting/` is a distinct preprocessing/analysis area, not part of the main training loop.

It contains scripts for:
- retargeting AMASS skeletons to the Human3.6M skeleton space
- visualizing AMASS and retargeted motions
- validating the retargeting results

Files:
- `trans.py`
- `vis-amass.py`
- `vis-h36.py`
- `vis-retar-h36.py`

This boundary matters because zero-shot experiments depend on this preprocessing step. In the architecture diagram, this should be shown as an offline data preparation pipeline feeding the main data directory, not as part of online inference.

#### 6) Major runtime/data boundaries

A useful architecture diagram should show these external boundaries:

##### External datasets
- Human3.6M
- HumanEva-I
- AMASS

##### External artifacts
- pretrained checkpoints under `./checkpoints`
- prepared benchmark data under `./data`

##### Outputs
- training checkpoints
- logs
- evaluation metrics
- GIF visualizations
- inference outputs under `./inference`

#### 7) End-to-end data flow

A good way to think about the repo is:

1. User starts `main.py` with a mode and config
2. Config selects dataset, sequence lengths, and experiment settings
3. Data loaders read benchmark files or retargeted AMASS data
4. Motion sequences are normalized into skeleton-based tensors
5. The diffusion model, backed by transformer components, trains on observed/future motion pairs
6. During inference, the model denoises from noise conditioned on observed motion
7. Outputs go to:
   - metrics for evaluation
   - checkpoints for reuse
   - GIFs for visualization
   - inference folders for prediction/switch/control/zero-shot modes

#### 8) What is architecturally most important

If you are drawing the repo, the key components are:

- `main.py` as the control hub
- `config.py` + `cfg/*.yml` as experiment configuration
- `data_loader/` as dataset/skeleton abstraction
- `models/` as the diffusion-based motion generator
- `utils/` as training/eval/visualization infrastructure
- `motion-retargeting/` as an offline preprocessing pipeline for zero-shot data

The core architectural idea is that the repo is not a generic ML framework; it is a motion prediction system built around a diffusion sampler conditioned on historical human motion, with special support for multimodal prediction, controllability, and dataset retargeting.

Main entry points: [`data_loader/dataset.py`](https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset.py) (Dataset base), [`motion-retargeting`](https://github.com/linghaochan/humanmac/tree/main/motion-retargeting) (Retargeting tools), [`main.py`](https://github.com/linghaochan/humanmac/blob/main/main.py) (main.py), [`models/diffusion.py`](https://github.com/linghaochan/humanmac/blob/main/models/diffusion.py) (Diffusion model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_runtime["Runtime & config"]
  node_main_py["main.py<br/>entrypoint<br/>[main.py]"]
  node_config_py["config.py<br/>[config.py]"]
  node_cfg_h36m["H36M config<br/>yaml config<br/>[h36m.yml]"]
  node_cfg_humaneva["HumanEva config<br/>yaml config<br/>[humaneva.yml]"]
end

subgraph group_group_data["Data layer"]
  node_dataset_base["Dataset base<br/>dataset abstraction<br/>[dataset.py]"]
  node_h36m_ds["H36M loader<br/>benchmark loader<br/>[dataset_h36m.py]"]
  node_humaneva_ds["HumanEva loader<br/>benchmark loader"]
  node_h36m_mm_ds["H36M multimodal<br/>multi-sample loader"]
  node_humaneva_mm_ds["HumanEva multimodal<br/>multi-sample loader"]
  node_skeleton_py["Skeleton<br/>kinematics schema<br/>[skeleton.py]"]
end

subgraph group_group_model["Model layer"]
  node_diffusion_py{{"Diffusion model<br/>motion generator<br/>[diffusion.py]"}}
  node_transformer_py["Transformer<br/>sequence backbone<br/>[transformer.py]"]
end

subgraph group_group_pipeline["Training & inference"]
  node_training_py["Training loop<br/>optimization pipeline<br/>[training.py]"]
  node_evaluation_py["Evaluation<br/>metrics pipeline<br/>[evaluation.py]"]
  node_metrics_py["Metrics<br/>[metrics.py]"]
  node_visualization_py["Visualization<br/>rendering pipeline<br/>[visualization.py]"]
  node_demo_visualize_py["Demo visualizer<br/>demo runner<br/>[demo_visualize.py]"]
end

subgraph group_group_retarget["Retargeting"]
  node_motion_retargeting["Retargeting tools<br/>offline preprocessing"]
  node_trans_py["Retarget transform<br/>[trans.py]"]
  node_vis_amass_py["AMASS viz<br/>retarget visualization<br/>[vis-amass.py]"]
  node_vis_h36_py["H36M viz<br/>retarget visualization<br/>[vis-h36.py]"]
  node_vis_retar_h36_py["Retarget viz<br/>retarget visualization<br/>[vis-retar-h36.py]"]
end

node_main_py -->|"loads"| node_config_py
node_config_py -->|"selects"| node_cfg_h36m
node_config_py -->|"selects"| node_cfg_humaneva
node_main_py -->|"train"| node_training_py
node_main_py -->|"eval"| node_evaluation_py
node_main_py -->|"pred/switch/control"| node_visualization_py
node_main_py -.->|"zero-shot"| node_motion_retargeting
node_training_py -->|"optimizes"| node_diffusion_py
node_evaluation_py -->|"samples"| node_diffusion_py
node_visualization_py -->|"renders"| node_diffusion_py
node_diffusion_py -->|"uses backbone"| node_transformer_py
node_training_py -->|"consumes"| node_dataset_base
node_dataset_base -->|"implements"| node_h36m_ds
node_dataset_base -->|"implements"| node_humaneva_ds
node_dataset_base -->|"implements"| node_h36m_mm_ds
node_dataset_base -->|"implements"| node_humaneva_mm_ds
node_dataset_base -->|"uses"| node_skeleton_py
node_evaluation_py -->|"computes"| node_metrics_py
node_visualization_py -->|"supports"| node_demo_visualize_py
node_motion_retargeting -->|"contains"| node_trans_py
node_motion_retargeting -->|"contains"| node_vis_amass_py
node_motion_retargeting -->|"contains"| node_vis_h36_py
node_motion_retargeting -->|"contains"| node_vis_retar_h36_py
node_trans_py -->|"aligns with"| node_skeleton_py
node_motion_retargeting -.->|"feeds"| node_dataset_base

click node_main_py "https://github.com/linghaochan/humanmac/blob/main/main.py"
click node_config_py "https://github.com/linghaochan/humanmac/blob/main/config.py"
click node_cfg_h36m "https://github.com/linghaochan/humanmac/blob/main/cfg/h36m.yml"
click node_cfg_humaneva "https://github.com/linghaochan/humanmac/blob/main/cfg/humaneva.yml"
click node_dataset_base "https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset.py"
click node_h36m_ds "https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset_h36m.py"
click node_humaneva_ds "https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset_humaneva.py"
click node_h36m_mm_ds "https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset_h36m_multimodal.py"
click node_humaneva_mm_ds "https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset_humaneva_multimodal.py"
click node_skeleton_py "https://github.com/linghaochan/humanmac/blob/main/data_loader/skeleton.py"
click node_diffusion_py "https://github.com/linghaochan/humanmac/blob/main/models/diffusion.py"
click node_transformer_py "https://github.com/linghaochan/humanmac/blob/main/models/transformer.py"
click node_training_py "https://github.com/linghaochan/humanmac/blob/main/utils/training.py"
click node_evaluation_py "https://github.com/linghaochan/humanmac/blob/main/utils/evaluation.py"
click node_metrics_py "https://github.com/linghaochan/humanmac/blob/main/utils/metrics.py"
click node_visualization_py "https://github.com/linghaochan/humanmac/blob/main/utils/visualization.py"
click node_demo_visualize_py "https://github.com/linghaochan/humanmac/blob/main/utils/demo_visualize.py"
click node_motion_retargeting "https://github.com/linghaochan/humanmac/tree/main/motion-retargeting"
click node_trans_py "https://github.com/linghaochan/humanmac/blob/main/motion-retargeting/trans.py"
click node_vis_amass_py "https://github.com/linghaochan/humanmac/blob/main/motion-retargeting/vis-amass.py"
click node_vis_h36_py "https://github.com/linghaochan/humanmac/blob/main/motion-retargeting/vis-h36.py"
click node_vis_retar_h36_py "https://github.com/linghaochan/humanmac/blob/main/motion-retargeting/vis-retar-h36.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_py,node_config_py,node_cfg_h36m,node_cfg_humaneva toneBlue
class node_dataset_base,node_h36m_ds,node_humaneva_ds,node_h36m_mm_ds,node_humaneva_mm_ds,node_skeleton_py toneAmber
class node_diffusion_py,node_transformer_py toneMint
class node_training_py,node_evaluation_py,node_metrics_py,node_visualization_py,node_demo_visualize_py toneRose
class node_motion_retargeting,node_trans_py,node_vis_amass_py,node_vis_h36_py,node_vis_retar_h36_py toneIndigo
```

## Components

### Runtime & config

Entry points and experiment settings

- [main.py](https://github.com/linghaochan/humanmac/blob/main/main.py): entrypoint · Mode switch for train, eval, prediction, switching, control, and zero-shot runs · `main.py`
- [config.py](https://github.com/linghaochan/humanmac/blob/main/config.py): config · CLI and experiment configuration boundary · `config.py`
- [H36M config](https://github.com/linghaochan/humanmac/blob/main/cfg/h36m.yml): yaml config · Dataset-specific settings for Human3.6M experiments · `cfg/h36m.yml`
- [HumanEva config](https://github.com/linghaochan/humanmac/blob/main/cfg/humaneva.yml): yaml config · Dataset-specific settings for HumanEva-I experiments · `cfg/humaneva.yml`

### Data layer

Dataset loading, motion layout, and benchmark boundaries

- [Dataset base](https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset.py): dataset abstraction · Shared dataset interface for motion samples · `data_loader/dataset.py`
- [H36M loader](https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset_h36m.py): benchmark loader · Human3.6M dataset implementation · `data_loader/dataset_h36m.py`
- [HumanEva loader](https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset_humaneva.py): benchmark loader · HumanEva-I dataset implementation · `data_loader/dataset_humaneva.py`
- [H36M multimodal](https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset_h36m_multimodal.py): multi-sample loader · Human3.6M loader for diverse or controllable futures · `data_loader/dataset_h36m_multimodal.py`
- [HumanEva multimodal](https://github.com/linghaochan/humanmac/blob/main/data_loader/dataset_humaneva_multimodal.py): multi-sample loader · HumanEva-I loader for diverse or controllable futures · `data_loader/dataset_humaneva_multimodal.py`
- [Skeleton](https://github.com/linghaochan/humanmac/blob/main/data_loader/skeleton.py): kinematics schema · Joint hierarchy and motion representation conventions · `data_loader/skeleton.py`

### Model layer

Diffusion-based motion prediction components

- [Diffusion model](https://github.com/linghaochan/humanmac/blob/main/models/diffusion.py): motion generator · Main denoising-based human motion predictor · `models/diffusion.py`
- [Transformer](https://github.com/linghaochan/humanmac/blob/main/models/transformer.py): sequence backbone · Temporal feature backbone used by the diffusion model · `models/transformer.py`

### Training & inference

Optimization, evaluation, and visualization flows

- [Training loop](https://github.com/linghaochan/humanmac/blob/main/utils/training.py): optimization pipeline · Epoch management, checkpointing, and optimization steps · `utils/training.py`
- [Evaluation](https://github.com/linghaochan/humanmac/blob/main/utils/evaluation.py): metrics pipeline · Benchmark evaluation over sampled motion predictions · `utils/evaluation.py`
- [Metrics](https://github.com/linghaochan/humanmac/blob/main/utils/metrics.py): motion metrics · APD, ADE, FDE, MMADE, and MMFDE computations · `utils/metrics.py`
- [Visualization](https://github.com/linghaochan/humanmac/blob/main/utils/visualization.py): rendering pipeline · GIF generation and qualitative motion inspection · `utils/visualization.py`
- [Demo visualizer](https://github.com/linghaochan/humanmac/blob/main/utils/demo_visualize.py): demo runner · Helper entry for visual demos and generated outputs · `utils/demo_visualize.py`

### Retargeting

Offline AMASS to Human3.6M preprocessing

- [Retargeting tools](https://github.com/linghaochan/humanmac/tree/main/motion-retargeting): offline preprocessing · AMASS to Human3.6M retargeting and inspection scripts · `motion-retargeting`
- [Retarget transform](https://github.com/linghaochan/humanmac/blob/main/motion-retargeting/trans.py): retarget transform · Skeleton transfer from AMASS space to Human3.6M space · `motion-retargeting/trans.py`
- [AMASS viz](https://github.com/linghaochan/humanmac/blob/main/motion-retargeting/vis-amass.py): retarget visualization · Visualizes source AMASS motions · `motion-retargeting/vis-amass.py`
- [H36M viz](https://github.com/linghaochan/humanmac/blob/main/motion-retargeting/vis-h36.py): retarget visualization · Visualizes Human3.6M target motions · `motion-retargeting/vis-h36.py`
- [Retarget viz](https://github.com/linghaochan/humanmac/blob/main/motion-retargeting/vis-retar-h36.py): retarget visualization · Visualizes retargeted Human3.6M motions · `motion-retargeting/vis-retar-h36.py`

## Connections

- main.py → config.py: loads · Selects runtime settings and experiment mode
- config.py → H36M config: selects · Human3.6M experiment configuration
- config.py → HumanEva config: selects · HumanEva-I experiment configuration
- main.py → Training loop: train · Starts optimization workflow
- main.py → Evaluation: eval · Runs benchmark evaluation workflow
- main.py → Visualization: pred/switch/control · Runs qualitative generation and rendering modes
- main.py → Retargeting tools: zero-shot · Uses retargeted data path for AMASS-based inference
- Training loop → Diffusion model: optimizes · Trains the generative motion model
- Evaluation → Diffusion model: samples · Evaluates model outputs under benchmark settings
- Visualization → Diffusion model: renders · Uses sampled predictions for visual output
- Diffusion model → Transformer: uses backbone · Temporal representation and conditioning support
- Training loop → Dataset base: consumes · Reads motion batches through shared dataset abstractions
- Dataset base → H36M loader: implements · Human3.6M standard loader
- Dataset base → HumanEva loader: implements · HumanEva-I standard loader
- Dataset base → H36M multimodal: implements · Human3.6M multimodal loader
- Dataset base → HumanEva multimodal: implements · HumanEva-I multimodal loader
- Dataset base → Skeleton: uses · Joint indexing and motion layout
- Evaluation → Metrics: computes · Derives APD, ADE, FDE, MMADE, MMFDE
- Visualization → Demo visualizer: supports · Shared demo rendering entrypoints
- Retargeting tools → Retarget transform: contains · Offline retargeting implementation
- Retargeting tools → AMASS viz: contains · Source motion inspection
- Retargeting tools → H36M viz: contains · Target skeleton inspection
- Retargeting tools → Retarget viz: contains · Retargeted motion inspection
- Retarget transform → Skeleton: aligns with · Maps motion into Human3.6M skeleton conventions
- Retargeting tools → Dataset base: feeds · Produces preprocessing outputs consumed by zero-shot loaders
