# twehrbein/footmr architecture

> Architecture diagram of the GitHub repository twehrbein/footmr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/twehrbein/footmr
- Repository: https://github.com/twehrbein/footmr

## Overview

FootMR refines 3D foot motion in monocular human-motion capture. The demo workflow preprocesses a video into tracked-person, 2D-pose, and camera-motion cues, obtains an initial human recovery, then lifts foot-keypoint sequences to refined 3D motion and renders results. Training and evaluation form a separate supported branch using motion and image-feature datasets and metrics on MOOF, MOYO, and RICH. The graph preserves documented capabilities where implementation excerpts are incomplete; some subsystem links therefore cite the README rather than sampled call sites.

Main entry points: [`hmr4d/model/footmr/pipeline/footmr_pipeline.py`](https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/pipeline/footmr_pipeline.py) (FootMR Pipeline), [`tools/demo.py`](https://github.com/twehrbein/footmr/blob/main/tools/demo.py) (Demo Runner), [`tools/train.py`](https://github.com/twehrbein/footmr/blob/main/tools/train.py) (Training Runner), [`hmr4d/datamodule/mocap_trainX_testY.py`](https://github.com/twehrbein/footmr/blob/main/hmr4d/datamodule/mocap_trainX_testY.py) (Motion Data Module)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_demo["Video Reconstruction"]
  node_demo_cli["Demo Runner<br/>[demo.py]"]
  node_tracker["Person Tracking<br/>[tracker.py]"]
  node_pose["2D Pose Extraction<br/>[vitpose.py]"]
  node_sapiens["Sapiens Pose<br/>[sapiens.py]"]
  node_slam["Camera Motion<br/>[slam.py]"]
  node_initial_recovery["Initial Recovery"]
  node_demo_model["Demo Model Adapter<br/>[footmr_pl_demo.py]"]
  node_renderer["Motion Rendering<br/>[renderer.py]"]
  node_video_output["Rendered Video<br/>[video_io_utils.py]"]
end

subgraph group_refine["Foot Refinement"]
  node_pipeline["FootMR Pipeline<br/>[footmr_pipeline.py]"]
  node_foot_network["Foot Transformer"]
  node_encoding["Motion Encoding<br/>[endecoder.py]"]
  node_motion_stats["Motion Statistics<br/>[stats_compose.py]"]
  node_postprocess["Motion Postprocess<br/>[postprocess.py]"]
end

subgraph group_training["Training and Evaluation"]
  node_train_cli["Training Runner<br/>[train.py]"]
  node_datamodule["Motion Data Module"]
  node_image_datasets["Image-Feature Data<br/>[base_dataset.py]"]
  node_motion_datasets["Motion Datasets"]
  node_footmr_module["FootMR Trainer<br/>[footmr_pl.py]"]
  node_metrics["Benchmark Metrics"]
  node_evaluation_utils["Evaluation Utilities<br/>[eval_utils.py]"]
end

subgraph group_geometry["Motion Foundations"]
  node_body_model["SMPL Body Model<br/>[smplx_utils.py]"]
  node_camera_geometry["Camera Geometry<br/>[hmr_cam.py]"]
  node_global_geometry["Global Coordinates<br/>[hmr_global.py]"]
  node_rotation_math["Rotation Math<br/>[matrix.py]"]
end

node_user(("Video User"))

node_user -->|"provides video"| node_demo_cli
node_demo_cli -->|"tracks person"| node_tracker
node_demo_cli -->|"extracts pose"| node_pose
node_demo_cli -.->|"optionally extracts"| node_sapiens
node_demo_cli -->|"estimates motion"| node_slam
node_demo_cli -->|"gets initial recovery"| node_initial_recovery
node_demo_cli -->|"runs refinement"| node_demo_model
node_demo_model -.->|"invokes pipeline"| node_pipeline
node_pipeline -->|"predicts motion"| node_foot_network
node_pipeline -->|"decodes output"| node_encoding
node_pipeline -->|"normalizes with"| node_motion_stats
node_pipeline -->|"postprocesses"| node_postprocess
node_pipeline -->|"computes translation"| node_camera_geometry
node_pipeline -->|"converts coordinates"| node_global_geometry
node_encoding -->|"uses statistics"| node_motion_stats
node_foot_network -.->|"outputs body motion"| node_body_model
node_pipeline -.->|"supplies refined motion"| node_renderer
node_demo_cli -->|"renders results"| node_renderer
node_renderer -->|"writes frames"| node_video_output
node_train_cli -->|"configures training"| node_datamodule
node_datamodule -.->|"loads image features"| node_image_datasets
node_datamodule -.->|"loads motion data"| node_motion_datasets
node_train_cli -->|"trains model"| node_footmr_module
node_footmr_module -.->|"uses refinement"| node_pipeline
node_train_cli -->|"runs benchmarks"| node_metrics
node_metrics -->|"computes metrics"| node_evaluation_utils
node_camera_geometry -->|"uses transforms"| node_rotation_math
node_global_geometry -->|"uses transforms"| node_rotation_math

click node_demo_cli "https://github.com/twehrbein/footmr/blob/main/tools/demo.py"
click node_tracker "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/preproc/tracker.py"
click node_pose "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/preproc/vitpose.py"
click node_sapiens "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/preproc/sapiens.py"
click node_slam "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/preproc/slam.py"
click node_initial_recovery "https://github.com/twehrbein/footmr/tree/main/hmr4d/network/hmr2"
click node_demo_model "https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/footmr_pl_demo.py"
click node_renderer "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/vis/renderer.py"
click node_video_output "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/video_io_utils.py"
click node_pipeline "https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/pipeline/footmr_pipeline.py"
click node_foot_network "https://github.com/twehrbein/footmr/blob/main/hmr4d/network/footmr/foot_transformer.py"
click node_encoding "https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/utils/endecoder.py"
click node_motion_stats "https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/utils/stats_compose.py"
click node_postprocess "https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/utils/postprocess.py"
click node_body_model "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/smplx_utils.py"
click node_camera_geometry "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/geo/hmr_cam.py"
click node_global_geometry "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/geo/hmr_global.py"
click node_rotation_math "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/matrix.py"
click node_train_cli "https://github.com/twehrbein/footmr/blob/main/tools/train.py"
click node_datamodule "https://github.com/twehrbein/footmr/blob/main/hmr4d/datamodule/mocap_trainX_testY.py"
click node_image_datasets "https://github.com/twehrbein/footmr/blob/main/hmr4d/dataset/imgfeat_motion/base_dataset.py"
click node_motion_datasets "https://github.com/twehrbein/footmr/tree/main/hmr4d/dataset/pure_motion"
click node_footmr_module "https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/footmr_pl.py"
click node_metrics "https://github.com/twehrbein/footmr/tree/main/hmr4d/model/footmr/callbacks"
click node_evaluation_utils "https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/eval/eval_utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_demo_cli,node_tracker,node_pose,node_sapiens,node_slam,node_initial_recovery,node_demo_model,node_renderer,node_video_output,node_user toneBlue
class node_pipeline,node_foot_network,node_encoding,node_motion_stats,node_postprocess toneAmber
class node_train_cli,node_datamodule,node_image_datasets,node_motion_datasets,node_footmr_module,node_metrics,node_evaluation_utils toneMint
class node_body_model,node_camera_geometry,node_global_geometry,node_rotation_math toneRose
```

## Components

### Video Reconstruction

- [Demo Runner](https://github.com/twehrbein/footmr/blob/main/tools/demo.py): `tools/demo.py`
- [Person Tracking](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/preproc/tracker.py): `hmr4d/utils/preproc/tracker.py`
- [2D Pose Extraction](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/preproc/vitpose.py): `hmr4d/utils/preproc/vitpose.py`
- [Sapiens Pose](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/preproc/sapiens.py): `hmr4d/utils/preproc/sapiens.py`
- [Camera Motion](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/preproc/slam.py): `hmr4d/utils/preproc/slam.py`
- [Initial Recovery](https://github.com/twehrbein/footmr/tree/main/hmr4d/network/hmr2): `hmr4d/network/hmr2`
- [Demo Model Adapter](https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/footmr_pl_demo.py): `hmr4d/model/footmr/footmr_pl_demo.py`
- [Motion Rendering](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/vis/renderer.py): `hmr4d/utils/vis/renderer.py`
- [Rendered Video](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/video_io_utils.py): `hmr4d/utils/video_io_utils.py`

### Foot Refinement

- [FootMR Pipeline](https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/pipeline/footmr_pipeline.py): `hmr4d/model/footmr/pipeline/footmr_pipeline.py`
- [Foot Transformer](https://github.com/twehrbein/footmr/blob/main/hmr4d/network/footmr/foot_transformer.py): `hmr4d/network/footmr/foot_transformer.py`
- [Motion Encoding](https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/utils/endecoder.py): `hmr4d/model/footmr/utils/endecoder.py`
- [Motion Statistics](https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/utils/stats_compose.py): `hmr4d/model/footmr/utils/stats_compose.py`
- [Motion Postprocess](https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/utils/postprocess.py): `hmr4d/model/footmr/utils/postprocess.py`

### Training and Evaluation

- [Training Runner](https://github.com/twehrbein/footmr/blob/main/tools/train.py): `tools/train.py`
- [Motion Data Module](https://github.com/twehrbein/footmr/blob/main/hmr4d/datamodule/mocap_trainX_testY.py): `hmr4d/datamodule/mocap_trainX_testY.py`
- [Image-Feature Data](https://github.com/twehrbein/footmr/blob/main/hmr4d/dataset/imgfeat_motion/base_dataset.py): `hmr4d/dataset/imgfeat_motion/base_dataset.py`
- [Motion Datasets](https://github.com/twehrbein/footmr/tree/main/hmr4d/dataset/pure_motion): `hmr4d/dataset/pure_motion`
- [FootMR Trainer](https://github.com/twehrbein/footmr/blob/main/hmr4d/model/footmr/footmr_pl.py): `hmr4d/model/footmr/footmr_pl.py`
- [Benchmark Metrics](https://github.com/twehrbein/footmr/tree/main/hmr4d/model/footmr/callbacks): `hmr4d/model/footmr/callbacks`
- [Evaluation Utilities](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/eval/eval_utils.py): `hmr4d/utils/eval/eval_utils.py`

### Motion Foundations

- [SMPL Body Model](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/smplx_utils.py): `hmr4d/utils/smplx_utils.py`
- [Camera Geometry](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/geo/hmr_cam.py): `hmr4d/utils/geo/hmr_cam.py`
- [Global Coordinates](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/geo/hmr_global.py): `hmr4d/utils/geo/hmr_global.py`
- [Rotation Math](https://github.com/twehrbein/footmr/blob/main/hmr4d/utils/matrix.py): `hmr4d/utils/matrix.py`

### Other components

- **Video User**

## Connections

- Video User → Demo Runner: provides video
- Demo Runner → Person Tracking: tracks person (evidence: `tools/demo.py`)
- Demo Runner → 2D Pose Extraction: extracts pose (evidence: `tools/demo.py`)
- Demo Runner → Sapiens Pose: optionally extracts (evidence: `tools/demo.py`)
- Demo Runner → Camera Motion: estimates motion (evidence: `tools/demo.py`)
- Demo Runner → Initial Recovery: gets initial recovery
- Demo Runner → Demo Model Adapter: runs refinement (evidence: `tools/demo.py`)
- Demo Model Adapter → FootMR Pipeline: invokes pipeline
- FootMR Pipeline → Foot Transformer: predicts motion (evidence: `hmr4d/model/footmr/pipeline/footmr_pipeline.py`)
- FootMR Pipeline → Motion Encoding: decodes output (evidence: `hmr4d/model/footmr/pipeline/footmr_pipeline.py`)
- FootMR Pipeline → Motion Statistics: normalizes with (evidence: `hmr4d/model/footmr/pipeline/footmr_pipeline.py`)
- FootMR Pipeline → Motion Postprocess: postprocesses (evidence: `hmr4d/model/footmr/pipeline/footmr_pipeline.py`)
- FootMR Pipeline → Camera Geometry: computes translation (evidence: `hmr4d/model/footmr/pipeline/footmr_pipeline.py`)
- FootMR Pipeline → Global Coordinates: converts coordinates (evidence: `hmr4d/model/footmr/pipeline/footmr_pipeline.py`)
- Motion Encoding → Motion Statistics: uses statistics (evidence: `hmr4d/model/footmr/utils/endecoder.py`)
- Foot Transformer → SMPL Body Model: outputs body motion
- FootMR Pipeline → Motion Rendering: supplies refined motion
- Demo Runner → Motion Rendering: renders results (evidence: `tools/demo.py`)
- Motion Rendering → Rendered Video: writes frames (evidence: `tools/demo.py`)
- Training Runner → Motion Data Module: configures training
- Motion Data Module → Image-Feature Data: loads image features
- Motion Data Module → Motion Datasets: loads motion data
- Training Runner → FootMR Trainer: trains model
- FootMR Trainer → FootMR Pipeline: uses refinement
- Training Runner → Benchmark Metrics: runs benchmarks
- Benchmark Metrics → Evaluation Utilities: computes metrics (evidence: `hmr4d/model/footmr/callbacks/metric_moof.py`)
- Camera Geometry → Rotation Math: uses transforms (evidence: `hmr4d/utils/geo/hmr_cam.py`)
- Global Coordinates → Rotation Math: uses transforms (evidence: `hmr4d/utils/geo/hmr_global.py`)
