# saidwivedi/tokenhmr architecture

> Architecture diagram of the GitHub repository saidwivedi/tokenhmr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saidwivedi/tokenhmr
- Repository: https://github.com/saidwivedi/tokenhmr

## Overview

##### Project type and purpose
- TokenHMR is a research-oriented, open-source computer-vision & machine-learning tool for 3D human mesh recovery. It comprises two major pipelines:
a. Tokenization (VQ-style pose token learning)
b. TokenHMR (human mesh recovery using tokenized poses and Threshold-Adaptive Loss Scaling)

##### File structure
- Top-level directories:
- tokenization/   ← code, configs, data loaders & training scripts for the tokenizer (Method a)
- tokenhmr/       ← code, Hydra configs, data loaders, models, train/eval/demo scripts for the HMR network (Method b)
- demo_sample/    ← sample images & video for quick demos
- data/           ← SMPL/SMPLH body-model files & downloaded checkpoints
- assets/         ← figures, teaser gifs
- tokenhmr/lib/   ← organized into configs, datasets, models, utils
- tokenization/…  ← split into configs, dataset, models, utils, scripts

##### README for additional insights
- Two-stage method:
1) Tokenization: continuous SMPLH poses → discrete tokens
2) TokenHMR: image/video → keypoints (Detectron2/PHALP) → pose tokens → mesh parameters
- Heavy use of Hydra for config management
- External dependencies: PyTorch, Detectron2, a fork of PHALP
- Data sources: AMASS, MOYO (tokenization), BEDLAM, 4DHumans, 3DPW, EMDB (HMR)

##### Don’t overthink—capture all essential components
- Labels match directory names for quick reference
- How scripts (train_poseVQ.py, train.py, eval.py, demo.py, track.py) orchestrate pipelines
- Checkpoints flow from training modules to demo modules

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Storage
    subgraph "Data Storage"
        RawPoseDS[(Raw Pose Datasets)]:::storage
        RawImgDS[(Raw Image/Video Datasets)]:::storage
        BodyModels[(Pretrained Body Models)]:::storage
        Checkpoints[(Checkpoints & Configs)]:::storage
    end

    %% Tokenization Pipeline
    subgraph "Tokenization Pipeline"
        Loader["Data Loader"]:::process
        VQVAE["vanilla_pose_vqvae.py"]:::process
        QuantCNN["quantize_cnn.py"]:::process
        ResNet["resnet.py"]:::process
        LossFunc["TALS Loss Function"]:::process
        Trainer["train_poseVQ.py"]:::process
        Vocabulary[(Pose Token Vocabulary)]:::storage
    end

    %% TokenHMR Pipeline
    subgraph "TokenHMR Pipeline"
        BedlamDS["bedlam_dataset_tar.py"]:::process
        EMDBDS["emdb_dataset.py"]:::process
        ImageDS["image_dataset.py"]:::process
        MoCapDS["mocap_dataset.py"]:::process
        VitDetDS["vitdet_dataset.py"]:::process
        Backbone["vit.py"]:::process
        PoseTrans["pose_transformer.py"]:::process
        TokenHead["token_classifier.py"]:::process
        SMPLHead["smpl_head.py"]:::process
        THMRLoss["losses.py"]:::process
        HydraConfig["Hydra Config Manager"]:::process
        THMRTrainer["train.py"]:::process
        THMREval["eval.py"]:::process
        MeshParams[(Mesh Parameters)]:::storage
    end

    %% Demo/Inference
    subgraph "Demo/Inference"
        ImageDemo["demo.py"]:::process
        VideoDemo["track.py"]:::process
        MeshRender["mesh_renderer.py"]:::process
        DetectUtils["utils_detectron2.py"]:::process
    end

    %% External Services/Libraries
    subgraph "External Services/Libraries"
        Detectron2["Detectron2"]:::external
        PHALP["PHALP"]:::external
    end

    %% Data Storage -> Tokenization
    RawPoseDS -->|"pose npz → tokenization"| Loader
    Loader --> VQVAE
    Loader --> QuantCNN
    Loader --> ResNet
    VQVAE --> LossFunc
    QuantCNN --> LossFunc
    ResNet --> LossFunc
    LossFunc --> Trainer
    Trainer --> Vocabulary

    %% Data Storage & Tokenization -> TokenHMR
    RawImgDS --> BedlamDS
    RawImgDS --> EMDBDS
    RawImgDS --> ImageDS
    RawPoseDS --> MoCapDS
    RawImgDS --> VitDetDS
    Vocabulary --> BedlamDS
    Vocabulary --> EMDBDS
    Vocabulary --> ImageDS
    Vocabulary --> MoCapDS
    Vocabulary --> VitDetDS
    BodyModels --> BedlamDS
    BodyModels --> EMDBDS
    BodyModels --> ImageDS
    BodyModels --> MoCapDS
    BodyModels --> VitDetDS

    BedlamDS --> Backbone
    EMDBDS --> Backbone
    ImageDS --> Backbone
    MoCapDS --> Backbone
    VitDetDS --> Backbone
    Backbone --> PoseTrans
    PoseTrans --> TokenHead
    TokenHead --> SMPLHead
    SMPLHead --> THMRLoss
    THMRLoss --> THMRTrainer
    HydraConfig --> THMRTrainer
    THMRTrainer --> MeshParams
    THMRTrainer --> THMREval

    %% TokenHMR -> Demo
    MeshParams -->|"checkpoint"| ImageDemo
    MeshParams -->|"checkpoint"| VideoDemo
    ImageDemo --> Detectron2
    VideoDemo --> PHALP
    ImageDemo --> MeshRender
    VideoDemo --> MeshRender
    DetectUtils --> ImageDemo

    %% Click Events
    click Loader "https://github.com/saidwivedi/tokenhmr/blob/main/tokenization/dataset/dataset_poseVQ.py"
    click VQVAE "https://github.com/saidwivedi/tokenhmr/blob/main/tokenization/models/vanilla_pose_vqvae.py"
    click QuantCNN "https://github.com/saidwivedi/tokenhmr/blob/main/tokenization/models/quantize_cnn.py"
    click ResNet "https://github.com/saidwivedi/tokenhmr/blob/main/tokenization/models/resnet.py"
    click LossFunc "https://github.com/saidwivedi/tokenhmr/blob/main/tokenization/utils/losses.py"
    click Trainer "https://github.com/saidwivedi/tokenhmr/blob/main/tokenization/train_poseVQ.py"

    click BedlamDS "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/datasets/bedlam_dataset_tar.py"
    click EMDBDS "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/datasets/emdb_dataset.py"
    click ImageDS "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/datasets/image_dataset.py"
    click MoCapDS "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/datasets/mocap_dataset.py"
    click VitDetDS "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/datasets/vitdet_dataset.py"

    click Backbone "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/models/backbones/vit.py"
    click PoseTrans "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/models/components/pose_transformer.py"
    click TokenHead "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/models/heads/token_classifier.py"
    click SMPLHead "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/models/heads/smpl_head.py"
    click THMRLoss "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/models/losses.py"
    click HydraConfig "https://github.com/saidwivedi/tokenhmr/tree/main/tokenhmr/lib/configs_hydra/"
    click THMRTrainer "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/train.py"
    click THMREval "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/eval.py"

    click ImageDemo "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/demo.py"
    click VideoDemo "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/track.py"
    click MeshRender "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/utils/mesh_renderer.py"
    click DetectUtils "https://github.com/saidwivedi/tokenhmr/blob/main/tokenhmr/lib/utils/utils_detectron2.py"

    %% Styles
    classDef storage fill:#bbdefb,stroke:#333,stroke-width:1px
    classDef process fill:#c8e6c9,stroke:#333,stroke-width:1px
    classDef external fill:#ffe0b2,stroke:#333,stroke-width:1px
```
