# gengzigang/pct architecture

> Architecture diagram of the GitHub repository gengzigang/pct, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gengzigang/pct
- Repository: https://github.com/gengzigang/pct

## Overview

##### Project Type and Purpose
- This is a research‐oriented deep learning codebase (“Human Pose as Compositional Tokens”) for 2D human‐pose estimation.
- It integrates with MMDetect/MMCV for person detection and MMPose ideas, implements a two‐stage PCT (tokenizer + classifier) training pipeline, and supplies demo/visualization tools.

##### File Structure Analysis
- Top level:
- configs/         ← model & training configurations (tokenizer/classifier, size variants)
- models/          ← PyTorch model definitions (builder, modules, head, loss, tokenizer)
- tools/           ← shell+Python scripts for distributed training/testing
- utils/           ← helper functions (checkpoint loading, optimizer scheduling)
- vis_tools/       ← visualization scripts leveraging MMDet demos
- demo/            ← sample GIFs
- requirements.txt
- Patterns:
- Config‐driven pipeline (all hyperparams & architectures defined in configs/*.py)
- Modular model code: backbone (SimMIM/heatmap), tokenizer, classifier head, loss
- Standard train→validate→test flow via tools/{dist_train.sh,dist_test.sh}
- Dependencies: PyTorch, MMCV, MMDet, COCO dataset, external pre‐trained weights (SimMIM, HRNet)

##### Key Insights from README
- Two‐stage training:
Stage I: train PCT tokenizer (configs/pct_*_tokenizer.py)
Stage II: train PCT classifier (configs/pct_*_classifier.py)
- Demo uses vis_tools/demo_img_with_mmdet.py with a Cascade R-CNN detector + PCT classifier
- External artifacts: COCO JSONs, detection results, pre‐trained weights

##### Main Components
A. Data Preparation
- COCO images & annotations
- HRNet person‐detection JSONs
B. Configuration Layer
- configs/*.py
- Defines data pipeline, model architecture, optimizer, schedule
C. Model Components
- Backbone (SimMIM‐pretrained swin or heatmap‐trained swin)
- Tokenizer module (VQ-VAE style or coord‐only)
- Classifier head (composition of tokens to keypoints)
- Loss functions (pct_loss.py)
D. Training & Testing Pipeline
- tools/dist_train.sh → calls tools/train.py
- tools/dist_test.sh → calls tools/test.py
- utils/ckpt_load.py, utils/optimzer_ld.py
E. Visualization & Demo
- vis_tools/demo_img_with_mmdet.py
- vis_tools/cascade_rcnn_*_coco.py for detector config
F. External Dependencies
- PyTorch, MMCV, mmdet, COCO dataset, SimMIM & HRNet weights

##### Relationships & Data Flow
1) Data Preparation → configs data pipeline → DataLoader
2) DataLoader → Backbone → Tokenizer → Classifier Head → Loss → Optimizer
3) Checkpoint saving/loading via utils/ckpt_load.py
4) Stage I checkpoint → Stage II input
5) After training: dist_test → outputs predictions
6) vis_tools ingest predictions + original images → generated demos

##### Architectural Patterns & Principles
- Config‐driven modularity: swap backbones, tokenizers, classifier heads by editing configs
- Two‐stage pipeline: training tokenizer then classifier
- Reuse of external frameworks (MMCV/MMDet) for detection & backbones
- Separation of concerns: models/, configs/, tools/, utils/, vis_tools/

##### Final Advice
- Essential components without overcomplicating—focus on the two‐stage PCT pipeline, its integration with MMDet/MMCV, and the demo path.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layer 1: External Data & Pre-trained Weights
    subgraph "Layer 1: External Data & Weights"
        direction TB
        COCO["COCO Dataset"]:::data
        HRNetJSON["HRNet Person-Detection JSONs"]:::data
        PretrainedWeights["SimMIM & HRNet Weights"]:::ext
        ExternalLibs["PyTorch / MMCV / MMDet"]:::ext
    end

    %% Layer 2: Configuration Layer
    subgraph "Layer 2: Configuration Layer"
        direction TB
        Configs["configs/"]:::config
        TokenizerCfg["pct_*_tokenizer.py"]:::config
        ClassifierCfg["pct_*_classifier.py"]:::config
    end

    %% Layer 3: Model Components
    subgraph "Layer 3: Model Components"
        direction TB
        Builder["builder.py & modules.py"]:::code
        Backbone["pct_swin_v2.py (Backbone)"]:::code
        Tokenizer["pct_tokenizer.py"]:::code
        Head["pct_head.py"]:::code
        Loss["pct_loss.py"]:::code
        Detector["pct_detector.py"]:::code
    end

    %% Layer 4: Training & Testing Scripts
    subgraph "Layer 4: Training & Testing Scripts & Utils"
        direction TB
        DistTrain["dist_train.sh"]:::script
        TrainPy["train.py"]:::script
        DistTest["dist_test.sh"]:::script
        TestPy["test.py"]:::script
        CKPTLoad["ckpt_load.py"]:::code
        OptSched["optimzer_ld.py"]:::code
    end

    %% Layer 5: Visualization & Demo Outputs
    subgraph "Layer 5: Visualization & Demo"
        direction TB
        DemoScript["demo_img_with_mmdet.py"]:::script
        CascadeCfg["cascade_rcnn_x101_64x4d_fpn_coco.py"]:::config
        Predictions["predictions.json"]:::data
        DemoOutput["Demo GIFs"]:::data
    end

    %% Connections: Data to Config
    COCO -->|"images & annotations"| Configs
    HRNetJSON -->|"detection JSONs"| Configs
    PretrainedWeights -->|"load weights"| Backbone
    ExternalLibs -->|"frameworks"| TrainPy

    %% Config to Model Components
    Configs -->|"defines pipeline & arch"| Builder
    TokenizerCfg -->|"Stage I config"| TrainPy
    ClassifierCfg -->|"Stage II config"| TrainPy

    %% Training Flow
    TrainPy -->|"loads configs & data"| Builder
    Builder -->|"instantiates backbone"| Backbone
    Backbone -->|"extract features"| Tokenizer
    Tokenizer -->|"generate tokens"| Head
    Head -->|"predict keypoints"| Loss
    Loss -->|"compute loss"| OptSched
    OptSched -->|"update weights"| Backbone
    Loss -->|"save checkpoint"| CKPTLoad
    CKPTLoad -->|"epoch_*.pth"| TrainPy

    %% Two-stage flow
    TokenizerCfg -->|"checkpoint for stage II"| ClassifierCfg
    CKPTLoad -->|"tokenizer checkpoint"| ClassifierCfg

    %% Testing Flow
    DistTest -->|"calls test.py"| TestPy
    TestPy -->|"loads model & data"| CKPTLoad
    TestPy -->|"outputs"| Predictions

    %% Visualization Flow
    Predictions -->|"input to"| DemoScript
    COCO -->|"original images"| DemoScript
    CascadeCfg -->|"detector config"| DemoScript
    DemoScript -->|"generates"| DemoOutput

    %% Click Events
    click Configs "https://github.com/gengzigang/pct/tree/main/configs/"
    click TokenizerCfg "https://github.com/gengzigang/pct/blob/main/configs/pct_*_tokenizer.py"
    click ClassifierCfg "https://github.com/gengzigang/pct/blob/main/configs/pct_*_classifier.py"
    click Backbone "https://github.com/gengzigang/pct/blob/main/models/pct_swin_v2.py"
    click Tokenizer "https://github.com/gengzigang/pct/blob/main/models/pct_tokenizer.py"
    click Head "https://github.com/gengzigang/pct/blob/main/models/pct_head.py"
    click Loss "https://github.com/gengzigang/pct/blob/main/models/pct_loss.py"
    click Builder "https://github.com/gengzigang/pct/blob/main/models/builder.py"
    click Builder "https://github.com/gengzigang/pct/blob/main/models/modules.py"
    click Detector "https://github.com/gengzigang/pct/blob/main/models/pct_detector.py"
    click DistTrain "https://github.com/gengzigang/pct/blob/main/tools/dist_train.sh"
    click TrainPy "https://github.com/gengzigang/pct/blob/main/tools/train.py"
    click DistTest "https://github.com/gengzigang/pct/blob/main/tools/dist_test.sh"
    click TestPy "https://github.com/gengzigang/pct/blob/main/tools/test.py"
    click CKPTLoad "https://github.com/gengzigang/pct/blob/main/utils/ckpt_load.py"
    click OptSched "https://github.com/gengzigang/pct/blob/main/utils/optimzer_ld.py"
    click DemoScript "https://github.com/gengzigang/pct/blob/main/vis_tools/demo_img_with_mmdet.py"
    click CascadeCfg "https://github.com/gengzigang/pct/blob/main/vis_tools/cascade_rcnn_x101_64x4d_fpn_coco.py"

    %% Styles
    classDef code fill:#cce5ff,stroke:#3399ff
    classDef data fill:#d4edda,stroke:#28a745
    classDef config fill:#fff3cd,stroke:#ffcd39
    classDef ext fill:#e2e3e5,stroke:#6c757d
    classDef script fill:#f0f0f0,stroke:#999999
```
