How gengzigang/pct works
Project Type and Purpose
• This is a research‐oriented deep learning codebase (“Human Pose as Compositional Tokens”) for 2D human‐pose estimation.
• It integrates with MMDetect/MMCV for person detection and MMPose ideas, implements a two‐stage PCT (tokenizer + classifier) training pipeline, and supplies demo/visualization tools.
File Structure Analysis
• Top level:
• configs/ ← model & training configurations (tokenizer/classifier, size variants)
• models/ ← PyTorch model definitions (builder, modules, head, loss, tokenizer)
• tools/ ← shell+Python scripts for distributed training/testing
• utils/ ← helper functions (checkpoint loading, optimizer scheduling)
• vis_tools/ ← visualization scripts leveraging MMDet demos
• demo/ ← sample GIFs
• requirements.txt
• Patterns:
• Config‐driven pipeline (all hyperparams & architectures defined in configs/*.py)
• Modular model code: backbone (SimMIM/heatmap), tokenizer, classifier head, loss
• Standard train→validate→test flow via tools/{dist_train.sh,dist_test.sh}
• Dependencies: PyTorch, MMCV, MMDet, COCO dataset, external pre‐trained weights (SimMIM, HRNet)
Key Insights from README
• Two‐stage training:
Stage I: train PCT tokenizer (configs/pct_*_tokenizer.py)
Stage II: train PCT classifier (configs/pct_*_classifier.py)
• Demo uses vis_tools/demo_img_with_mmdet.py with a Cascade R-CNN detector + PCT classifier
• External artifacts: COCO JSONs, detection results, pre‐trained weights
Main Components
A. Data Preparation
• COCO images & annotations
• HRNet person‐detection JSONs
B. Configuration Layer
• configs/*.py
• Defines data pipeline, model architecture, optimizer, schedule
C. Model Components
• Backbone (SimMIM‐pretrained swin or heatmap‐trained swin)
• Tokenizer module (VQ-VAE style or coord‐only)
• Classifier head (composition of tokens to keypoints)
• Loss functions (pct_loss.py)
D. Training & Testing Pipeline
• tools/dist_train.sh → calls tools/train.py
• tools/dist_test.sh → calls tools/test.py
• utils/ckpt_load.py, utils/optimzer_ld.py
E. Visualization & Demo
• vis_tools/demo_img_with_mmdet.py
• vis_tools/cascade_rcnn_*_coco.py for detector config
F. External Dependencies
• PyTorch, MMCV, mmdet, COCO dataset, SimMIM & HRNet weights
Relationships & Data Flow
1) Data Preparation → configs data pipeline → DataLoader
2) DataLoader → Backbone → Tokenizer → Classifier Head → Loss → Optimizer
3) Checkpoint saving/loading via utils/ckpt_load.py
4) Stage I checkpoint → Stage II input
5) After training: dist_test → outputs predictions
6) vis_tools ingest predictions + original images → generated demos
Architectural Patterns & Principles
• Config‐driven modularity: swap backbones, tokenizers, classifier heads by editing configs
• Two‐stage pipeline: training tokenizer then classifier
• Reuse of external frameworks (MMCV/MMDet) for detection & backbones
• Separation of concerns: models/, configs/, tools/, utils/, vis_tools/
Final Advice
• Essential components without overcomplicating—focus on the two‐stage PCT pipeline, its integration with MMDet/MMCV, and the demo path.