How idea-research/x-pose works
Project Type and Purpose
• X-Pose (formerly UniPose) is an end-to-end, prompt-based keypoint detection framework built in Python.
• It provides both a command-line interface (inference_on_a_image.py) and a web demo (app.py via Gradio) to run inference on arbitrary images using textual or visual prompts.
File-Structure Analysis
• Top-Level Entrypoints
– inference_on_a_image.py: CLI script for single-image inference
– app.py: Gradio-based web interface
– config_model/: model configuration files (UniPose_SwinT.py, coco_transformer.py)
– models/UniPose/: core model code, including backbone definitions, attention modules, deformable transformers, fusion modules, position encodings, and custom CUDA/CPU ops
– models/registry.py: factory/registry for instantiating models based on config
– predefined_keypoints.py: textual prompt definitions for keypoint skeletons
– transforms.py and util/: preprocessing, post-processing, logging, metrics, plotting, etc.
Key Components
A. Presentation Layer
• CLI Inference (inference_on_a_image.py)
• Web Demo (app.py + Gradio)
B. Configuration & Registry
• Config files (config_model/*.py) drive hyperparameters, backbone choice, transformer depths
• Registry (models/registry.py) maps config → model class
C. Data Flow & Preprocessing
• Image & Prompt Input
– Image transforms (transforms.py)
– Prompt loader (predefined_keypoints.py for text; placeholder for visual prompts)
D. Model Core
1. Prompt Encoder
– Textual: tokenization of keypoint descriptions
– Visual: (future extension) ROI crop, feature extractor
2. Backbone
– Swin Transformer (models/UniPose/swin_transformer.py)
3. Feature Fusion Modules
– Fuse image features with prompt embeddings (fuse_modules.py)
4. Positional Encoding
– position_encoding.py
5. Deformable Transformer
– Encoder/Decoder stacks (deformable_transformer.py)
– Attention heads (attention.py)
6. Output Head
– Keypoint heatmap or coordinate regression (unipose.py, mask_generate.py)
E. Custom Operators
• models/UniPose/ops/: CPU & CUDA implementations of multi-scale deformable attention; compiled via setup.py
F. Post-Processing & Output
• Thresholding, NMS, mapping back to image coordinates (util/keypoint_ops.py, util/plot_utils.py)
• Saving annotated image to disk or returning via Gradio
Relationships & Data Flow
1. User → CLI/Web UI
2. CLI/Web UI → Load config & model (registry)
3. CLI/Web UI → Preprocessing (transforms + prompt loader)
4. Preprocessed data → Prompt Encoder & Backbone
5. Backbone features + prompt embeddings → Feature Fusion → Deformable Transformer → Output Head
6. Output head → Post-processing → Annotated image
Architectural Patterns & Principles
• Modular pipeline: each stage (preprocess, encode, fuse, transform, head) is a separate module
• Config-driven instantiation: decouple code from hyperparameters
• Plugin/registry for adding new backbones or heads without changing entry scripts
• Custom ops isolated under models/UniPose/ops for performance
Technologies & Frameworks
• Python 3.x, PyTorch
• NVIDIA CUDA (custom C++/CUDA extensions)
• Gradio for web UI
• Standard ML utilities: logging, metrics, plotting