# idea-research/x-pose architecture

> Architecture diagram of the GitHub repository idea-research/x-pose, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/idea-research/x-pose
- Repository: https://github.com/idea-research/x-pose

## Overview

##### Project Type and Purpose
- X-Pose (formerly UniPose) is an end-to-end, prompt-based keypoint detection framework built in Python.
- It provides both a command-line interface (`inference_on_a_image.py`) and a web demo (`app.py` via Gradio) to run inference on arbitrary images using textual or visual prompts.

##### File-Structure Analysis
- Top-Level Entrypoints
– `inference_on_a_image.py`: CLI script for single-image inference
– `app.py`: Gradio-based web interface
– `config_model/`: model configuration files (`UniPose_SwinT.py`, `coco_transformer.py`)
– `models/UniPose/`: core model code, including backbone definitions, attention modules, deformable transformers, fusion modules, position encodings, and custom CUDA/CPU ops
– `models/registry.py`: factory/registry for instantiating models based on config
– `predefined_keypoints.py`: textual prompt definitions for keypoint skeletons
– `transforms.py` and `util/`: preprocessing, post-processing, logging, metrics, plotting, etc.

##### Key Components
A. Presentation Layer
- CLI Inference (`inference_on_a_image.py`)
- Web Demo (`app.py` + Gradio)
B. Configuration & Registry
- Config files (`config_model/*.py`) drive hyperparameters, backbone choice, transformer depths
- Registry (`models/registry.py`) maps config → model class
C. Data Flow & Preprocessing
- Image & Prompt Input
– Image transforms (`transforms.py`)
– Prompt loader (`predefined_keypoints.py` for text; placeholder for visual prompts)
D. Model Core
1. Prompt Encoder
– Textual: tokenization of keypoint descriptions
– Visual: (future extension) ROI crop, feature extractor
2. Backbone
– Swin Transformer (`models/UniPose/swin_transformer.py`)
3. Feature Fusion Modules
– Fuse image features with prompt embeddings (`fuse_modules.py`)
4. Positional Encoding
– `position_encoding.py`
5. Deformable Transformer
– Encoder/Decoder stacks (`deformable_transformer.py`)
– Attention heads (`attention.py`)
6. Output Head
– Keypoint heatmap or coordinate regression (`unipose.py`, `mask_generate.py`)
E. Custom Operators
- `models/UniPose/ops/`: CPU & CUDA implementations of multi-scale deformable attention; compiled via `setup.py`
F. Post-Processing & Output
- Thresholding, NMS, mapping back to image coordinates (`util/keypoint_ops.py`, `util/plot_utils.py`)
- Saving annotated image to disk or returning via Gradio

##### Relationships & Data Flow
1. User → CLI/Web UI
2. CLI/Web UI → Load config & model (registry)
3. CLI/Web UI → Preprocessing (transforms + prompt loader)
4. Preprocessed data → Prompt Encoder & Backbone
5. Backbone features + prompt embeddings → Feature Fusion → Deformable Transformer → Output Head
6. Output head → Post-processing → Annotated image

##### Architectural Patterns & Principles
- Modular pipeline: each stage (preprocess, encode, fuse, transform, head) is a separate module
- Config-driven instantiation: decouple code from hyperparameters
- Plugin/registry for adding new backbones or heads without changing entry scripts
- Custom ops isolated under `models/UniPose/ops` for performance

##### Technologies & Frameworks
- Python 3.x, PyTorch
- NVIDIA CUDA (custom C++/CUDA extensions)
- Gradio for web UI
- Standard ML utilities: logging, metrics, plotting

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Dependencies
    PyTorch["PyTorch"]:::external
    CUDA["CUDA & C++ Ext"]:::external
    Gradio["Gradio"]:::external

    %% Presentation Layer
    subgraph "Presentation Layer"
        CLI["CLI Inference\n(inference_on_a_image.py)"]:::presentation
        Web["Web Demo\n(app.py)"]:::presentation
    end

    %% Configuration & Registry
    subgraph "Configuration & Registry"
        Config["Config Files\n(UniPose_SwinT.py,\ncoco_transformer.py)"]:::config
        Registry["Model Registry\n(models/registry.py)"]:::config
    end

    %% Preprocessing & Prompts
    subgraph "Preprocessing & Prompts"
        Transforms["Image Transforms\n(transforms.py)"]:::preprocess
        TextPrompts["Textual Prompts\n(predefined_keypoints.py)"]:::preprocess
        UtilImgs["Prompt/Image Utils\n(util/utils.py,\nutil/box_ops.py)"]:::preprocess
    end

    %% Model Core
    subgraph "Model Core"
        subgraph "Prompt Encoder"
            TextEncoder["Text Encoder\n(models/UniPose/utils.py)"]:::core
            VisualEncoder["Visual Prompt Encoder\n(placeholder)"]:::core
        end
        Backbone["Swin Transformer\n(models/UniPose/swin_transformer.py)"]:::core
        Fusion["Feature Fusion\n(models/UniPose/fuse_modules.py)"]:::core
        PosEnc["Positional Encoding\n(models/UniPose/position_encoding.py)"]:::core
        DeformTrans["Deformable Transformer\n(models/UniPose/deformable_transformer.py)"]:::core
        Attention["Attention Heads\n(models/UniPose/attention.py)"]:::core
        AltTrans["Vanilla Transformer\n(models/UniPose/transformer_vanilla.py)"]:::core
        OutputKP["Keypoint Head\n(models/UniPose/unipose.py)"]:::core
        OutputMask["Mask/Heatmap Head\n(models/UniPose/mask_generate.py)"]:::core
    end

    %% Custom Operators
    subgraph "Custom Operators"
        SetupOps["Setup\n(models/UniPose/ops/setup.py)"]:::ops
        OpsFuncs["MS Deform Attn Func\n(ops/functions/ms_deform_attn_func.py)"]:::ops
        OpsMod1["MS Deform Attn Module\n(ms_deform_attn.py)"]:::ops
        OpsMod2["Key-aware Attn Module\n(ms_deform_attn_key_aware.py)"]:::ops
        OpsCPUSrc["CPU Src\n(src/cpu/ms_deform_attn_cpu.cpp/.h)"]:::ops
        OpsCUDASrc["CUDA Src\n(src/cuda/ms_deform_attn_cuda.cu/.h,\nsrc/cuda/ms_deform_im2col_cuda.cuh)"]:::ops
    end

    %% Post-Processing & Output
    subgraph "Post-Processing & Output"
        PostKP["Keypoint Ops\n(util/keypoint_ops.py)"]:::post
        Plot["Plot Utils\n(util/plot_utils.py)"]:::post
        Logger["Logging & Metrics\n(util/logger.py,\nutil/metrics.py)"]:::post
        ConfUtil["Config Utils\n(util/config.py,\nutil/get_param_dicts.py)"]:::post
    end

    %% Flow Connections
    PyTorch -->|"runs on"| CUDA
    CLI -->|uses| Config
    CLI -->|calls| Registry
    Web -->|via Gradio| Gradio
    Web -->|uses| Config
    Web -->|calls| Registry

    CLI -->|loads| Transforms
    CLI -->|loads| TextPrompts
    CLI -->|loads| UtilImgs
    Web -->|loads| Transforms
    Web -->|loads| TextPrompts
    Web -->|loads| UtilImgs

    Transforms -->|prepares image| Backbone
    TextPrompts -->|provides tokens| TextEncoder
    UtilImgs -->|provides data| VisualEncoder

    TextEncoder -->|embeddings| Fusion
    VisualEncoder -->|embeddings| Fusion
    Backbone -->|features| Fusion
    Fusion -->|fused features| PosEnc
    PosEnc -->|positioned| DeformTrans
    DeformTrans -->|encoded features| Attention
    Attention -->|attended features| OutputKP
    Attention -->|attended features| OutputMask

    OutputKP -->|raw output| PostKP
    OutputMask -->|raw masks| PostKP
    PostKP -->|processed| Plot
    Plot -->|results| CLI
    Plot -->|results| Web
    Logger -->|logs metrics| CLI
    Logger -->|logs metrics| Web
    ConfUtil -->|parses params| Config

    SetupOps -->|builds| OpsFuncs
    SetupOps -->|builds| OpsMod1
    SetupOps -->|builds| OpsMod2
    OpsFuncs -->|backend| DeformTrans
    OpsMod1 -->|backend| DeformTrans
    OpsMod2 -->|backend| DeformTrans
    OpsCPUSrc -->|implement| OpsMod1
    OpsCUDASrc -->|implement| OpsMod1
    OpsCUDASrc -->|implement| OpsMod2

    %% Click Events
    click CLI "https://github.com/idea-research/x-pose/blob/master/inference_on_a_image.py"
    click Web "https://github.com/idea-research/x-pose/blob/master/app.py"
    click Config "https://github.com/idea-research/x-pose/blob/master/config_model/UniPose_SwinT.py"
    click Config "https://github.com/idea-research/x-pose/blob/master/config_model/coco_transformer.py"
    click Registry "https://github.com/idea-research/x-pose/blob/master/models/registry.py"
    click TextPrompts "https://github.com/idea-research/x-pose/blob/master/predefined_keypoints.py"
    click Transforms "https://github.com/idea-research/x-pose/blob/master/transforms.py"
    click UtilImgs "https://github.com/idea-research/x-pose/blob/master/util/utils.py"
    click UtilImgs "https://github.com/idea-research/x-pose/blob/master/util/box_ops.py"
    click TextEncoder "https://github.com/idea-research/x-pose/blob/master/models/UniPose/utils.py"
    click Backbone "https://github.com/idea-research/x-pose/blob/master/models/UniPose/swin_transformer.py"
    click Fusion "https://github.com/idea-research/x-pose/blob/master/models/UniPose/fuse_modules.py"
    click PosEnc "https://github.com/idea-research/x-pose/blob/master/models/UniPose/position_encoding.py"
    click DeformTrans "https://github.com/idea-research/x-pose/blob/master/models/UniPose/deformable_transformer.py"
    click Attention "https://github.com/idea-research/x-pose/blob/master/models/UniPose/attention.py"
    click AltTrans "https://github.com/idea-research/x-pose/blob/master/models/UniPose/transformer_vanilla.py"
    click OutputKP "https://github.com/idea-research/x-pose/blob/master/models/UniPose/unipose.py"
    click OutputMask "https://github.com/idea-research/x-pose/blob/master/models/UniPose/mask_generate.py"
    click SetupOps "https://github.com/idea-research/x-pose/blob/master/models/UniPose/ops/setup.py"
    click OpsFuncs "https://github.com/idea-research/x-pose/blob/master/models/UniPose/ops/functions/ms_deform_attn_func.py"
    click OpsMod1 "https://github.com/idea-research/x-pose/blob/master/models/UniPose/ops/modules/ms_deform_attn.py"
    click OpsMod2 "https://github.com/idea-research/x-pose/blob/master/models/UniPose/ops/modules/ms_deform_attn_key_aware.py"
    click OpsCPUSrc "https://github.com/idea-research/x-pose/blob/master/models/UniPose/ops/src/cpu/ms_deform_attn_cpu.cpp"
    click OpsCUDASrc "https://github.com/idea-research/x-pose/blob/master/models/UniPose/ops/src/cuda/ms_deform_attn_cuda.cu"

    click PostKP "https://github.com/idea-research/x-pose/blob/master/util/keypoint_ops.py"
    click Plot "https://github.com/idea-research/x-pose/blob/master/util/plot_utils.py"
    click Logger "https://github.com/idea-research/x-pose/blob/master/util/logger.py"
    click Logger "https://github.com/idea-research/x-pose/blob/master/util/metrics.py"
    click ConfUtil "https://github.com/idea-research/x-pose/blob/master/util/config.py"
    click ConfUtil "https://github.com/idea-research/x-pose/blob/master/util/get_param_dicts.py"

    %% Styles
    classDef presentation fill:#FFD580,stroke:#DAA520,color:#000;
    classDef config fill:#B0E0E6,stroke:#4682B4,color:#000;
    classDef preprocess fill:#E6E6FA,stroke:#9370DB,color:#000;
    classDef core fill:#98FB98,stroke:#2E8B57,color:#000;
    classDef ops fill:#FFC0CB,stroke:#FF69B4,color:#000;
    classDef post fill:#F0E68C,stroke:#BDB76B,color:#000;
    classDef external fill:#D3D3D3,stroke:#A9A9A9,color:#000;
```
