# tranxuantuyen/planner-refiner architecture

> Architecture diagram of the GitHub repository tranxuantuyen/planner-refiner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tranxuantuyen/planner-refiner
- Repository: https://github.com/tranxuantuyen/planner-refiner

## Overview

##### Project type and purpose
- This is a research‐oriented PyTorch codebase implementing “Planner-Refiner” for Referring Video Object Segmentation (RVOS) and Video Temporal Grounding (VTG).
- It provides data pipelines, model definitions (backbones, pixel decoder, transformer decoder, matching and loss modules), training/evaluation engines, configuration files, and utility scripts.

##### File structure
- Top‐level packages:
- ITR/ – core implementation of the Planner-Refiner (“ITR”) model
- mask2former/ – external segmentation library integrated as a submodule
- configs/ – YAML configuration files for different experiments
- scripts/ – shell scripts to train, test, and validate
- tools/ – evaluation & checkpoint utilities
- train_net_itr.py – entry point for launching experiments
- ITR/ subpackages mimic an “ML pipeline” layering:
- data/ – dataset definitions, augmentation, data loaders
- modeling/ – model architectures (transformer decoder, criterion, matcher)
- engine/ – training loop, hooks, distributed launch
- evaluation/ – evaluator interface for metrics
- utils/ – miscellaneous helpers
- mask2former/ itself is organized similarly with separate data mappers, backbones, pixel‐decoder, transformer‐decoder, etc.

##### README insights
- Relies on MeViS dataset + custom CUDA kernel for MSDeformAttn
- Requires Anaconda environment, Python 3.8, PyTorch
- Uses pre‐trained backbone weights and external downloads for benchmarks
- Entrypoints: scripts/train.sh, scripts/eval.sh, train_net_itr.py

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        RawData[/"MeViS Video Frames"/]:::data
        Expr[/"Referring Expressions"/]:::data
        Aug["Augmentation"]:::module
        Build["Build Dataset"]:::module
        Combined["Combined Loader"]:::module
        DMap["Dataset Mapper"]:::module
        Mevis["MeViS Dataset"]:::module
        MMappers["mask2former/data/dataset_mappers/"]:::module
        MDatasets["mask2former/data/datasets/"]:::module
        DataLoader["ITR DataLoader"]:::module
    end

    click Aug "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/data/augmentation.py"
    click Build "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/data/build.py"
    click Combined "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/data/combined_loader.py"
    click DMap "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/data/dataset_mapper.py"
    click Mevis "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/data/datasets/mevis.py"
    click MMappers "https://github.com/tranxuantuyen/planner-refiner/tree/main/mask2former/data/dataset_mappers/"
    click MDatasets "https://github.com/tranxuantuyen/planner-refiner/tree/main/mask2former/data/datasets/"

    RawData -->|ingest| Aug
    Expr -->|feed| DMap
    Aug --> Build --> Combined --> DMap --> Mevis --> DataLoader
    MMappers --> DataLoader
    MDatasets --> DataLoader

    %% Model Architecture
    subgraph "Model Architecture"
        Backbone["Backbone\n(resnet.py, swin.py)"]:::module
        PixelDecoder["Pixel Decoder\n(fpn.py, msdeformattn.py)"]:::module
        TransformerDecoder["Planner-Refiner\nTransformer Decoder"]:::module
        Heads["Heads\n(classification & mask)"]:::module
        ITRModel["ITR Model Entry"]:::module
    end

    click Backbone "https://github.com/tranxuantuyen/planner-refiner/blob/main/mask2former/modeling/backbone/resnet.py"
    click Backbone "https://github.com/tranxuantuyen/planner-refiner/blob/main/mask2former/modeling/backbone/swin.py"
    click PixelDecoder "https://github.com/tranxuantuyen/planner-refiner/blob/main/mask2former/modeling/pixel_decoder/fpn.py"
    click PixelDecoder "https://github.com/tranxuantuyen/planner-refiner/blob/main/mask2former/modeling/pixel_decoder/msdeformattn.py"
    click TransformerDecoder "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/modeling/transformer_decoder/itr_transformer_decoder.py"
    click TransformerDecoder "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/modeling/transformer_decoder/itr_decoder_model.py"
    click TransformerDecoder "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/modeling/transformer_decoder/position_encoding.py"
    click TransformerDecoder "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/modeling/transformer_decoder/fuse_modules.py"
    click ITRModel "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/itr_model.py"

    DataLoader --> Backbone --> PixelDecoder --> TransformerDecoder --> Heads --> ITRModel

    %% Matching & Loss
    subgraph "Matching & Loss"
        Matcher["Hungarian Matcher"]:::module
        Criterion["Loss Criterion"]:::module
    end

    click Matcher "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/modeling/vita_matcher.py"
    click Criterion "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/modeling/vita_criterion.py"

    ITRModel --> Matcher
    ITRModel --> Criterion
    Matcher --> Criterion

    %% Training Engine
    subgraph "Training Engine"
        Launch["Distributed Launch"]:::module
        TrainLoop["Train Loop"]:::module
        Hooks["Hooks & Logging"]:::module
        Defaults["Engine Defaults"]:::module
    end

    click Launch "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/engine/launch.py"
    click TrainLoop "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/engine/train_loop.py"
    click Hooks "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/engine/hooks.py"
    click Defaults "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/engine/defaults.py"

    Criterion --> TrainLoop
    Matcher --> TrainLoop
    ITRModel --> TrainLoop
    Defaults --> Launch --> TrainLoop
    Hooks --> TrainLoop

    %% Evaluation
    subgraph "Evaluation"
        Evaluator["ITR Evaluator"]:::module
        EvalTool["tools/eval_mevis.py"]:::module
        ValScript["scripts/val.sh"]:::script
    end

    click Evaluator "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/evaluation/evaluator.py"
    click EvalTool "https://github.com/tranxuantuyen/planner-refiner/blob/main/tools/eval_mevis.py"
    click ValScript "https://github.com/tranxuantuyen/planner-refiner/blob/main/scripts/val.sh"

    TrainLoop --> Evaluator --> EvalTool
    EvalTool --> ValScript

    %% Configuration & CLI
    subgraph "Configuration & CLI"
        YAML["configs/*.yaml"]:::config
        TrainEntry["train_net_itr.py"]:::script
        TrainSh["scripts/train.sh"]:::script
        TestSh["scripts/test.sh"]:::script
    end

    click YAML "https://github.com/tranxuantuyen/planner-refiner/blob/main/configs/itr_swin_tiny.yaml"
    click TrainEntry "https://github.com/tranxuantuyen/planner-refiner/blob/main/train_net_itr.py"
    click TrainSh "https://github.com/tranxuantuyen/planner-refiner/blob/main/scripts/train.sh"
    click TestSh "https://github.com/tranxuantuyen/planner-refiner/blob/main/scripts/test.sh"

    YAML --> TrainEntry --> Launch
    TrainSh --> TrainEntry
    TestSh --> TrainEntry

    %% Utilities & External Dependencies
    subgraph "Utilities & External"
        MiscITR["ITR utils"]:::module
        MiscMask["mask2former utils"]:::module
        CUDAOps["CUDA Kernel Build"]:::cloud
    end

    click MiscITR "https://github.com/tranxuantuyen/planner-refiner/blob/main/ITR/utils/misc.py"
    click MiscMask "https://github.com/tranxuantuyen/planner-refiner/blob/main/mask2former/utils/misc.py"
    click CUDAOps "https://github.com/tranxuantuyen/planner-refiner/blob/main/mask2former/modeling/pixel_decoder/ops/setup.py"

    CUDAOps --> PixelDecoder
    MiscITR --> TrainLoop
    MiscMask --> PixelDecoder

    %% Flow to Optimizer & Checkpoints
    ITRModel -->|"forward"| Matcher
    Matcher -->|"match"| Criterion
    Criterion -->|"compute loss"| TrainLoop
    TrainLoop -->|"update params"| ITRModel
    TrainLoop -->|"save ckpt"| Defaults

    classDef data fill:#FDEBD0,stroke:#E67E22,color:#000
    classDef module fill:#D6EAF8,stroke:#3498DB,color:#000
    classDef script fill:#D5F5E3,stroke:#27AE60,color:#000
    classDef config fill:#FCF3CF,stroke:#F1C40F,color:#000
    classDef cloud fill:#E8DAEF,stroke:#8E44AD,color:#000
```
