# ux-decoder/segment-everything-everywhere-all-at-once architecture

> Architecture diagram of the GitHub repository ux-decoder/segment-everything-everywhere-all-at-once, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ux-decoder/segment-everything-everywhere-all-at-once
- Repository: https://github.com/ux-decoder/segment-everything-everywhere-all-at-once

## Overview

##### Project type and purpose
- From the README and file tree, this is a research‐grade, full‐stack deep learning project implementing SEEM (“Segment Everything Everywhere All at Once”), a universal, interactive, multi‐modal image/video segmentation model built on top of X-Decoder.
- It includes data preparation, training, evaluation, inference scripts, a live demo application, and comprehensive configuration files.

##### File structure
- Top‐level scripts and entry points:
- entry.py: CLI entry for training/inference pipelines
- assets/scripts/run_demo.sh: one-line demo launcher
- demo/seem/app.py: web service for live demo
- Configuration:
- configs/…/*.yaml: model, dataset, training hyperparameters (Hydra‐style)
- Data handling:
- datasets/: “build.py”, mappers/, evaluation/, registration/
- Modeling:
- modeling/architectures/: high-level SEEM and X-Decoder model definitions
- modeling/body/, modeling/language/, modeling/vision/: subcomponents (backbone, encoder, language encoder, heads)
- modeling/interface/: prompt encoder and decoder interfaces for different SEEM versions
- modeling/modules/: reusable attention, matcher, loss, postprocessing, etc.
- modeling/utils/: supporting utilities (config parsing, operations)
- Pipelines and trainers:
- pipeline/XDecoderPipeline.py: orchestrates full flow (build model → load data → train/infer)
- trainer/: default_trainer.py (single-GPU), distributed_trainer.py, xdecoder_trainer.py
- Inference:
- inference/xdecoder/*.py: standalone inference scripts per task (captioning, instseg, panoseg, etc.)
- demo/…: wraps inference into a web app supporting multi-modal prompts
- Utilities:
- utils/: argument parsing, distributed training helpers, visualization, etc.

##### README for additional insights
- References to external backbones (FocalNet, DaViT), UniCL for contrastive learning, and Whisper for audio transcription.

##### Detail level
- Break modeling into subcomponents (Backbone, Encoder, Prompt, Decoder).
- Trainer utilities—distributed vs. default.
- Inference scripts grouped by task.
- Demo UI layer connecting to inference server via REST or Python API.

[DATA Layer]
↓
[CONFIG] → [PIPELINE orchestrator] → [TRAINER]
↓
[MODEL: {Backbone, LanguageEnc, PromptEnc, Decoder}]
↓
[INFERENCE Scripts & Demo App]
↓
[USER / VISUALIZER]

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        datasets_build["build.py"]:::data
        registration["datasets/registration"]:::data
        mappers["datasets/dataset_mappers"]:::data
        evaluation["datasets/evaluation"]:::data
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        yaml_configs["configs/seem/*.yaml\nconfigs/xdecoder/*.yaml"]:::data
        requirements["assets/requirements/requirements.txt"]:::data
        requirements_custom["assets/requirements/requirements_custom.txt"]:::data
    end

    %% Pipeline & Trainer
    subgraph "Pipeline & Trainer"
        pipeline["XDecoderPipeline.py"]:::pipeline
        subgraph "Trainer Classes"
            default_trainer["default_trainer.py"]:::pipeline
            distributed_trainer["distributed_trainer.py"]:::pipeline
            xdecoder_trainer["xdecoder_trainer.py"]:::pipeline
            trainer_utils["trainer/utils"]:::pipeline
        end
    end

    %% Modeling Layer
    subgraph "Modeling Layer"
        subgraph "Architectures"
            seem_v0["seem_model_v0.py"]:::modeling
            seem_v1["seem_model_v1.py"]:::modeling
            seem_demo["seem_model_demo.py"]:::modeling
            xdecoder_model["xdecoder_model.py"]:::modeling
        end
        subgraph "Vision Backbone"
            vb_dir["modeling/vision/backbone"]:::modeling
        end
        subgraph "Vision Encoder & CUDA Ops"
            encoder_dir["modeling/vision/encoder"]:::modeling
            ops_dir["modeling/vision/encoder/ops"]:::modeling
        end
        subgraph "Language Encoder"
            lang_build["modeling/language/LangEncoder/build.py"]:::modeling
            transformer["modeling/language/LangEncoder/transformer.py"]:::modeling
            vlpencoder["modeling/language/vlpencoder.py"]:::modeling
        end
        subgraph "Prompt Interface"
            interface_dir["modeling/interface"]:::modeling
        end
        subgraph "Body & Heads"
            xdecoder_head["modeling/body/xdecoder_head.py"]:::modeling
            body_build["modeling/body/build.py"]:::modeling
        end
        subgraph "Modules & Utils"
            modules_dir["modeling/modules"]:::modeling
            utils_dir["modeling/utils"]:::modeling
        end
    end

    %% Inference & Demo
    subgraph "Inference & Demo"
        subgraph "Inference Scripts"
            semseg["infer_semseg.py"]:::inference
            instseg["infer_instseg.py"]:::inference
            panoseg["infer_panoseg.py"]:::inference
            refseg["infer_refseg.py"]:::inference
            retrieval["infer_region_retrieval.py"]:::inference
            captioning["infer_captioning.py"]:::inference
        end
        demo_app["demo/seem/app.py"]:::inference
        interactive_tasks["demo/seem/tasks/interactive.py"]:::inference
        run_demo["assets/scripts/run_demo.sh"]:::inference
        entry_cli["entry.py"]:::inference
    end

    %% Flow
    datasets_build -->|feeds data| pipeline
    registration --> pipeline
    mappers --> pipeline
    evaluation --> pipeline

    yaml_configs --> pipeline
    yaml_configs --> architectures
    requirements --> pipeline
    requirements_custom --> pipeline

    pipeline -->|calls| default_trainer
    pipeline -->|calls| distributed_trainer
    pipeline -->|calls| xdecoder_trainer

    pipeline -->|builds model| seem_v0
    pipeline --> architectures
    pipeline --> vb_dir
    pipeline --> encoder_dir
    pipeline --> lang_build
    pipeline --> interface_dir
    pipeline --> xdecoder_head

    default_trainer --> xdecoder_model
    distributed_trainer --> xdecoder_model
    xdecoder_trainer --> xdecoder_model

    xdecoder_model --> semseg
    xdecoder_model --> instseg
    xdecoder_model --> panoseg
    xdecoder_model --> refseg
    xdecoder_model --> retrieval
    xdecoder_model --> captioning

    semseg --> demo_app
    instseg --> demo_app
    panoseg --> demo_app
    refseg --> demo_app
    retrieval --> demo_app
    captioning --> demo_app

    interactive_tasks --> demo_app
    run_demo --> demo_app
    entry_cli --> pipeline

    demo_app -->|"to user"| USER["User / Visualizer"]:::inference

    %% Click Events
    click datasets_build "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/datasets/build.py"
    click registration "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/tree/v1.0/datasets/registration/"
    click mappers "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/tree/v1.0/datasets/dataset_mappers/"
    click evaluation "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/tree/v1.0/datasets/evaluation/"
    click yaml_configs "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/tree/v1.0/configs/seem/"
    click requirements "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/assets/requirements/requirements.txt"
    click requirements_custom "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/assets/requirements/requirements_custom.txt"
    click pipeline "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/pipeline/XDecoderPipeline.py"
    click default_trainer "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/trainer/default_trainer.py"
    click distributed_trainer "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/trainer/distributed_trainer.py"
    click xdecoder_trainer "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/trainer/xdecoder_trainer.py"
    click seem_v0 "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/modeling/architectures/seem_model_v0.py"
    click seem_v1 "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/modeling/architectures/seem_model_v1.py"
    click seem_demo "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/modeling/architectures/seem_model_demo.py"
    click xdecoder_model "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/modeling/architectures/xdecoder_model.py"
    click vb_dir "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/tree/v1.0/modeling/vision/backbone/"
    click encoder_dir "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/tree/v1.0/modeling/vision/encoder/"
    click ops_dir "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/tree/v1.0/modeling/vision/encoder/ops/"
    click lang_build "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/modeling/language/LangEncoder/build.py"
    click transformer "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/modeling/language/LangEncoder/transformer.py"
    click vlpencoder "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/modeling/language/vlpencoder.py"
    click interface_dir "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/tree/v1.0/modeling/interface/"
    click xdecoder_head "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/modeling/body/xdecoder_head.py"
    click body_build "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/modeling/body/build.py"
    click modules_dir "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/tree/v1.0/modeling/modules/"
    click utils_dir "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/tree/v1.0/modeling/utils/"
    click semseg "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/inference/xdecoder/infer_semseg.py"
    click instseg "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/inference/xdecoder/infer_instseg.py"
    click panoseg "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/inference/xdecoder/infer_panoseg.py"
    click refseg "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/inference/xdecoder/infer_refseg.py"
    click retrieval "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/inference/xdecoder/infer_region_retrieval.py"
    click captioning "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/inference/xdecoder/infer_captioning.py"
    click demo_app "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/demo/seem/app.py"
    click interactive_tasks "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/demo/seem/tasks/interactive.py"
    click run_demo "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/assets/scripts/run_demo.sh"
    click entry_cli "https://github.com/ux-decoder/segment-everything-everywhere-all-at-once/blob/v1.0/entry.py"

    %% Styles
    classDef data fill:#CCE5FF,stroke:#000;
    classDef modeling fill:#CCFFCC,stroke:#000;
    classDef pipeline fill:#FFD699,stroke:#000;
    classDef inference fill:#D4CCFF,stroke:#000;
```
