# pq-yang/matanyone architecture

> Architecture diagram of the GitHub repository pq-yang/matanyone, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pq-yang/matanyone
- Repository: https://github.com/pq-yang/matanyone

## Overview

##### Project type and purpose
- MatAnyone is a video-matting framework (full-stack inference tool) that:
– Provides CLI scripts for offline inference (inference_matanyone.py, inference_hf.py).
– Provides an interactive Gradio demo (hugging_face/app.py).
– Encapsulates the core inference logic in a Python package (`matanyone`).

##### File structure
- Top-level components:
– `hugging_face/` → Gradio-based demo, wrapper around the core library.
– `inference_matanyone.py` & `inference_hf.py` → CLI entry points.
– `matanyone/` → core package with submodules:
- `config/` (Hydra configuration files)
- `inference/` (the orchestration of video frames, memory store, object manager)
- `model/` (PyTorch model definitions: backbone, transformer, attention modules)
- `utils/` (helper functions: I/O, tensor ops, model loading)
– `assets/`, `inputs/` → static assets, test videos & masks.
– `pyproject.toml`, `setup.py` (packaging).

##### README for additional insights
- The “Overview” section shows a four-stage pipeline:
1. Load first-frame mask (SAM/SAM2)
2. Extract features & propagate memory
3. Perform matting via transformer modules
4. Composite foreground + alpha layers
- Dependencies: PyTorch, Hydra, Gradio, Segment Anything, FFmpeg.

##### Main components
a. Frontends
- CLI Layer
– `inference_matanyone.py` (video+mask → InferenceCore → results folder)
– `inference_hf.py` (HF model hub loader → InferenceCore)
- Web Demo
– Gradio server (`hugging_face/app.py`)
– Painter & mask-assignment tools (`hugging_face/tools/…`)
b. Core Library (`matanyone`)
- Config Manager (Hydra)
- Inference Orchestrator (`InferenceCore` in `matanyone/inference/inference_core.py`)
- Memory Store (key/value memory for temporal propagation)
- Object Manager (handles multiple targets)
- ImageFeatureStore (backbone feature extraction)
c. Model Components
- Backbone & auxiliary modules (`model/resnet.py`, `aux_modules.py`)
- Transformer modules (`object_transformer.py`, `object_summarizer.py`, etc.)
- Channel attention, positional encoding, group modules
d. Utils & I/O
- File I/O, pre/post processing
- Tensor utilities, model loading helpers

##### Relationships & Interactions
- Frontend → calls → Core Library (InferenceCore)
- InferenceCore → loads → Hydra config → instantiates → Model + MemoryManager + ObjectManager
- Model → processes → current frame + memory features → yields → alpha & foreground
- MemoryManager → updates → key/value store across frames
- Outputs → written to disk (video/images) or returned to Gradio browser

##### Architectural patterns & design principles
- Modular pipeline: clear separation of stages (feature extraction, memory propagation, matting)
- Config-driven instantiation via Hydra (easy hyperparameter sweep)
- Stateful memory store for temporal consistency (KV-memory pattern)
- Extensible frontends (CLI vs. Web) using the same InferenceCore API

##### Technologies, frameworks & libraries
- PyTorch for model definitions & inference
- Hydra for config management
- Gradio for interactive web demo
- Segment Anything & SAM2 for mask initialization
- FFmpeg for video I/O
- Hugging Face model hub integration for remote weight loading

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        CLI_MatAnyone["CLI: inference_matanyone.py"]:::frontend
        CLI_HF["CLI: inference_hf.py"]:::frontend
        Web_Server["Gradio UI: app.py"]:::frontend
        Mask_Painter["Mask Painter"]:::frontend
        Base_Segmenter["Base Segmenter"]:::frontend
        Interact_Tools["Interaction Tools"]:::frontend
    end

    %% Core Library
    subgraph "Core Library"
        Config["Config Manager\n(Hydra)"]:::core
        InferenceCore["InferenceCore\n(Orchestrator)"]:::core
        ImageFeatureStore["Image Feature Store"]:::core
        KVMemory["KV Memory Store"]:::core
        MemoryMgr["Memory Manager"]:::core
        ObjMgr["Object Manager"]:::core
        ObjInfo["Object Info"]:::core
        ArgsUtils["CLI Utils"]:::core
    end

    %% Model Components
    subgraph "Model Components"
        ModelEntry["MatAnyone Model"]:::model
        AuxModules["Aux Modules"]:::model
        ResNetUtils["ResNet Utils"]:::model
        ChannelAttn["Channel Attention"]:::model
        GroupModules["Group Modules"]:::model
        BigModules["Big Modules"]:::model
        subgraph Transformers
            TrObj["Object Transformer"]:::model
            TrSumm["Object Summarizer"]:::model
            PosEnc["Positional Encoding"]:::model
            TrLayers["Transformer Layers"]:::model
        end
    end

    %% Model Utils
    subgraph "Model Utilities"
        ParamGroups["Parameter Groups"]:::utils
        MemoryUtils["Memory Utils"]:::utils
    end

    %% Utils & I/O
    subgraph "Utilities & I/O"
        DefaultModelLoader["Default Model Loader"]:::utils
        InferenceUtils["Inference Helpers"]:::utils
        TensorUtils["Tensor Utils"]:::utils
    end

    %% Static Assets
    subgraph "Static Assets"
        InputsVid["Input Videos"]:::external
        InputsMask["Input Masks"]:::external
        Assets["Project Assets"]:::external
    end

    %% External Services
    subgraph "External Services"
        HFHub["Hugging Face Hub"]:::external
        SAMAPI["Segment Anything API"]:::external
        FFmpeg["FFmpeg"]:::external
    end

    %% Connections - CLI and Inputs
    InputsVid -->|video input| CLI_MatAnyone
    InputsMask -->|mask input| CLI_MatAnyone
    CLI_MatAnyone -->|calls| InferenceCore
    CLI_HF -->|loads model/config| InferenceCore

    %% Web interactions
    Web_Server -->|brush mask| Mask_Painter
    Web_Server -->|segment| Base_Segmenter
    Web_Server -->|interact| Interact_Tools
    Mask_Painter -->|mask| InferenceCore
    Base_Segmenter -->|initial mask| InferenceCore
    Interact_Tools -->|params| InferenceCore

    %% Core workflow
    InferenceCore -->|load config| Config
    InferenceCore -->|instantiate model| ModelEntry
    InferenceCore -->|extract features| ImageFeatureStore
    InferenceCore -->|access memory| KVMemory
    InferenceCore -->|update memory| MemoryMgr
    InferenceCore -->|manage objects| ObjMgr
    InferenceCore -->|load weights| DefaultModelLoader
    InferenceCore -->|helpers| InferenceUtils
    InferenceCore -->|tensor ops| TensorUtils
    InferenceCore -->|mask init| SAMAPI

    %% Model internals
    ModelEntry --> AuxModules
    ModelEntry --> ResNetUtils
    ModelEntry --> ChannelAttn
    ModelEntry --> GroupModules
    ModelEntry --> BigModules
    ModelEntry --> TrObj
    TrObj --> TrSumm
    TrObj --> PosEnc
    TrObj --> TrLayers

    %% Utils & external links
    DefaultModelLoader -->|from hub| HFHub
    InferenceUtils -->|video I/O| FFmpeg

    %% Click Events
    click CLI_MatAnyone "https://github.com/pq-yang/matanyone/blob/main/inference_matanyone.py"
    click CLI_HF "https://github.com/pq-yang/matanyone/blob/main/inference_hf.py"
    click Web_Server "https://github.com/pq-yang/matanyone/blob/main/hugging_face/app.py"
    click Mask_Painter "https://github.com/pq-yang/matanyone/blob/main/hugging_face/tools/mask_painter.py"
    click Base_Segmenter "https://github.com/pq-yang/matanyone/blob/main/hugging_face/tools/base_segmenter.py"
    click Interact_Tools "https://github.com/pq-yang/matanyone/blob/main/hugging_face/tools/interact_tools.py"
    click Config "https://github.com/pq-yang/matanyone/tree/main/matanyone/config/"
    click InferenceCore "https://github.com/pq-yang/matanyone/blob/main/matanyone/inference/inference_core.py"
    click ImageFeatureStore "https://github.com/pq-yang/matanyone/blob/main/matanyone/inference/image_feature_store.py"
    click KVMemory "https://github.com/pq-yang/matanyone/blob/main/matanyone/inference/kv_memory_store.py"
    click MemoryMgr "https://github.com/pq-yang/matanyone/blob/main/matanyone/inference/memory_manager.py"
    click ObjMgr "https://github.com/pq-yang/matanyone/blob/main/matanyone/inference/object_manager.py"
    click ObjInfo "https://github.com/pq-yang/matanyone/blob/main/matanyone/inference/object_info.py"
    click ArgsUtils "https://github.com/pq-yang/matanyone/blob/main/matanyone/inference/utils/args_utils.py"
    click ModelEntry "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/matanyone.py"
    click AuxModules "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/aux_modules.py"
    click ResNetUtils "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/utils/resnet.py"
    click ChannelAttn "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/channel_attn.py"
    click GroupModules "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/group_modules.py"
    click BigModules "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/big_modules.py"
    click TrObj "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/transformer/object_transformer.py"
    click TrSumm "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/transformer/object_summarizer.py"
    click PosEnc "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/transformer/positional_encoding.py"
    click TrLayers "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/transformer/transformer_layers.py"
    click ParamGroups "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/utils/parameter_groups.py"
    click MemoryUtils "https://github.com/pq-yang/matanyone/blob/main/matanyone/model/utils/memory_utils.py"
    click DefaultModelLoader "https://github.com/pq-yang/matanyone/blob/main/matanyone/utils/get_default_model.py"
    click InferenceUtils "https://github.com/pq-yang/matanyone/blob/main/matanyone/utils/inference_utils.py"
    click TensorUtils "https://github.com/pq-yang/matanyone/blob/main/matanyone/utils/tensor_utils.py"
    click InputsVid "https://github.com/pq-yang/matanyone/tree/main/inputs/video/"
    click InputsMask "https://github.com/pq-yang/matanyone/tree/main/inputs/mask/"
    click Assets "https://github.com/pq-yang/matanyone/tree/main/assets/"

    %% Styles
    classDef frontend fill:#E0F7FA,stroke:#0288D1,stroke-width:1px
    classDef core fill:#C8E6C9,stroke:#2E7D32,stroke-width:1px
    classDef model fill:#FFE0B2,stroke:#EF6C00,stroke-width:1px
    classDef utils fill:#EEEEEE,stroke:#757575,stroke-width:1px
    classDef external fill:#E1BEE7,stroke:#8E24AA,stroke-width:1px
```
