# peterl1n/robustvideomatting architecture

> Architecture diagram of the GitHub repository peterl1n/robustvideomatting, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/peterl1n/robustvideomatting
- Repository: https://github.com/peterl1n/robustvideomatting

## Overview

##### Project type and purpose
- This is an open-source machine-learning toolkit for high-performance human video matting (RVM). It includes:
- Data ingestion and augmentation modules
- Model definitions (encoder–decoder with recurrent temporal states + guided filters)
- Training and evaluation pipelines
- Inference/conversion APIs (video ↔ alpha/foreground)
- Multi-framework export (TorchScript, ONNX, TF, CoreML, TF.js)

##### File structure
- Top-level:
- dataset/ – Python classes for loading COCO, Imagenette, SPD, YouTubeVIS, video matting datasets, plus augmentation scripts
- model/ – network definitions (MattingNetwork), encoder backbones (MobileNetV3, ResNet50), ASPP, decoder, guided filters (fast and deep)
- train.py, train_config.py, train_loss.py – training orchestration, hyperparameters, loss functions
- inference.py, inference_utils.py, inference_speed_test.py – inference API (convert_video), VideoReader/Writer, speed benchmarking
- evaluation/ – scripts to compute quality metrics and generate composite outputs
- hubconf.py – TorchHub entrypoints
- documentation/ – Markdown guides for training, inference, misc test lists
- requirements_*.txt – separate dependencies for training vs. inference

##### README for additional insights
- Core model is a recurrent neural network that maintains temporal memory across frames
- Provides real-time performance (HD 104 FPS, 4K 76 FPS)
- Supports live demos (webcam, Colab, browser) via multiple export formats
- Distributed under GPL-3.0, pretrained weights hosted on GitHub and Google Drive

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        Datasets["Datasets: COCO, ImageMatte, SPD, VideoMatte, YouTubeVIS"]:::data
        Augmentation["DataLoader + Augmentation"]:::compute
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        TrainScript["train.py"]:::compute
        TrainConfig["train_config.py"]:::compute
        TrainLoss["train_loss.py"]:::compute
        subgraph "Model Definitions"
            direction TB
            MattingNetwork["model/model.py\n(MattingNetwork)"]:::compute
            Backbone1["model/mobilenetv3.py\n(Backbone)"]:::compute
            Backbone2["model/resnet.py\n(Backbone)"]:::compute
            ASPP["model/lraspp.py\n(ASPP)"]:::compute
            Decoder["model/decoder.py"]:::compute
            FastGF["model/fast_guided_filter.py"]:::compute
            DeepGF["model/deep_guided_filter.py"]:::compute
        end
    end

    %% Evaluation Pipeline
    subgraph "Evaluation Pipeline"
        direction TB
        EvalHR["evaluation/evaluate_hr.py"]:::compute
        EvalLR["evaluation/evaluate_lr.py"]:::compute
        GenIMG_BG_IMG["evaluation/generate_imagematte_with_background_image.py"]:::compute
        GenIMG_BG_VID["evaluation/generate_imagematte_with_background_video.py"]:::compute
        GenVID_BG_IMG["evaluation/generate_videomatte_with_background_image.py"]:::compute
        GenVID_BG_VID["evaluation/generate_videomatte_with_background_video.py"]:::compute
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        direction TB
        InferenceAPI["inference.py"]:::compute
        InferenceUtils["inference_utils.py"]:::io
        SpeedTest["inference_speed_test.py"]:::compute
    end

    %% Export & Deployment
    subgraph "Export & Deployment"
        direction TB
        HubConf["hubconf.py"]:::export
        ReqInfer["requirements_inference.txt"]:::export
        ReqTrain["requirements_training.txt"]:::export
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        PyTorch["PyTorch, torchvision"]:::external
        VideoIO["FFmpeg Bindings"]:::external
        GPU["GPU Hardware"]:::external
    end

    %% Flows
    Datasets -->|feeds| Augmentation
    Augmentation -->|batches| TrainScript
    TrainScript -->|loads| MattingNetwork
    MattingNetwork --> Backbone1
    MattingNetwork --> Backbone2
    Backbone1 --> ASPP
    Backbone2 --> ASPP
    ASPP --> Decoder
    Decoder --> FastGF
    Decoder --> DeepGF
    TrainScript -->|uses| TrainConfig
    TrainScript -->|uses| TrainLoss

    TrainScript -->|outputs checkpoints| EvalHR
    TrainScript -->|outputs checkpoints| InferenceAPI
    TrainScript -->|outputs checkpoints| HubConf

    EvalHR -->|computes metrics| PyTorch
    EvalLR -->|computes metrics| PyTorch
    EvalHR -->|generates composites| VideoIO
    GenIMG_BG_IMG --> VideoIO
    GenIMG_BG_VID --> VideoIO
    GenVID_BG_IMG --> VideoIO
    GenVID_BG_VID --> VideoIO

    InferenceAPI -->|reads| InferenceUtils
    InferenceAPI -->|benchmarks| SpeedTest
    InferenceAPI -->|exports models| HubConf

    HubConf -->|exports to| ReqInfer
    HubConf -->|exports to| ReqTrain

    %% Click Events
    click Datasets "https://github.com/peterl1n/robustvideomatting/tree/master/dataset/"
    click Augmentation "https://github.com/peterl1n/robustvideomatting/blob/master/dataset/augmentation.py"
    click TrainScript "https://github.com/peterl1n/robustvideomatting/blob/master/train.py"
    click TrainConfig "https://github.com/peterl1n/robustvideomatting/blob/master/train_config.py"
    click TrainLoss "https://github.com/peterl1n/robustvideomatting/blob/master/train_loss.py"
    click MattingNetwork "https://github.com/peterl1n/robustvideomatting/blob/master/model/model.py"
    click Backbone1 "https://github.com/peterl1n/robustvideomatting/blob/master/model/mobilenetv3.py"
    click Backbone2 "https://github.com/peterl1n/robustvideomatting/blob/master/model/resnet.py"
    click ASPP "https://github.com/peterl1n/robustvideomatting/blob/master/model/lraspp.py"
    click Decoder "https://github.com/peterl1n/robustvideomatting/blob/master/model/decoder.py"
    click FastGF "https://github.com/peterl1n/robustvideomatting/blob/master/model/fast_guided_filter.py"
    click DeepGF "https://github.com/peterl1n/robustvideomatting/blob/master/model/deep_guided_filter.py"
    click EvalHR "https://github.com/peterl1n/robustvideomatting/blob/master/evaluation/evaluate_hr.py"
    click EvalLR "https://github.com/peterl1n/robustvideomatting/blob/master/evaluation/evaluate_lr.py"
    click GenIMG_BG_IMG "https://github.com/peterl1n/robustvideomatting/blob/master/evaluation/generate_imagematte_with_background_image.py"
    click GenIMG_BG_VID "https://github.com/peterl1n/robustvideomatting/blob/master/evaluation/generate_imagematte_with_background_video.py"
    click GenVID_BG_IMG "https://github.com/peterl1n/robustvideomatting/blob/master/evaluation/generate_videomatte_with_background_image.py"
    click GenVID_BG_VID "https://github.com/peterl1n/robustvideomatting/blob/master/evaluation/generate_videomatte_with_background_video.py"
    click InferenceAPI "https://github.com/peterl1n/robustvideomatting/blob/master/inference.py"
    click InferenceUtils "https://github.com/peterl1n/robustvideomatting/blob/master/inference_utils.py"
    click SpeedTest "https://github.com/peterl1n/robustvideomatting/blob/master/inference_speed_test.py"
    click HubConf "https://github.com/peterl1n/robustvideomatting/blob/master/hubconf.py"
    click ReqInfer "https://github.com/peterl1n/robustvideomatting/blob/master/requirements_inference.txt"
    click ReqTrain "https://github.com/peterl1n/robustvideomatting/blob/master/requirements_training.txt"

    %% Styles
    classDef data fill:#cce5ff,stroke:#004085,color:#004085
    classDef compute fill:#d4edda,stroke:#155724,color:#155724
    classDef io fill:#fff3cd,stroke:#856404,color:#856404
    classDef export fill:#e2e3e5,stroke:#383d41,color:#383d41
    classDef external fill:#f8d7da,stroke:#721c24,color:#721c24
```
