How peterl1n/robustvideomatting works
Project type and purpose
• This is an open-source machine-learning toolkit for high-performance human video matting (RVM). It includes:
• Data ingestion and augmentation modules
• Model definitions (encoder–decoder with recurrent temporal states + guided filters)
• Training and evaluation pipelines
• Inference/conversion APIs (video ↔ alpha/foreground)
• Multi-framework export (TorchScript, ONNX, TF, CoreML, TF.js)
File structure
• Top-level:
• dataset/ – Python classes for loading COCO, Imagenette, SPD, YouTubeVIS, video matting datasets, plus augmentation scripts
• model/ – network definitions (MattingNetwork), encoder backbones (MobileNetV3, ResNet50), ASPP, decoder, guided filters (fast and deep)
• train.py, train_config.py, train_loss.py – training orchestration, hyperparameters, loss functions
• inference.py, inference_utils.py, inference_speed_test.py – inference API (convert_video), VideoReader/Writer, speed benchmarking
• evaluation/ – scripts to compute quality metrics and generate composite outputs
• hubconf.py – TorchHub entrypoints
• documentation/ – Markdown guides for training, inference, misc test lists
• requirements_*.txt – separate dependencies for training vs. inference
README for additional insights
• Core model is a recurrent neural network that maintains temporal memory across frames
• Provides real-time performance (HD 104 FPS, 4K 76 FPS)
• Supports live demos (webcam, Colab, browser) via multiple export formats
• Distributed under GPL-3.0, pretrained weights hosted on GitHub and Google Drive