# facebookresearch/videoseal architecture

> Architecture diagram of the GitHub repository facebookresearch/videoseal, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/videoseal
- Repository: https://github.com/facebookresearch/videoseal

## Overview

##### Main Components
- Model & Core Logic: This is at the heart of Video Seal. Inside the “videoseal” directory you have submodules for models, data, losses, modules (like discriminators, VAE, U-Net, etc.), and augmentation. These pieces represent the functionality for watermark embedding, extraction, and loss calculations.
- Augmentation and Preprocessing Pipelines: Clearly from the “videoseal/augmentation” directory, where functions related to geometric alterations, masking, and sequential augmentations are defined.
- Data Handling: The “videoseal/data” directory shows how datasets are loaded and transformed, which is essential for training and inference pipelines.
- Evaluation Modules: Found under “videoseal/evals” that cover speed, full evaluation, metrics, etc.
- Configuration Management: Found in “configs/”, which includes YAML files for datasets, model parameters for embedding and extraction, and ablations. This allows tuning of the watermarking process.
- Inference Scripts: “inference_av.py” and “inference_streaming.py”, alongside notebooks in “notebooks/”, highlight different inference modes (full memory load vs. streaming) and show how models are called via TorchScript.
- Training Scripts: Namely “train.py” and scripts in “scripts/” (for tasks like inflating models for temporal aspects) that indicate how the training pipeline is designed.
- Supporting Utilities: Under “videoseal/utils”, which contains utilities for logging, configuration, tensorboard, and image handling.

##### Relationships and Data Flow
- Flow into the Model: Data (images or video frames) then flows into the core model defined in “videoseal/models” (e.g., embedder, extractor, videoseal main model) to perform watermark embedding or detection.
- Evaluation and Feedback Loop: The resulting watermarked videos/images are then passed into evaluation pipelines (via “videoseal/evals”) to calculate performance metrics such as imperceptibility, robustness, and speed.
- Training Loop: During training, the training script uses configuration settings (from the “configs” directory) to set up the model, loss functions (defined in “videoseal/losses”), and datasets. The training loop may also involve logging and monitoring (through the “videoseal/utils/logger.py”, “tensorboard.py”, etc.).
- Inference Variants: There are two main inference routes – a direct batch inference (inference_av.py) and a streaming, clip-by-clip approach (inference_streaming.py). These need to be clearly differentiated in the architecture with separate components handling video chunk loading, processing, and reassembly.

##### Architectural Patterns and Technologies
- Modular Design: The project follows a modular design with decoupled components for training, inference, augmentation, and evaluation.
- Config-Driven Behavior: Many aspects are configuration-driven (with YAML files) which implies that many components interact via these configs, making the system flexible and customizable.
- Use of TorchScript for Model Export: The inclusion of TorchScript for model serialization (as seen in the quick-start code snippet) is an important component linking the training and inference pipelines for efficiency.

##### Elements
- Clear Labels: Each main component (Data Loader, Augmentation, Core Model, Loss Functions, Evaluation, Training Script, Inference Scripts, Config Manager) should have clear labels.

c. Under Core Processing, sub-divide into modules like “Embedder”, “Extractor”, “Common Utilities”, “Loss Calculations” and “Model Baselines”.
d. Configuration management as a central service that feeds parameters into both the Training and Inference blocks.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Overall Video Seal System Diagram

    %% Input & Preprocessing Layer
    subgraph "Input & Preprocessing"
        D["Data Handling"]:::input
        A["Augmentation Pipeline"]:::input
    end

    %% Core Processing Layer
    subgraph "Core Processing"
        M["Models"]:::core
        MD["Modules"]:::core
        U["Utilities"]:::core
    end

    %% Output & Evaluation Layer
    subgraph "Output & Evaluation"
        E["Evaluation"]:::output
        IF["Inference - Full"]:::output
        IS["Inference - Streaming"]:::output
        NB["Notebooks"]:::output
    end

    %% Configuration & Training Layer
    subgraph "Configuration & Training"
        C["Config Management"]:::training
        TS["Training Script"]:::training
        HS["Helper Scripts"]:::training
    end

    %% External Dependencies
    subgraph "External Dependencies"
        PT["PyTorch/TorchScript"]:::external
        DC["Decord"]:::external
    end

    %% Data Flow Connections
    D -->|"feeds"| M
    D -->|"feeds"| MD
    A -->|"augments"| M
    A -->|"augments"| MD

    %% Training Flow
    C -->|"configures"| TS
    TS -->|"trains"| M
    TS -->|"trains"| MD
    U -->|"logs"| TS
    C -->|"configures"| HS

    %% Core to Output Flow
    M -->|"processed_output"| E
    MD -->|"processed_output"| E
    M -->|"inference_input"| IF
    MD -->|"inference_input"| IF
    M -->|"inference_input"| IS
    MD -->|"inference_input"| IS
    NB -->|"demonstrates"| IF
    NB -->|"demonstrates"| IS

    %% External Dependencies Connections
    PT ---|"executes"| M
    PT ---|"executes"| MD
    DC ---|"supplies_data"| D

    %% Styles
    classDef input fill:#FFEEAD,stroke:#333,stroke-width:2px;
    classDef core fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef output fill:#C1FFC1,stroke:#333,stroke-width:2px;
    classDef training fill:#FFD1DC,stroke:#333,stroke-width:2px;
    classDef external fill:#F4AAAA,stroke:#333,stroke-width:2px;

    %% Click Events
    click D "https://github.com/facebookresearch/videoseal/tree/main/videoseal/data"
    click A "https://github.com/facebookresearch/videoseal/tree/main/videoseal/augmentation"
    click M "https://github.com/facebookresearch/videoseal/tree/main/videoseal/models"
    click MD "https://github.com/facebookresearch/videoseal/tree/main/videoseal/modules"
    click U "https://github.com/facebookresearch/videoseal/tree/main/videoseal/utils"
    click E "https://github.com/facebookresearch/videoseal/tree/main/videoseal/evals"
    click C "https://github.com/facebookresearch/videoseal/tree/main/configs/"
    click IF "https://github.com/facebookresearch/videoseal/blob/main/inference_av.py"
    click IS "https://github.com/facebookresearch/videoseal/blob/main/inference_streaming.py"
    click NB "https://github.com/facebookresearch/videoseal/tree/main/notebooks/"
    click TS "https://github.com/facebookresearch/videoseal/blob/main/train.py"
    click HS "https://github.com/facebookresearch/videoseal/tree/main/scripts/"
```
