# hkchengrex/mivos architecture

> Architecture diagram of the GitHub repository hkchengrex/mivos, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hkchengrex/mivos
- Repository: https://github.com/hkchengrex/mivos

## Overview

##### Project Type and Main Components

a. Interaction-to-Mask (Scribble-to-Mask)
b. Mask Propagation
c. Difference-Aware Fusion

##### High-Level Architecture Design

A. Input Layer
- Video input
- User interaction input (scribbles)

B. Core Processing Modules
##### Interactive Module (fbrs/ directory)
- Scribble-to-Mask conversion
- BRS (Backpropagating Refinement Scheme)

##### Propagation Module (model/propagation/ directory)
- Mask propagation between frames
- Modified ResNet backbone

##### Fusion Module (model/fusion_net.py)
- Difference-aware fusion
- Attention mechanism (model/attn_network.py)

C. Output Layer
- Segmentation masks
- Interactive GUI output

##### Key Relationships to Highlight

- Data flow between modules
- User interaction loops
- Feedback mechanisms
- Training pipelines

##### Technical Components

Supporting components in a separate section:
- Dataset handlers (dataset/ directory)
- Training infrastructure (train.py)
- Evaluation systems (eval_interactive_davis.py)
- GUI interface (interactive_gui.py)

##### Implementation Details to Highlight

Specialized components:
- FBRS controller (fbrs/controller.py)
- Propagation network (model/propagation/prop_net.py)
- Fusion network (model/fusion_net.py)
- Interactive GUI (interact/ directory)

##### Additional Elements

Include:
- Training data flow (BL30K, DAVIS datasets)
- Model evaluation paths
- Real-time processing pipeline
- Multi-object handling capability

Remember to maintain a balance between showing the complexity of the system while keeping it readable and understandable at a glance.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Input Layer
    subgraph Input
        VideoInput["Video Input"]:::input
        UserInput["User Scribbles"]:::input
    end

    %% Core Processing Modules
    subgraph Processing
        subgraph Interactive
            FBRS["FBRS Controller"]:::core
            S2M["Scribble-to-Mask Network"]:::core
            BRS["BRS Predictor"]:::core
            click FBRS "https://github.com/hkchengrex/MiVOS/blob/main/fbrs/controller.py"
            click S2M "https://github.com/hkchengrex/MiVOS/blob/main/model/s2m/s2m_network.py"
            click BRS "https://github.com/hkchengrex/MiVOS/blob/main/fbrs/inference/predictors/brs.py"
        end

        subgraph Propagation
            PropNet["Propagation Network"]:::core
            ResNet["Modified ResNet"]:::core
            PropModules["Propagation Modules"]:::core
            click PropNet "https://github.com/hkchengrex/MiVOS/blob/main/model/propagation/prop_net.py"
            click ResNet "https://github.com/hkchengrex/MiVOS/blob/main/model/propagation/mod_resnet.py"
            click PropModules "https://github.com/hkchengrex/MiVOS/blob/main/model/propagation/modules.py"
        end

        subgraph Fusion
            FusionNet["Fusion Network"]:::core
            AttnNet["Attention Network"]:::core
            click FusionNet "https://github.com/hkchengrex/MiVOS/blob/main/model/fusion_net.py"
            click AttnNet "https://github.com/hkchengrex/MiVOS/blob/main/model/attn_network.py"
        end
    end

    %% Support Infrastructure
    subgraph Infrastructure
        subgraph Dataset
            Davis["DAVIS Dataset"]:::data
            BL30K["BL30K Dataset"]:::data
            click Davis "https://github.com/hkchengrex/MiVOS/blob/main/dataset/davis_test_dataset.py"
            click BL30K "https://github.com/hkchengrex/MiVOS/blob/main/dataset/bl_test_dataset.py"
        end

        subgraph Training
            TrainSys["Training System"]:::support
            Losses["Loss Functions"]:::support
            click TrainSys "https://github.com/hkchengrex/MiVOS/blob/main/train.py"
            click Losses "https://github.com/hkchengrex/MiVOS/blob/main/model/losses.py"
        end

        subgraph Evaluation
            EvalSys["Evaluation System"]:::support
            EvalMetrics["Evaluation Metrics"]:::support
            click EvalSys "https://github.com/hkchengrex/MiVOS/blob/main/eval_interactive_davis.py"
        end
    end

    %% GUI Layer
    subgraph Interface
        GUI["Interactive GUI"]:::ui
        InteractUtils["Interaction Utilities"]:::ui
        click GUI "https://github.com/hkchengrex/MiVOS/blob/main/interactive_gui.py"
        click InteractUtils "https://github.com/hkchengrex/MiVOS/blob/main/interact/interactive_utils.py"
    end

    %% Core Pipeline
    Pipeline["Inference Core"]:::core
    Aggregator["Result Aggregator"]:::core
    click Pipeline "https://github.com/hkchengrex/MiVOS/blob/main/inference_core.py"
    click Aggregator "https://github.com/hkchengrex/MiVOS/blob/main/model/aggregate.py"

    %% Connections
    VideoInput --> PropNet
    UserInput --> FBRS
    FBRS --> S2M
    S2M --> BRS
    BRS --> FusionNet
    PropNet --> FusionNet
    ResNet --> PropNet
    PropModules --> PropNet
    FusionNet --> AttnNet
    AttnNet --> Aggregator
    Pipeline --> GUI

    %% Training Connections
    Davis --> TrainSys
    BL30K --> TrainSys
    TrainSys --> Losses
    EvalSys --> EvalMetrics

    %% Styles
    classDef core fill:#2374ab,stroke:#2374ab,color:#fff
    classDef input fill:#57a773,stroke:#57a773,color:#fff
    classDef data fill:#484d6d,stroke:#484d6d,color:#fff
    classDef support fill:#f7a072,stroke:#f7a072,color:#fff
    classDef ui fill:#7d82b8,stroke:#7d82b8,color:#fff
```
