# omerbt/text2live architecture

> Architecture diagram of the GitHub repository omerbt/text2live, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/omerbt/text2live
- Repository: https://github.com/omerbt/text2live

## Overview

##### Project type and purpose
- This is a research-oriented machine-learning framework (“Text2LIVE”) for text-driven layered editing of images and videos.

##### File structure
- Top level
- CLIP/        – packaged copy of OpenAI’s CLIP code (model, tokenizer, vocab)
- configs/     – YAML configs for image/video experiments and example prompts
- datasets/    – data loader abstractions for images and videos
- models/      – defines the edit generators (image_model.py, video_model.py), clip_extractor, backbones, NLA integration
- util/        – loss functions (atlas_loss, augmentations), utilities (logging, atlas_utils)
- train_image.py, train_video.py – entry-point scripts that glue together data loading, model, losses, optimizer
- Dependencies: PyTorch ≥1.10, CLIP, Neural Layered Atlases (for video consistency)

##### Key insights from the README
- The core loop: load an image or sequence, feed it plus a text prompt into the pipeline, optimize a small network that outputs an edit layer, composite edits over the original, and backpropagate through CLIP-based losses.
- Video mode uses pre-trained NLA models for temporal consistency.
- Config files control hyperparameters, prompt text, dataset paths, and logging frequency.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration
    Config["Configuration Loader"]:::data
    click Config "https://github.com/omerbt/text2live/tree/main/configs/"

    %% Data Loaders
    subgraph "Data Loader"
        direction TB
        ImageLoader["Image Dataset Loader"]:::data
        click ImageLoader "https://github.com/omerbt/text2live/blob/main/datasets/image_dataset.py"
        VideoLoader["Video Dataset Loader"]:::data
        click VideoLoader "https://github.com/omerbt/text2live/blob/main/datasets/video_dataset.py"
    end

    %% Generator Model
    subgraph "Edit Generator Model" 
        direction TB
        ImageModel["ImageModel"]:::trainable
        click ImageModel "https://github.com/omerbt/text2live/blob/main/models/image_model.py"
        VideoModel["VideoModel"]:::trainable
        click VideoModel "https://github.com/omerbt/text2live/blob/main/models/video_model.py"
        subgraph "Backbone Modules"
            direction TB
            Common["common.py"]:::trainable
            click Common "https://github.com/omerbt/text2live/blob/main/models/backbone/common.py"
            Downsampler["downsampler.py"]:::trainable
            click Downsampler "https://github.com/omerbt/text2live/blob/main/models/backbone/downsampler.py"
            Skip["skip.py"]:::trainable
            click Skip "https://github.com/omerbt/text2live/blob/main/models/backbone/skip.py"
        end
    end

    %% NLA for video
    NLA["NLA Module"]:::external
    click NLA "https://github.com/omerbt/text2live/blob/main/util/atlas_utils.py"
    click NLA "https://github.com/omerbt/text2live/blob/main/models/implicit_neural_networks.py"

    %% Composite Step
    Composite["Composite(Edit Layer + Input)"]:::data

    %% Edited Output
    EditedOutput["Edited Output"]:::data

    %% CLIP Module (Frozen)
    subgraph "CLIP Module"
        direction TB
        ClipCore["clip.py"]:::external
        click ClipCore "https://github.com/omerbt/text2live/blob/main/CLIP/clip/clip.py"
        ClipModel["model.py"]:::external
        click ClipModel "https://github.com/omerbt/text2live/blob/main/CLIP/clip/model.py"
        Tokenizer["simple_tokenizer.py"]:::external
        click Tokenizer "https://github.com/omerbt/text2live/blob/main/CLIP/clip/simple_tokenizer.py"
    end

    %% CLIP Explainability & Extraction
    subgraph "CLIP Explainability"
        direction TB
        ClipExplainer["clip.py"]:::external
        click ClipExplainer "https://github.com/omerbt/text2live/blob/main/CLIP/clip_explainability/clip.py"
        ExplModel["model.py"]:::external
        click ExplModel "https://github.com/omerbt/text2live/blob/main/CLIP/clip_explainability/model.py"
        CLIPExtractor["clip_extractor.py"]:::external
        click CLIPExtractor "https://github.com/omerbt/text2live/blob/main/models/clip_extractor.py"
        CLIPRelevancy["clip_relevancy.py"]:::external
        click CLIPRelevancy "https://github.com/omerbt/text2live/blob/main/models/clip_relevancy.py"
    end

    %% Loss Computation
    LossComp["Loss Computation Unit"]:::loss
    click LossComp "https://github.com/omerbt/text2live/blob/main/util/losses.py"
    click LossComp "https://github.com/omerbt/text2live/blob/main/util/atlas_loss.py"
    click LossComp "https://github.com/omerbt/text2live/blob/main/util/aug_utils.py"

    %% Optimizer & Training Loop
    TrainLoop["Optimizer & Training Loop"]:::trainable
    click TrainLoop "https://github.com/omerbt/text2live/blob/main/train_image.py"
    click TrainLoop "https://github.com/omerbt/text2live/blob/main/train_video.py"

    %% Logger & Results
    Logger["Logger & Results Saver"]:::util
    click Logger "https://github.com/omerbt/text2live/blob/main/util/video_logger.py"
    click Logger "https://github.com/omerbt/text2live/blob/main/util/util.py"

    %% Connections
    Config -->|"config YAML"| ImageLoader
    Config -->|"config YAML"| VideoLoader
    ImageLoader -->|"image + prompt"| ImageModel
    VideoLoader -->|"video frames + prompt"| VideoModel
    VideoModel -->|"uses NLA"| NLA
    ImageModel -->|"edit layer"| Composite
    VideoModel -->|"edit layer"| Composite
    Composite -->|"composited output"| EditedOutput
    EditedOutput -->|"edited + prompt"| ClipCore
    ClipCore -->|"embeddings"| ClipExplainer
    ClipExplainer -->|"CLIP scores"| LossComp
    EditedOutput -->|"atlas features"| LossComp
    LossComp -->|"loss"| TrainLoop
    TrainLoop -->|"updated weights"| ImageModel
    TrainLoop -->|"updated weights"| VideoModel
    TrainLoop -->|"metrics & outputs"| Logger

    %% Styles
    classDef data fill:#cce5ff,stroke:#004085,color:#004085
    classDef trainable fill:#d4edda,stroke:#155724,color:#155724
    classDef external fill:#fff3cd,stroke:#856404,color:#856404
    classDef util fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef loss fill:#ffeeba,stroke:#856404,color:#856404
```
