How omerbt/text2live works
Project type and purpose
• This is a research-oriented machine-learning framework (“Text2LIVE”) for text-driven layered editing of images and videos.
File structure
• Top level
• CLIP/ – packaged copy of OpenAI’s CLIP code (model, tokenizer, vocab)
• configs/ – YAML configs for image/video experiments and example prompts
• datasets/ – data loader abstractions for images and videos
• models/ – defines the edit generators (image_model.py, video_model.py), clip_extractor, backbones, NLA integration
• util/ – loss functions (atlas_loss, augmentations), utilities (logging, atlas_utils)
• train_image.py, train_video.py – entry-point scripts that glue together data loading, model, losses, optimizer
• Dependencies: PyTorch ≥1.10, CLIP, Neural Layered Atlases (for video consistency)
Key insights from the README
• The core loop: load an image or sequence, feed it plus a text prompt into the pipeline, optimize a small network that outputs an edit layer, composite edits over the original, and backpropagate through CLIP-based losses.
• Video mode uses pre-trained NLA models for temporal consistency.
• Config files control hyperparameters, prompt text, dataset paths, and logging frequency.