How penn-pal-lab/liv works
Project Type and Purpose
• LIV is a research‐oriented Python library for learning and evaluating language‐conditioned vision representations for robotic control.
• It supports:
• Pre‐training and fine‐tuning of a multi‐modal model (LIV built atop CLIP)
• Reward‐curve generation (animated plots)
• Inference examples
Top‐Level Component Breakdown
a. Configuration Management (Hydra)
• Directory: liv/cfgs
• YAML files for dataset, training, launcher, output
b. Data Pipeline
• Script to download & manifest datasets (liv/dataset)
• DataLoader utilities (liv/utils/data_loaders.py)
• Supported datasets: EpicKitchen, realrobot, custom
c. Model Layer
• LIV model wrapper (liv/models/model_liv.py)
• Underlying CLIP submodule (liv/models/clip/clip)
d. Training Orchestration
• Entry point: train_liv.py
• Trainer logic: liv/trainer.py
• Checkpointing & evaluation modes
e. Logging & Visualization
• Logger (liv/utils/logger.py)
• Plotter for reward curves (liv/utils/plotter.py)
f. Examples & Inference
• Static inference: liv/examples/liv_static.py
• Dynamic reward‐curve generation: liv/examples/liv_example.py
g. Packaging & Setup
• setup.py at root
• liv/models/clip/setup.py for CLIP
Relationships & Data Flow
• User invokes CLI (train_liv.py or example scripts)
• Hydra loads configs → selects dataset & training regime
• Data pipeline: read manifest → DataLoader yields (frames, text) batches
• Forward pass through LIV model (vision/text encoders + similarity head)
• Loss computation & backward pass in Trainer
• Checkpoints saved; Logger records metrics
• Plotter generates (animated) reward‐curve outputs
• For inference, examples load a saved LIV model and compute embeddings/values
Architectural Patterns & Principles
• Modular separation of concerns: config, data, model, training, utils
• Hydra for reproducible, composable configurations
• PyTorch’s standard training loop pattern encapsulated in trainer.py
• Plugin‐style dataset support via YAML entries in liv/cfgs/dataset
Components & Technologies to Label
• Python 3.9, PyTorch, Torchvision, Hydra, CLIP
• Directories as packages: liv/, liv/models/, liv/utils/, liv/cfgs/, liv/dataset/, liv/examples
• External artifacts: raw video frame directories, manifest.csv, pre-trained CLIP weights, output folders