How xizaoqu/worldmem works
Project type and purpose
• This is a research-oriented, full-stack machine-learning system for “WorldMem,” a long-term world simulation model built on diffusion and memory modules.
• Key purpose: train and infer a generative model (VAE + DiT + diffusion + memory) on Minecraft video data, log experiments (Weights & Biases), and serve a demo via app.py.
File structure
• Top-level orchestration:
• main.py, app.py – entry points for training/inference or serving.
• Shell scripts (train_stage_*.sh, infer.sh) – multi-stage training, inference pipelines.
• requirements.txt – dependencies.
• Core algorithm modules (algorithms/):
• common/ – base abstractions (algos, PyTorch wrappers), metrics, models (CNN, MLP).
• worldmem/ – domain-specific code (data fusion, video processing), submodules (attention, diffusion, VAE, pose prediction).
• Configuration management (configurations/):
• YAML files for algorithms, datasets, experiments, HuggingFace, training.
• Dataset handling (datasets/): base & Minecraft-specific video dataset classes.
• Experiment orchestration (experiments/): abstract experiment classes and video experiments.
• Utilities (utils/): checkpoint, cluster, distributed, logging, printing, wandb integration.
README for additional insights
• Multi-stage training on 4 H100 GPUs with wandb logging.
• Option to load separate or combined checkpoints for inference.
• Uses external resources: Minecraft dataset on HF, pretrained Open-Oasis weights.
Important notes
• Be exhaustive: represent each major directory as a component (e.g., algorithms/worldmem/models) when it encapsulates a distinct functionality.
• Ability to switch between separate vs. combined checkpoint loading in the inference path.
• Multi-GPU / distributed utilities adjacent to the training component.