# xizaoqu/worldmem architecture

> Architecture diagram of the GitHub repository xizaoqu/worldmem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xizaoqu/worldmem
- Repository: https://github.com/xizaoqu/worldmem

## Overview

##### Project type and purpose
- This is a research-oriented, full-stack machine-learning system for “WorldMem,” a long-term world simulation model built on diffusion and memory modules.
- Key purpose: train and infer a generative model (VAE + DiT + diffusion + memory) on Minecraft video data, log experiments (Weights & Biases), and serve a demo via `app.py`.

##### File structure
- Top-level orchestration:
- `main.py`, `app.py` – entry points for training/inference or serving.
- Shell scripts (`train_stage_*.sh`, `infer.sh`) – multi-stage training, inference pipelines.
- `requirements.txt` – dependencies.
- Core algorithm modules (`algorithms/`):
- `common/` – base abstractions (algos, PyTorch wrappers), metrics, models (CNN, MLP).
- `worldmem/` – domain-specific code (data fusion, video processing), submodules (attention, diffusion, VAE, pose prediction).
- Configuration management (`configurations/`):
- YAML files for algorithms, datasets, experiments, HuggingFace, training.
- Dataset handling (`datasets/`): base & Minecraft-specific video dataset classes.
- Experiment orchestration (`experiments/`): abstract experiment classes and video experiments.
- Utilities (`utils/`): checkpoint, cluster, distributed, logging, printing, wandb integration.

##### README for additional insights
- Multi-stage training on 4 H100 GPUs with wandb logging.
- Option to load separate or combined checkpoints for inference.
- Uses external resources: Minecraft dataset on HF, pretrained Open-Oasis weights.

##### Important notes
- Be exhaustive: represent each major directory as a component (e.g., `algorithms/worldmem/models`) when it encapsulates a distinct functionality.
- Ability to switch between separate vs. combined checkpoint loading in the inference path.
- Multi-GPU / distributed utilities adjacent to the training component.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Actions
    UserCLI("User CLI: python main.py / train_stage_x.sh"):::user
    UserInfer("User CLI: python app.py / bash infer.sh"):::user

    %% Data Layer
    subgraph "Data Layer" 
        direction TB
        BVD["Base Video Dataset"]:::data
        MVD["Minecraft Video Dataset"]:::data
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        AlgoCfg["Algorithm Configs"]:::config
        DataCfg["Dataset Configs"]:::config
        ExpCfg["Experiment Configs"]:::config
        HFTrainCfg["HuggingFace & Training Configs"]:::config
    end

    %% Training & Experiment Orchestrator
    subgraph "Orchestrator" 
        direction TB
        MainEP["main.py"]:::orchestrator
        Scripts["Multi-Stage Training Scripts"]:::orchestrator
        ExpClasses["Experiment Classes"]:::orchestrator
    end

    %% Algorithm Engine
    subgraph "Algorithm Engine"
        direction TB
        subgraph "Common Algorithms"
            direction TB
            BaseAlgo["base_algo.py"]:::algo
            BaseTorchAlgo["base_pytorch_algo.py"]:::algo
        end
        subgraph "Common Models & Metrics"
            direction TB
            CNN["cnn.py"]:::algo
            MLP["mlp.py"]:::algo
            FID["fid.py"]:::algo
            FVD["fvd.py"]:::algo
            LPIPS["lpips.py"]:::algo
        end
        subgraph "WorldMem Modules"
            direction TB
            DFBase["df_base.py"]:::algo
            DFVideo["df_video.py"]:::algo
            Attention["attention.py"]:::algo
            CameraCtrl["cameractrl_module.py"]:::algo
            Diffusion["diffusion.py"]:::algo
            DiT["dit.py"]:::algo
            PosePredMod["pose_prediction.py"]:::algo
            Rotary["rotary_embedding_torch.py"]:::algo
            VAE["vae.py"]:::algo
            WMUtils["utils.py"]:::algo
        end
        PoseWrapper["pose_prediction.py (wrapper)"]:::algo
    end

    %% Metrics & Checkpointing
    subgraph "Metrics & Checkpointing"
        direction TB
        CkptUtils["ckpt_utils.py"]:::utils
        DistUtils["distributed_utils.py"]:::utils
    end

    %% Logging & Monitoring
    subgraph "Logging & Monitoring"
        direction TB
        LogUtils["logging_utils.py"]:::utils
        WandbUtils["wandb_utils.py"]:::utils
    end

    %% Inference & Serving
    subgraph "Inference & Serving"
        direction TB
        APIServer["app.py"]:::orchestrator
        InferScript["infer.sh"]:::orchestrator
    end

    %% External Services
    subgraph "External Services"
        direction TB
        WandB["Weights & Biases"]:::external
        HFHub["HuggingFace"]:::external
    end

    %% Utilities
    subgraph "Utilities"
        direction TB
        Cluster["cluster_utils.py"]:::utils
        PrintU["print_utils.py"]:::utils
    end

    %% Connections
    UserCLI --> MainEP
    UserCLI --> Scripts
    UserInfer --> APIServer
    UserInfer --> InferScript

    BVD -->|data source| MainEP
    MVD -->|data source| MainEP

    AlgoCfg --> MainEP
    DataCfg --> MainEP
    ExpCfg --> MainEP
    HFTrainCfg --> MainEP

    MainEP --> ExpClasses
    Scripts --> ExpClasses

    ExpClasses -->|invoke| BaseAlgo
    ExpClasses --> BaseTorchAlgo
    ExpClasses --> CNN
    ExpClasses --> MLP
    ExpClasses --> DFBase
    ExpClasses --> DFVideo

    BaseAlgo --> PoseWrapper
    BaseTorchAlgo --> PoseWrapper

    PoseWrapper --> Attention
    PoseWrapper --> CameraCtrl
    PoseWrapper --> Diffusion
    PoseWrapper --> DiT
    PoseWrapper --> VAE
    PoseWrapper --> Rotary
    PoseWrapper --> WMUtils
    PoseWrapper --> PosePredMod

    PoseWrapper -->|outputs| CkptUtils
    PoseWrapper -->|outputs| DistUtils

    CkptUtils -->|logs| WandbUtils
    DistUtils -->|supports| BaseTorchAlgo

    PoseWrapper -->|inference model| APIServer

    APIServer -->|optionally load| HFHub
    APIServer -->|log| WandbUtils
    InferScript --> APIServer

    WandbUtils --> WandB

    %% Click Events
    click BVD "https://github.com/xizaoqu/worldmem/blob/main/datasets/video/base_video_dataset.py"
    click MVD "https://github.com/xizaoqu/worldmem/blob/main/datasets/video/minecraft_video_dataset.py"
    click AlgoCfg "https://github.com/xizaoqu/worldmem/tree/main/configurations/algorithm/"
    click DataCfg "https://github.com/xizaoqu/worldmem/tree/main/configurations/dataset/"
    click ExpCfg "https://github.com/xizaoqu/worldmem/tree/main/configurations/experiment/"
    click HFTrainCfg "https://github.com/xizaoqu/worldmem/blob/main/configurations/huggingface.yaml"
    click HFTrainCfg "https://github.com/xizaoqu/worldmem/blob/main/configurations/training.yaml"
    click MainEP "https://github.com/xizaoqu/worldmem/blob/main/main.py"
    click Scripts "https://github.com/xizaoqu/worldmem/blob/main/train_stage_1.sh"
    click Scripts "https://github.com/xizaoqu/worldmem/blob/main/train_stage_2.sh"
    click Scripts "https://github.com/xizaoqu/worldmem/blob/main/train_stage_3.sh"
    click ExpClasses "https://github.com/xizaoqu/worldmem/blob/main/experiments/exp_base.py"
    click ExpClasses "https://github.com/xizaoqu/worldmem/blob/main/experiments/exp_video.py"
    click BaseAlgo "https://github.com/xizaoqu/worldmem/blob/main/algorithms/common/base_algo.py"
    click BaseTorchAlgo "https://github.com/xizaoqu/worldmem/blob/main/algorithms/common/base_pytorch_algo.py"
    click CNN "https://github.com/xizaoqu/worldmem/blob/main/algorithms/common/models/cnn.py"
    click MLP "https://github.com/xizaoqu/worldmem/blob/main/algorithms/common/models/mlp.py"
    click FID "https://github.com/xizaoqu/worldmem/blob/main/algorithms/common/metrics/fid.py"
    click FVD "https://github.com/xizaoqu/worldmem/blob/main/algorithms/common/metrics/fvd.py"
    click LPIPS "https://github.com/xizaoqu/worldmem/blob/main/algorithms/common/metrics/lpips.py"
    click DFBase "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/df_base.py"
    click DFVideo "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/df_video.py"
    click Attention "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/models/attention.py"
    click CameraCtrl "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/models/cameractrl_module.py"
    click Diffusion "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/models/diffusion.py"
    click DiT "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/models/dit.py"
    click PosePredMod "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/models/pose_prediction.py"
    click Rotary "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/models/rotary_embedding_torch.py"
    click VAE "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/models/vae.py"
    click WMUtils "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/models/utils.py"
    click PoseWrapper "https://github.com/xizaoqu/worldmem/blob/main/algorithms/worldmem/pose_prediction.py"
    click CkptUtils "https://github.com/xizaoqu/worldmem/blob/main/utils/ckpt_utils.py"
    click DistUtils "https://github.com/xizaoqu/worldmem/blob/main/utils/distributed_utils.py"
    click LogUtils "https://github.com/xizaoqu/worldmem/blob/main/utils/logging_utils.py"
    click WandbUtils "https://github.com/xizaoqu/worldmem/blob/main/utils/wandb_utils.py"
    click APIServer "https://github.com/xizaoqu/worldmem/blob/main/app.py"
    click InferScript "https://github.com/xizaoqu/worldmem/blob/main/infer.sh"
    click Cluster "https://github.com/xizaoqu/worldmem/blob/main/utils/cluster_utils.py"
    click PrintU "https://github.com/xizaoqu/worldmem/blob/main/utils/print_utils.py"

    %% Styles
    classDef data fill:#ccffcc,stroke:#006600
    classDef config fill:#ccffcc,stroke:#006600
    classDef orchestrator fill:#ffcc99,stroke:#cc6600
    classDef algo fill:#cce5ff,stroke:#004085
    classDef utils fill:#e2e3e5,stroke:#6c757d
    classDef external fill:#e5ccff,stroke:#660066
    classDef user fill:#ffffff,stroke:#000000,stroke-dasharray: 2 2
```
