# apple/ml-4m architecture

> Architecture diagram of the GitHub repository apple/ml-4m, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/ml-4m
- Repository: https://github.com/apple/ml-4m

## Overview

##### Project type and purpose
- This is a research-grade, Python-based machine-learning framework (not a web app or microservice) for “any-to-any” multimodal masked modeling. It provides end-to-end pipelines for:
- Preprocessing and tokenization of tens of modalities (vision, text, depth, edges, poses, etc.)
- Config-driven training of foundation models via PyTorch (+ FSDP, distributed)
- Inference/generation of multiple modalities from any input modality
- Utilities for logging, checkpointing, model-hub integration

##### File structure
Top‐level directories and scripts:
- cfgs/ : hundreds of YAML configuration files (training, generation, tokenization), organized by default profiles, datasets, model sizes, modalities.
- fourm/ : the core Python package, subdivided into:
– fourm/data: dataset definitions, augmentation, modality transforms, masking
– fourm/models: foundation model (FM) definitions, embeddings, forward-generation logic, LoRA utils
– fourm/vq: VQVAE/DiVAE tokenizers, quantizers, diffusion pipelines
– fourm/utils: distributed training support (dist, fsdp_utils), checkpointing, logging, schedulers, optimizers, huggingface-hub & S3 I/O, plotting
- notebooks/: Jupyter notebooks for generation and retrieval examples
- run_training_*.py, run_generation.py, demo_4M_sampler.py: CLI entry-points for training jobs and inference
- pyproject.toml: package and dependency definitions

##### README for additional insights
- README.md describes usage, installation, training, generation, data preparation, tokenization, model zoo, huggingface/hub downloads.
- It reinforces a config-driven, modular architecture: users pick a YAML under cfgs/, then call a run_*.py script or the Python API (fourm.demo_4M_sampler) to launch jobs.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interfaces
    subgraph "User Interfaces"
        UI1["run_training_4m.py"]:::ui
        UI2["run_training_4m_fsdp.py"]:::ui
        UI3["run_generation.py"]:::ui
        UI4["fourm.demo_4M_sampler"]:::ui
        UI5["Notebooks"]:::ui
    end

    %% Configuration Manager
    subgraph "Configuration Manager"
        CM["YAML Loader (cfgs/)"]:::config
    end

    %% Data Pipeline
    subgraph "Data Pipeline"
        DP1["dataset_utils"]:::data
        DP2["image_augmenter"]:::data
        DP3["masking"]:::data
        DP4["modality_info"]:::data
        DP5["modality_transforms"]:::data
        DP6["multimodal_dataset_folder"]:::data
        DP7["pretrain_utils"]:::data
        DP8["transfer_utils"]:::data
        DP9["unified_datasets"]:::data
    end

    %% Tokenization Layer
    subgraph "Tokenization Layer"
        TL1["text_tokenizer"]:::token
        TL2["simple_tokenizer"]:::token
        TL3["clip.py"]:::token
        TL4["vqvae"]:::token
        TL5["vq models"]:::token
        TL6["quantizers"]:::token
        TL7["diffusion_pipeline"]:::token
        TL8["scheduling"]:::token
    end

    %% Model Core
    subgraph "Model Core"
        MC1["fm.py"]:::model
        MC2["fm_vit.py"]:::model
        MC3["fm_utils.py"]:::model
        MC4["encoder_embeddings"]:::model
        MC5["decoder_embeddings"]:::model
        MC6["lora_utils"]:::model
    end

    %% Training Orchestrator
    subgraph "Training Orchestrator"
        TO1["dist"]:::orchestrator
        TO2["fsdp_utils"]:::orchestrator
        TO3["optim_factory"]:::orchestrator
        TO4["scheduler"]:::orchestrator
        TO5["checkpoint"]:::orchestrator
        TO6["logger"]:::orchestrator
        TO7["run_name"]:::orchestrator
        TO8["seeds"]:::orchestrator
        TO9["native_scaler"]:::orchestrator
    end

    %% Generation Engine
    subgraph "Generation Engine"
        GE1["generate.py"]:::gen
        GE2["generation.py"]:::gen
        GE3["generation_datasets"]:::gen
    end

    %% Storage & External Services
    subgraph "Storage & External Services"
        ST1["S3 utils"]:::storage
        ST2["checkpoint (HF Hub/safetensors)"]:::storage
    end

    %% Connections: Training Flow
    UI1 -->|select config| CM
    UI2 -->|select config| CM
    CM -->|hyperparams & paths| TO3
    TO3 -->|spawn loaders| DP1
    DP9 -->|raw data| DP1
    DP1 -->|augmented data| DP2
    DP2 -->|masked data| DP3
    DP3 -->|modality info| DP4
    DP4 -->|apply transforms| DP5
    DP5 -->|dataset folder| DP6
    DP6 -->|pretrain utils| DP7
    DP7 -->|transfer utils| DP8
    DP8 -->|unified registry| DP9
    DP3 -->|raw modalities| TL1
    TL1 -->|token IDs| TL4
    TL2 -->|token IDs| TL4
    TL3 -->|token IDs| TL4
    TL4 -->|quantized tokens| TL6
    TL6 -->|diffused tokens| TL7
    TL7 -->|scheduled tokens| TL8
    TL8 -->|token IDs| MC1
    MC1 -->|embeddings| MC4
    MC1 -->|embeddings| MC5
    MC1 -->|forward/backward| TO1
    TO1 -->|FSDP sync| TO2
    TO1 -->|update| MC1
    TO3 -->|optimization| TO4
    TO4 -->|scheduling| TO3
    TO5 -->|save/load| ST1
    TO5 -->|save/load| ST2
    TO6 -->|logs| ST1

    %% Connections: Inference Flow
    UI3 -->|select config| CM
    UI4 -->|API call| CM
    UI5 -->|notebook| CM
    UI3 -->|start gen| GE1
    GE1 -->|sample| GE2
    GE2 -->|use tokenizers| TL4
    TL8 -->|token IDs| MC1
    MC1 -->|gen tokens| GE2
    GE2 -->|detokenize| TL5
    GE2 -->|output modalities| UI5

    %% Storage post-inference
    GE1 -->|save outputs| ST1
    GE1 -->|upload weights| ST2

    %% Click Events
    click UI1 "https://github.com/apple/ml-4m/blob/main/./run_training_4m.py"
    click UI2 "https://github.com/apple/ml-4m/blob/main/./run_training_4m_fsdp.py"
    click UI3 "https://github.com/apple/ml-4m/blob/main/./run_generation.py"
    click UI4 "https://github.com/apple/ml-4m/blob/main/fourm/demo_4M_sampler.py"
    click UI5 "https://github.com/apple/ml-4m/tree/main/./notebooks/"
    click CM "https://github.com/apple/ml-4m/tree/main/./cfgs/"
    click DP1 "https://github.com/apple/ml-4m/blob/main/fourm/data/dataset_utils.py"
    click DP2 "https://github.com/apple/ml-4m/blob/main/fourm/data/image_augmenter.py"
    click DP3 "https://github.com/apple/ml-4m/blob/main/fourm/data/masking.py"
    click DP4 "https://github.com/apple/ml-4m/blob/main/fourm/data/modality_info.py"
    click DP5 "https://github.com/apple/ml-4m/blob/main/fourm/data/modality_transforms.py"
    click DP6 "https://github.com/apple/ml-4m/blob/main/fourm/data/multimodal_dataset_folder.py"
    click DP7 "https://github.com/apple/ml-4m/blob/main/fourm/data/pretrain_utils.py"
    click DP8 "https://github.com/apple/ml-4m/blob/main/fourm/data/transfer_utils.py"
    click DP9 "https://github.com/apple/ml-4m/blob/main/fourm/data/unified_datasets.py"
    click TL1 "https://github.com/apple/ml-4m/blob/main/fourm/utils/tokenizer/text_tokenizer.py"
    click TL2 "https://github.com/apple/ml-4m/blob/main/fourm/utils/clip/simple_tokenizer.py"
    click TL3 "https://github.com/apple/ml-4m/blob/main/fourm/utils/clip/clip.py"
    click TL4 "https://github.com/apple/ml-4m/blob/main/fourm/vq/vqvae.py"
    click TL5 "https://github.com/apple/ml-4m/tree/main/fourm/vq/models/"
    click TL6 "https://github.com/apple/ml-4m/tree/main/fourm/vq/quantizers/"
    click TL7 "https://github.com/apple/ml-4m/blob/main/fourm/vq/scheduling/diffusion_pipeline.py"
    click TL8 "https://github.com/apple/ml-4m/blob/main/fourm/vq/scheduling/scheduling_*.py"
    click MC1 "https://github.com/apple/ml-4m/blob/main/fourm/models/fm.py"
    click MC2 "https://github.com/apple/ml-4m/blob/main/fourm/models/fm_vit.py"
    click MC3 "https://github.com/apple/ml-4m/blob/main/fourm/models/fm_utils.py"
    click MC4 "https://github.com/apple/ml-4m/blob/main/fourm/models/encoder_embeddings.py"
    click MC5 "https://github.com/apple/ml-4m/blob/main/fourm/models/decoder_embeddings.py"
    click MC6 "https://github.com/apple/ml-4m/blob/main/fourm/models/lora_utils.py"
    click TO1 "https://github.com/apple/ml-4m/blob/main/fourm/utils/dist.py"
    click TO2 "https://github.com/apple/ml-4m/blob/main/fourm/utils/fsdp_utils.py"
    click TO3 "https://github.com/apple/ml-4m/blob/main/fourm/utils/optim_factory.py"
    click TO4 "https://github.com/apple/ml-4m/blob/main/fourm/utils/scheduler.py"
    click TO5 "https://github.com/apple/ml-4m/blob/main/fourm/utils/checkpoint.py"
    click TO6 "https://github.com/apple/ml-4m/blob/main/fourm/utils/logger.py"
    click TO7 "https://github.com/apple/ml-4m/blob/main/fourm/utils/run_name.py"
    click TO8 "https://github.com/apple/ml-4m/blob/main/fourm/utils/seeds.py"
    click TO9 "https://github.com/apple/ml-4m/blob/main/fourm/utils/native_scaler.py"
    click GE1 "https://github.com/apple/ml-4m/blob/main/fourm/models/generate.py"
    click GE2 "https://github.com/apple/ml-4m/blob/main/fourm/utils/generation.py"
    click GE3 "https://github.com/apple/ml-4m/tree/main/fourm/utils/generation_datasets/"
    click ST1 "https://github.com/apple/ml-4m/blob/main/fourm/utils/s3_utils.py"
    click ST2 "https://github.com/apple/ml-4m/blob/main/fourm/utils/checkpoint.py"

    %% Styles
    classDef ui fill:#fff3b0,stroke:#f2c14e;
    classDef config fill:#fff3b0,stroke:#f2c14e;
    classDef data fill:#bae8e8,stroke:#70c1b3;
    classDef token fill:#bae8e8,stroke:#70c1b3;
    classDef model fill:#bae8e8,stroke:#70c1b3;
    classDef orchestrator fill:#c1fba4,stroke:#63c132;
    classDef gen fill:#c1fba4,stroke:#63c132;
    classDef storage fill:#ffd6a5,stroke:#ff9f1c;
```
