How apple/ml-4m works
Project type and purpose
• This is a research-grade, Python-based machine-learning framework (not a web app or microservice) for “any-to-any” multimodal masked modeling. It provides end-to-end pipelines for:
• Preprocessing and tokenization of tens of modalities (vision, text, depth, edges, poses, etc.)
• Config-driven training of foundation models via PyTorch (+ FSDP, distributed)
• Inference/generation of multiple modalities from any input modality
• Utilities for logging, checkpointing, model-hub integration
File structure
Top‐level directories and scripts:
• cfgs/ : hundreds of YAML configuration files (training, generation, tokenization), organized by default profiles, datasets, model sizes, modalities.
• fourm/ : the core Python package, subdivided into:
– fourm/data: dataset definitions, augmentation, modality transforms, masking
– fourm/models: foundation model (FM) definitions, embeddings, forward-generation logic, LoRA utils
– fourm/vq: VQVAE/DiVAE tokenizers, quantizers, diffusion pipelines
– fourm/utils: distributed training support (dist, fsdp_utils), checkpointing, logging, schedulers, optimizers, huggingface-hub & S3 I/O, plotting
• notebooks/: Jupyter notebooks for generation and retrieval examples
• run_training_*.py, run_generation.py, demo_4M_sampler.py: CLI entry-points for training jobs and inference
• pyproject.toml: package and dependency definitions
README for additional insights
• README.md describes usage, installation, training, generation, data preparation, tokenization, model zoo, huggingface/hub downloads.
• It reinforces a config-driven, modular architecture: users pick a YAML under cfgs/, then call a run_*.py script or the Python API (fourm.demo_4M_sampler) to launch jobs.