How nvidia-nemo/automodel works
Project Type & Purpose
• Full-stack, modular Python library and CLI for large-scale training and fine-tuning of LLMs and VLMs.
• Key goals: Day-0 Hugging Face integration, PyTorch DTensor-native SPMD parallelism, recipe-driven workflows, and scalable multi-node deployment (Slurm, Kubernetes).
High-Level Structure (from file tree + README)
Top-level directories:
• nemo_automodel/ – core library (CLI, components, recipes, shared utilities)
• examples/ – end-user recipes for LLM/VLM pretrain, finetune, knowledge distillation, generation
• docs/ – Sphinx conf, guides, architecture docs, performance summary
• tests/ – unit + functional test suites organized by component
• docker/, .github/, workflows – CI/CD, containerization, release pipelines
Main Components & Their Relationships
a. CLI Launcher (“automodel”)
– Path: nemo_automodel/_cli/app.py
– Parses CLI args, selects recipe (LLM or VLM), loads YAML config via Config Loader, and invokes the main training loop.
b. Recipe Layer
– Path: nemo_automodel/recipes/llm and /vlm
– Implements the domain-specific “train loop” (pretrain, SFT, PEFT, KD).
– Depends on core components: datasets, models, distributed, loss, optim, quantization, checkpointing, loggers.
c. Core Component Library (nemo_automodel/components)
• Config Loader – parse YAML/CLI overrides → builds Python dataclass config.
• Datasets – LLM and VLM dataset builders, column-mapped, packers.
• Models – wrappers around Hugging Face Transformers / Diffusers via registry (auto_model).
• Distributed – DTensor mesh setup, FSDP2/HSDP, data/tensor/sequence/pipeline parallelism, Megatron bridge.
• MoE – expert token dispatchers, custom MoE kernels.
• Loss – optimized cross-entropy, KD, Triton kernels.
• Optim – scheduler, FP8 quantization modules.
• Checkpointing – distributed SafeTensors shards, merge/reshard utilities.
• Launcher – Slurm script generator, K8s job spec (in docker/), universal “torchrun” wrapper.
• Loggers – metric logging, Weights & Biases integration.
• Utils – compile/jit helpers, flops, YAML, signature utils.
d. External Dependencies & Infrastructure
• PyTorch (DTensor, torch.compile, torchrun)
• Huggingface Hub (model/tokenizer I/O)
• Storage (local FS, network-attached, S3, HF cache)
• Cluster Manager – Slurm or Kubernetes (via job scripts)
• Container image (docker/Dockerfile, uv lockfiles)
• CI/CD – GitHub Actions, codecov, release workflows
Architectural Patterns & Design Principles
• Modular, hackable plugin architecture (registry pattern for models, datasets).
• Recipe-driven orchestration: decoupled “what to run” (YAML) vs. “how to parallelize” (DTensor placement).
• Single code-path for any scale (1 GPU to thousands via mesh config).
• Separation of concerns: CLI, config, data, model, distributed strategy, checkpointing, logging.