How mosaicml/composer works
Main Components
• Trainer & Training Loop – This is at the heart of Composer (found in composer/trainer/trainer.py), encapsulating the training loop, scheduling of events, and integration with callbacks.
• Callback System – Modules in composer/callbacks enable insertion of custom logic (e.g., early stopping, logging, speed monitoring) at various training events.
• Algorithms – Located under composer/algorithms, these modules implement training speedups and modifications (such as Label Smoothing, CutMix, ChannelsLast, etc.) to be applied during training.
• Core Components – Under composer/core, you have the definition of algorithm and callback interfaces, event handling, state management, and timing.
• Logging & Monitoring – Loggers in composer/loggers provide integrations with external platforms (Weights & Biases, MLFlow, Neptune, etc.) and support both in-memory and file-based logging.
• Checkpointing – The checkpoint directory and related files (composer/checkpoint) handle saving, loading, and elastic sharded checkpointing (to support changes in hardware during distributed training).
• Distributed & Device Management – Components in composer/devices and composer/distributed abstract hardware types (CPU, GPU, TPU, etc.) and parallelism (e.g., FSDP, DDP) to support scale-out training.
• Model Building – Composer models and tasks under composer/models and composer/models/tasks provide support for various model types (including custom Hugging Face integration).
• CLI & User Interaction – The CLI layer (composer/cli) and examples illustrate how users launch training jobs, configure the environment, and integrate with Docker/containerized environments.
Relationships and Interactions
• Trainer as a central orchestrator that initializes the model, the training loop, and integrates with callbacks and algorithms.
• A data flow from the Dataset (user’s dataloader) through the Trainer to the underlying model, and then back to logging systems.
• External integrations (tracking systems, cloud storage for checkpoint handling, Docker, and CI/CD pipelines via GitHub workflows) as boundaries that interface with the core framework.
Architectural Patterns & Design Principles
• Modularity: The code structure is highly modular. Each major component (Trainer, Algorithms, Callbacks, etc.) is self-contained and extends through clearly defined interfaces.
• Plugin Architecture: The Trainer supports pluggable algorithms and callbacks, following a design that lets users customize training dynamically.
• Abstraction & Separation of Concerns: Core, devices, distributed, and logging components handle specific concerns, making the system easier to extend.
• Configuration & Extensibility: With many configuration scripts (Makefile, Dockerfiles, and pre-commit configurations), the system is built for reproducibility and integration in varied environments.
Diagramming Relevant Technologies & Libraries
• Distributed Training: Mention abstractions around PyTorch DDP, FSDP, and additional cloud integrations.
• Docker & CI/CD: Include CI components (GitHub workflows) and Docker layers that support containerized execution.
• External Services: Represent loggers that integrate with services like WandB, MLFlow, and others.