How mlcommons/training works
Major Components
– Language Models (BERT, GPT3, LLama2 70B-LoRA)
– Vision and Image Processing (Stable Diffusion, Single-Stage Detector, etc.)
– Graph Neural Networks (e.g., RGAT implemented in “graph_neural_network”)
– Recommendation (e.g., DLRMv2 implemented in “recommendation_v2/torchrec_dlrm”)
– (Additional benchmarks, retired benchmarks, etc. can be indicated as legacy or alternate paths.)
• Shared Infrastructure and Tools:
– Docker Setup: Almost every benchmark folder contains its own Dockerfiles (and in some cases Docker-compose or related build scripts). Represent this as a common “Containerization” component that ensures reproducibility and environment isolation.
– Data Download & Verification: There are several “download_dataset.sh” and “verify_dataset.sh” scripts in many subdirectories. Illustrate these as part of the “Data Ingestion” or “Dataset Management” sub-system connected to each benchmark.
– Logging/MLPerf Utilities: A set of scripts such as “mlperf_logging_utils.py” appear in multiple folders. Represent these common logging and metrics components as a shared service or library.
– CI/CD and Developer Operations: The “.github” directory in the root indicates usage of workflows (like “cla.yml”), and tools like “install_cuda_docker.sh” ensure system setup. Add these as supporting infrastructure.
Interactions and Relationships
• Data Flow: For each benchmark, illustrate how:
– The dataset is first downloaded and verified.
– The Docker container is built (using a dedicated Dockerfile) and then the training process is triggered (via bash scripts or pipelined commands).
– Model training, logging, and measurement are performed, with outcomes (e.g., timing results) printed at the end.
• Layering: Think of the system in three main layers:
– Infrastructure Layer: Contains container orchestration, dependency installation (scripts like “install_cuda_docker.sh”) and CI/CD pipelines.
– Application/Benchmark Layer: This includes the actual training implementations (organized by benchmark category). For instance, the “language_model/tensorflow/bert” node would further capture elements like data pre-processing, model definition (in “modeling.py”, “optimization.py”), and training runner scripts (“run_pretraining.py”).
– Logging & Metric Layer: Utilizes common MLPerf logging utilities that capture performance metrics.
• Dependencies: Use directional flows to mark that each benchmark’s training process depends on data ingestion, environment setup (Docker), and then reports metrics (logging). A “common” module (or shared libraries/services) can be drawn as a side-car component that interacts with multiple benchmarks.
Architectural Patterns and Technologies
• Microservice/Modular Design: Emphasize that each benchmark is an independent module that can be built, run, and tested separately. Use containers (Docker components) to encapsulate the environment for each benchmark.
• Framework Diversity: Note the difference in frameworks – some benchmarks are implemented using TensorFlow (e.g., BERT), and others with PyTorch (e.g., Stable Diffusion, LLama2, recommendation models). Clearly annotate these nodes with the underlying frameworks or libraries.