How ashleve/lightning-hydra-template works
Main Components
• Configurations (Hydra & YAML): The “configs” directory is the heart of the dynamic configuration system. It includes subdirectories for callbacks, data, model settings, trainer setups, experiment configs, and even hyperparameter search (Optuna). This is where parameters are set and later instantiated via Hydra.
• Source Code (“src”):
• Data: The “src/data” folder contains modules (like the MNIST datamodule) that manage data loading and processing.
• Models: The “src/models” directory includes model definitions and building blocks. For example, you have “mnist_module.py” and components (e.g. “simple_dense_net.py”) for network architecture.
• Utilities: Under “src/utils”, various helpers are defined (like instantiation logic and logging utilities) that support training and configuration.
• Entrypoints: “src/train.py” (for training the model) and “src/eval.py” (for evaluating a model) serve as the command-line interfaces where the Hydra config and dynamic instantiation are executed.
• Data & Logs:
• The “data” directory is intended for datasets (even if it only contains a placeholder).
• The “logs” directory is structured by Hydra to store output from training runs (including checkpoints, logger outputs, etc.).
• Tests & CI: The “tests” folder holds the testing logic (using pytest) and the “.github/workflows” directory contains CI configurations that run tests, formatting via pre-commit, and code quality jobs.
• Additional Support Scripts: Files like the Makefile, pre-commit configuration, and scripts (e.g. “scripts/schedule.sh”) support development workflow and deployment.
Relationships and Interactions
• At launch (via “src/train.py”), Hydra reads the configuration specified by “configs/train.yaml” (or an experiment config in “configs/experiment”) which in turn references model, data, trainer, logger, and callback configurations.
• Hydra instantiation in “src/train.py” dynamically creates the datamodule (from “src/data/mnist_datamodule.py”) and the Lightning module (from “src/models/mnist_module.py”). These components interact via the abstraction provided by PyTorch Lightning.
• Logger configurations (in “configs/logger”) set up experiment tracking with tools like Weights & Biases, Tensorboard, or MLFlow.
• Hyperparameter search and multi-run experiments are supported by Hydra’s sweeper integration, with dedicated configuration entries in “configs/hparams_search”.
• The tests directory interacts with nearly all core functionalities, ensuring that training, evaluation, and configuration instantiation execute as expected.
• CI workflows (in “.github/workflows”) continually run tests and check code quality to ensure that changes do not break any of these interactions.
Architectural Patterns & Design Principles
• Configuration by Composition: The project uses Hydra to layer configurations (using defaults and overrides), which makes the architecture very modular and flexible.
• Dependency Injection: Components (e.g., models, data modules, callbacks) are dynamically instantiated based on configuration, which decouples their implementation from the orchestration.
• Separation of Concerns: There is a clear split between configuration (configs/), application logic (src/), data management (data/ and src/data), and artifacts/logs (logs/).
• Extensibility: The design encourages adding new models, datasets, experiments, and logging strategies simply by adding new configuration files and Python modules.
Relevant Technologies, Frameworks, and Libraries
• Python 3.8–3.10, PyTorch, and PyTorch Lightning (for abstraction of training logic).
• Hydra (for hierarchical dynamic configuration).
• Various logger integrations (e.g., Tensorboard, W&B, MLFlow) as per the logger configs.
• Optuna (for hyperparameter search via Hydra’s integration).
• Testing frameworks (pytest) and CI tools (GitHub Actions, pre-commit).
• At the top, list “Hydra Configuration” with branches into “Callbacks,” “Data,” “Model,” “Trainer,” “Logger,” and “Experiment” configs.
• This to the “src” layer where “train.py” uses these configurations to instantiate:
• The Data Module (src/data)
• The Model (src/models)
• Utilities (src/utils)
• From “train.py” → Training Loop (managed by PyTorch Lightning) → Logging and Checkpoints saving (logs/)
• Optionally, add CI/CD and testing as peripheral components that continuously validate the primary process.
• That every component is modular and configurable, which allows for rapid experimentation and easy extension for new data, models, and training paradigms.