How project-lighter/lighter works
Project type and purpose
• Project Type: An open‐source Python framework for machine-learning experiment orchestration (not a web or mobile app).
• Purpose: Provide configurable building blocks—data adapters, callback hooks, runners, schema validation—to streamline training, evaluation, and logging of ML models.
File structure
• Top‐level modules:
• lighter/engine – Core orchestration (config loading, resolver, runner, schema).
• lighter/adapters.py – Data‐source adapters plugin interface.
• lighter/callbacks – Hookable lifecycle callbacks (writers, freezers, metrics, etc.).
• lighter/system.py – High‐level entry point that ties engine + callbacks + adapters.
• lighter/utils – Shared utilities (dynamic imports, logging, type definitions).
• Supporting folders:
• projects/ – Example usage (e.g., CIFAR-10 experiment).
• docker/, docs/, tests/ — Dev/deployment and documentation.
• Config & tooling:
• pyproject.toml, Makefile, .github/workflows/ci.yml – Build, test, and release pipelines.
• YAML files in projects and tests – Declarative experiment configurations.
Extract insights from README
• Emphasizes extensibility via callbacks and adapters.
• Shows how end users define experiments in YAML and run them via the CLI entry point.
• Mentions features like “inferers,” “freezers,” and “writers” (all implemented as callbacks).
a. Main components
1. CLI / Entry Point (lighter/system.py)
2. Configuration Loader (YAML → Python objects)
3. Schema Resolver (engine/resolver.py + engine/schema.py)
4. Runner (engine/runner.py) – Orchestrates training/inference loops
5. Adapters (lighter/adapters.py) – Data ingestion plugins
6. Callbacks (lighter/callbacks/…)
– Writers (file, table)
– Freezers (model checkpointing)
– Metrics & inferers
7. Utilities (lighter/utils/…) – logging, dynamic imports, type helpers
8. User Experiment (YAML + model code under projects/)
9. External ML Frameworks (e.g., PyTorch, TensorBoard, Weights & Biases)
10. Output Stores (files, tables, dashboards)
b. Relationships & data flow
• Start: User invokes CLI → Configuration Loader
• Loader passes config into Schema Resolver → validated PipelineSpec
• Runner takes PipelineSpec → registers Adapters for data, instantiates Model & Optimizer
• Runner enters loop, triggering Callbacks at hook points (on_epoch_start, on_batch_end, on_train_end)
• Writers output logs, metrics to files or remote dashboards
• Freezers checkpoint models to disk
• Inferers run evaluation on validation/test sets
• Utilities (logging, dynamic import) assist at every stage
c. Architectural patterns
• Plugin pattern via dynamic imports (adapters, callbacks)
• Observer pattern for callback hooks
• Declarative config driving imperative engine logic
d. Key technologies
• Python 3.9+
• YAML for configs
• pytest for tests
• docker for reproducible environments
• mkdocs for docs
• CI via GitHub Actions