# project-lighter/lighter architecture

> Architecture diagram of the GitHub repository project-lighter/lighter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/project-lighter/lighter
- Repository: https://github.com/project-lighter/lighter

## Overview

##### Project type and purpose
- **Project Type:** An open‐source Python framework for machine-learning experiment orchestration (not a web or mobile app).
- **Purpose:** Provide configurable building blocks—data adapters, callback hooks, runners, schema validation—to streamline training, evaluation, and logging of ML models.

##### File structure
- **Top‐level modules:**
- `lighter/engine` – Core orchestration (config loading, resolver, runner, schema).
- `lighter/adapters.py` – Data‐source adapters plugin interface.
- `lighter/callbacks` – Hookable lifecycle callbacks (writers, freezers, metrics, etc.).
- `lighter/system.py` – High‐level entry point that ties engine + callbacks + adapters.
- `lighter/utils` – Shared utilities (dynamic imports, logging, type definitions).
- **Supporting folders:**
- `projects/` – Example usage (e.g., CIFAR-10 experiment).
- `docker/`, `docs/`, `tests/` — Dev/deployment and documentation.
- **Config & tooling:**
- `pyproject.toml`, `Makefile`, `.github/workflows/ci.yml` – Build, test, and release pipelines.
- YAML files in `projects` and `tests` – Declarative experiment configurations.

##### Extract insights from README
- Emphasizes extensibility via callbacks and adapters.
- Shows how end users define experiments in YAML and run them via the CLI entry point.
- Mentions features like “inferers,” “freezers,” and “writers” (all implemented as callbacks).

a. **Main components**
1. **CLI / Entry Point** (`lighter/system.py`)
2. **Configuration Loader** (YAML → Python objects)
3. **Schema Resolver** (`engine/resolver.py` + `engine/schema.py`)
4. **Runner** (`engine/runner.py`) – Orchestrates training/inference loops
5. **Adapters** (`lighter/adapters.py`) – Data ingestion plugins
6. **Callbacks** (`lighter/callbacks/…`)
– Writers (file, table)
– Freezers (model checkpointing)
– Metrics & inferers
7. **Utilities** (`lighter/utils/…`) – logging, dynamic imports, type helpers
8. **User Experiment** (YAML + model code under `projects/`)
9. **External ML Frameworks** (e.g., PyTorch, TensorBoard, Weights & Biases)
10. **Output Stores** (files, tables, dashboards)

b. **Relationships & data flow**
- **Start:** User invokes CLI → Configuration Loader
- Loader passes config into **Schema Resolver** → validated PipelineSpec
- **Runner** takes PipelineSpec → registers **Adapters** for data, instantiates **Model** & **Optimizer**
- **Runner** enters loop, triggering **Callbacks** at hook points (on_epoch_start, on_batch_end, on_train_end)
- **Writers** output logs, metrics to files or remote dashboards
- **Freezers** checkpoint models to disk
- **Inferers** run evaluation on validation/test sets
- **Utilities** (logging, dynamic import) assist at every stage

c. **Architectural patterns**
- **Plugin pattern** via dynamic imports (adapters, callbacks)
- **Observer pattern** for callback hooks
- **Declarative config** driving imperative engine logic

d. **Key technologies**
- Python 3.9+
- YAML for configs
- pytest for tests
- docker for reproducible environments
- mkdocs for docs
- CI via GitHub Actions

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Engine
    subgraph "Core Engine" 
        direction TB
        CLI["CLI / Entry Point"]:::core
        Config["Configuration Loader"]:::core
        Resolver["Schema Resolver"]:::core
        Runner["Runner"]:::core
    end

    %% Extensions
    subgraph "Plugin Extensions"
        direction TB
        Adapter["Adapters Interface"]:::plugin
        subgraph "Callbacks"
            direction TB
            Freezer["Callback: Freezer"]:::plugin
            WriterBase["Writer: Base"]:::plugin
            WriterFile["Writer: File"]:::plugin
            WriterTable["Writer: Table"]:::plugin
            CallbackUtils["Callback Utilities"]:::plugin
        end
    end

    %% Utilities
    subgraph "Utilities"
        direction TB
        DataUtils["Data Helpers"]:::util
        DynImport["Dynamic Imports"]:::util
        LoggingUtil["Logging"]:::util
        MiscUtils["Miscellaneous Utilities"]:::util
        ModelUtils["Model Utilities"]:::util
        Patches["Patches"]:::util
        Types["Type Definitions"]:::util
    end

    %% User Experiment
    subgraph "User Experiment"
        direction TB
        ExpConfig["Experiment Configs"]:::user
        ModelCode["Model Code"]:::user
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        ExtML["External ML Frameworks"]:::external
        Outputs["Output Stores"]:::external
    end

    %% Documentation
    subgraph "Documentation"
        direction TB
        DocsAdapters["How-To Adapters"]:::docs
        DocsFreezers["How-To Freezers"]:::docs
        DocsInferers["How-To Inferers"]:::docs
        DocsMetrics["How-To Metrics"]:::docs
        DocsWriters["How-To Writers"]:::docs
        DocsProject["How-To Project Module"]:::docs
        DocsRun["How-To Run"]:::docs
    end

    %% Core flow
    CLI -->|"loads"| Config
    Config -->|"validates"| Resolver
    Resolver -->|"produces PipelineSpec"| Runner
    Runner -->|"registers"| Adapter
    Runner -->|"instantiates Model & Optimizer"| ModelCode
    Runner -.->|"uses helpers"| DataUtils
    Runner -.-> DynImport
    Runner -.-> LoggingUtil

    %% Callback hooks
    Runner -->|"on_epoch_start/on_batch_end/on_train_end"| Freezer
    Runner --> WriterBase
    Runner --> WriterFile
    Runner --> WriterTable
    Runner --> CallbackUtils

    %% Data flow
    Adapter -->|"provides batches"| Runner
    Renewable[Dummy]:::hidden

    %% Outputs
    WriterFile -->|"writes to files"| Outputs
    WriterTable -->|"writes to tables"| Outputs
    Freezer -->|"checkpoints to disk"| Outputs
    CallbackUtils -->|"evaluates with"| ExtML

    %% User experiment integration
    ExpConfig -->|"drives"| Config
    ExpConfig -->|"specifies overrides"| Runner

    %% External frameworks
    Runner --> ExtML

    %% Documentation links
    CLI --> DocsRun
    Adapter --> DocsAdapters
    Freezer --> DocsFreezers
    CallbackUtils --> DocsInferers
    WriterBase --> DocsMetrics
    WriterFile --> DocsWriters
    ExpConfig --> DocsProject

    %% Click events
    click CLI "https://github.com/project-lighter/lighter/blob/main/lighter/system.py"
    click Config "https://github.com/project-lighter/lighter/blob/main/lighter/engine/config.py"
    click Resolver "https://github.com/project-lighter/lighter/blob/main/lighter/engine/resolver.py"
    click Resolver "https://github.com/project-lighter/lighter/blob/main/lighter/engine/schema.py"
    click Runner "https://github.com/project-lighter/lighter/blob/main/lighter/engine/runner.py"
    click Adapter "https://github.com/project-lighter/lighter/blob/main/lighter/adapters.py"
    click Freezer "https://github.com/project-lighter/lighter/blob/main/lighter/callbacks/freezer.py"
    click WriterBase "https://github.com/project-lighter/lighter/blob/main/lighter/callbacks/writer/base.py"
    click WriterFile "https://github.com/project-lighter/lighter/blob/main/lighter/callbacks/writer/file.py"
    click WriterTable "https://github.com/project-lighter/lighter/blob/main/lighter/callbacks/writer/table.py"
    click CallbackUtils "https://github.com/project-lighter/lighter/blob/main/lighter/callbacks/utils.py"
    click DataUtils "https://github.com/project-lighter/lighter/blob/main/lighter/utils/data.py"
    click DynImport "https://github.com/project-lighter/lighter/blob/main/lighter/utils/dynamic_imports.py"
    click LoggingUtil "https://github.com/project-lighter/lighter/blob/main/lighter/utils/logging.py"
    click MiscUtils "https://github.com/project-lighter/lighter/blob/main/lighter/utils/misc.py"
    click ModelUtils "https://github.com/project-lighter/lighter/blob/main/lighter/utils/model.py"
    click Patches "https://github.com/project-lighter/lighter/blob/main/lighter/utils/patches.py"
    click Types "https://github.com/project-lighter/lighter/tree/main/lighter/utils/types/"
    click ExpConfig "https://github.com/project-lighter/lighter/blob/main/projects/cifar10/experiments/example.yaml"
    click ExpConfig "https://github.com/project-lighter/lighter/blob/main/projects/cifar10/experiments/example_overrides.yaml"
    click ModelCode "https://github.com/project-lighter/lighter/blob/main/projects/cifar10/models/net.py"
    click DocsAdapters "https://github.com/project-lighter/lighter/blob/main/docs/how-to/adapters.md"
    click DocsFreezers "https://github.com/project-lighter/lighter/blob/main/docs/how-to/freezers.md"
    click DocsInferers "https://github.com/project-lighter/lighter/blob/main/docs/how-to/inferers.md"
    click DocsMetrics "https://github.com/project-lighter/lighter/blob/main/docs/how-to/metrics.md"
    click DocsWriters "https://github.com/project-lighter/lighter/blob/main/docs/how-to/writers.md"
    click DocsProject "https://github.com/project-lighter/lighter/blob/main/docs/how-to/project_module.md"
    click DocsRun "https://github.com/project-lighter/lighter/blob/main/docs/how-to/run.md"

    %% Styles
    classDef core fill:#cdefff,stroke:#009,stroke-width:1px
    classDef plugin fill:#e0f7e0,stroke:#070,stroke-width:1px
    classDef util fill:#ffe0b2,stroke:#e65c00,stroke-width:1px
    classDef user fill:#fff9c4,stroke:#cca800,stroke-width:1px
    classDef external fill:#f8bbd0,stroke:#c2185b,stroke-width:1px
    classDef docs fill:#e0e0e0,stroke:#555,stroke-width:1px
    classDef hidden fill:#fff,stroke:none;color:#fff
```
