# microsoft/lora architecture

> Architecture diagram of the GitHub repository microsoft/lora, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/lora
- Repository: https://github.com/microsoft/lora

## Overview

The LoRA repository is an open‐source Python library focused on “Low-Rank Adaptation” to efficiently fine-tune large language models by adding low-rank matrices into select weights. The overall system is a library (loralib) that you integrate with existing deep learning frameworks (mainly PyTorch) and transformer models from Hugging Face, with two major example use cases (NLG for generation and NLU for language understanding/classification tasks).

##### Main components
- Core Library (loralib): This is the heart of the project. It contains modules (e.g., layers.py, loralib/utils.py) that implement adaptations for neural network layers. It provides wrappers for PyTorch layers (Linear, Embedding, Conv2d) and adds low-rank training capabilities.
- Integration Examples:
– NLG: Contains examples implementing LoRA on text-generation tasks using GPT-2. It includes data, training scripts, evaluation scripts, pretrained checkpoint handling, and even notebook examples.
– NLU: Contains examples applying LoRA to transformer-based classification and language understanding (e.g., RoBERTa, DeBERTa) as used in benchmarks like GLUE. This directory is structured with subdirectories for training, evaluation, docker setups, tests, and extensive integration with Hugging Face’s transformers.
- Test and Documentation Components: There are extensive tests (in tests/) to ensure functionality and documentation (README, docs/, figures) that further explain use cases, guidelines, and examples.
- Utility & Configuration: The repository also includes scripts for checkpoint handling (saving/loading LoRA-specific state_dict), various training helpers (mark_only_lora_as_trainable), and configuration files for environment and building.

##### Establish the relationships and data flow
- The Core Library (loralib) is imported by both the NLG and NLU example pipelines. In these examples, existing transformer models (e.g., GPT-2, RoBERTa) from Hugging Face are “adapted” by replacing or wrapping certain layers with LoRA counterparts.
- During training, the system follows this flow:
– Load a pre-trained checkpoint from Hugging Face.
– Apply the LoRA wrappers to adapt selected layers (for instance, the “q” and “v” projection layers in Transformer architectures).
– Mark LoRA parameters as trainable while freezing the original weights.
– Save a specialized checkpoint (using a custom LoRA state_dict that only contains adaptation parameters).
– When running inference, the LoRA parameters can be merged with the original pre-trained weights to eliminate extra latency.

##### Architectural patterns and design principles
- Modularity: The repository is divided into a core library and multiple independent example integrations. This separation allows flexibility in adapting the LoRA methodology to different model architectures and tasks.
- Reusability: The same utility functions (e.g., marking LoRA parameters trainable, merging weights) are reused across various examples.
- Testability: A dedicated tests directory ensures separation of test code from production/example code.
- Extensibility: Templates provided (under examples/NLU/templates, etc.) allow adding new models or examples quickly.

##### Specific elements
- Sub-components from examples (for NLG: data handling, training, evaluation; for NLU: pipelines, docker configurations, docs) to reflect differences in use cases.
- Checkpoint flow (pretrained checkpoint → LoRA modification → training → saving reduced state_dict → merged inference) and the training configuration options (e.g., which biases are trainable).

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Dependencies
    subgraph "External Dependencies"
        EP["Pre-trained Models (Hugging Face & PyTorch)"]:::external
    end

    %% Core Library
    subgraph "Core Layer"
        LORA["loralib Core Library\n(Low-Rank Adapter for PyTorch)"]:::core
    end

    %% Example Integrations
    subgraph "Example Integrations"
        NLG["Example: NLG Integration\n(GPT-2 Based)"]:::example
        NLU["Example: NLU Integration\n(Transformer Classification)"]:::example
    end

    %% Training & Inference Workflow
    subgraph "Training & Inference Workflow"
        T1["LoadPretrainedCheckpoint"]:::workflow
        T2["ApplyLoRAWrappers"]:::workflow
        T3["Train&FreezeOriginalWeights"]:::workflow
        T4["SaveLoRAStateDict"]:::workflow
        T5["MergeForInference"]:::workflow
    end

    %% Infrastructure: Testing & Documentation and Utility & Configuration
    subgraph "Infrastructure"
        subgraph "Testing & Documentation"
            TEST["Tests"]:::infra
            DOC["Documentation"]:::infra
        end
        subgraph "Utility & Configuration"
            SETUP["Setup"]:::infra
            ENV["Env&Utils"]:::infra
        end
    end

    %% Connections
    EP -->|"loads"| T1
    T1 -->|"applies"| T2
    NLG -->|"calls"| LORA
    NLU -->|"calls"| LORA
    LORA -->|"integrated_in"| T2
    T2 -->|"proceeds"| T3
    T3 -->|"saves"| T4
    T4 -->|"prepares"| T5
    T5 -->|"feeds_into"| NLG
    T5 -->|"feeds_into"| NLU

    NLU -->|"tested_by"| TEST
    NLG -->|"documented_by"| DOC
    NLU -->|"documented_by"| DOC
    NLU -->|"configured_by"| SETUP
    NLU -->|"configured_by"| ENV

    %% Styles
    classDef core fill:#ffcccc,stroke:#333,stroke-width:2px;
    classDef example fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef workflow fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef external fill:#ffebcc,stroke:#333,stroke-width:2px;
    classDef infra fill:#e6ccff,stroke:#333,stroke-width:2px;

    %% Click Events
    click LORA "https://github.com/microsoft/lora/tree/main/loralib/"
    click NLG "https://github.com/microsoft/lora/tree/main/examples/NLG/"
    click NLU "https://github.com/microsoft/lora/tree/main/examples/NLU/"
    click TEST "https://github.com/microsoft/lora/tree/main/examples/NLU/tests/"
    click DOC "https://github.com/microsoft/lora/blob/main/README.md"
    click SETUP "https://github.com/microsoft/lora/blob/main/setup.py"
    click ENV "https://github.com/microsoft/lora/blob/main/examples/NLU/environment.yml"
```
