# tebmer/awesome-knowledge-distillation-of-llms architecture

> Architecture diagram of the GitHub repository tebmer/awesome-knowledge-distillation-of-llms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tebmer/awesome-knowledge-distillation-of-llms
- Repository: https://github.com/tebmer/awesome-knowledge-distillation-of-llms

## Overview

Below are the guidelines tailored for this project:

##### Main “Components” of the Repository
- Content Repository – The README file is the heart of the project. It acts as the landing “interface” that presents multiple research categories.
- Categories / Sections – The README is divided into sections such as:
– Introduction – Explaining knowledge distillation (KD) in LLMs and the survey’s purpose.
– KD Algorithms (further subdivided into Knowledge Elicitation and Distillation Algorithms with numerous subcategories like Labeling, Expansion, Curation, etc.)
– Skill Distillation – Covering aspects like Context Following, Alignment, Agent, NLP Task Specialization, and Multi-Modality.
– Verticalization Distillation – Covering domains like Law, Medical & Healthcare, Finance, Science, etc.
– Encoder-based KD – An extra note on related work.
– Citation, Update News, and Star History.
- Update Process – The repository is updated weekly, and the “News” and “Contributing” sections reflect a dynamic element (process of regularly feeding in new research).

##### Relationships and Flow Between Components
- Branch out into high-level groups:
– KD Algorithms, Skill Distillation, Verticalization Distillation, and Encoder-based KD.
- Under each group, further branch into the subcategories (for example, under KD Algorithms, include “Knowledge Elicitation” and “Distillation Algorithms” with their respective subnodes like Labeling, Feedback, etc.).
- Additional components for external elements:
– “External Links & Resources” associated with each paper (e.g., links to arXiv, GitHub, datasets).
- Continuous contribution/feedback loop with nodes for “News” and “Contributing.”

##### Architectural Patterns and Design Principles
- Modular Categorization: Each category (KD Algorithms, Skill Distillation, etc.) is treated as an independent module that itself may have submodules.
- Update Cycle: Emphasize that the repository is designed to be updated regularly (weekly updates) to incorporate new papers and changes in the taxonomy.

##### Relevant Technologies/Frameworks
– Use one style (e.g., rounded rectangles) for main text sections.

- “Central Repository / Landing Page” node (the README as the central hub).
- Branches representing the major sections:
– KD Algorithms (with sub-nodes for the various knowledge distillation methods).
– Skill Distillation (with branches like Instruction Following, Multi-turn Dialogue, RAG Capability, etc.)
– Verticalization Distillation (with domain-specific branches).
– Encoder-based KD.
- Side nodes for supplementary information:
– Visual Assets (imgs folder)
– Update Process (weekly contribution and update log)
– Citation and Star History sections.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Central Node
    A["Central Repository / Landing Page (README)"]:::center

    %% Introduction Node
    B["Introduction: Overview & Purpose"]:::section
    A -->|"presents"| B

    %% KD Algorithms Subgraph
    subgraph "KD Algorithms"
        KD1["Knowledge Elicitation"]:::subcat
        KD2["Distillation Algorithms"]:::subcat
    end

    %% Knowledge Elicitation Subcategories
    KD1a["Labeling"]:::subcatDetail
    KD1b["Expansion"]:::subcatDetail
    KD1c["Curation"]:::subcatDetail
    KD1d["Feature"]:::subcatDetail
    KD1e["Feedback"]:::subcatDetail
    KD1f["Self-Knowledge"]:::subcatDetail
    KD1 --> KD1a
    KD1 --> KD1b
    KD1 --> KD1c
    KD1 --> KD1d
    KD1 --> KD1e
    KD1 --> KD1f

    %% Distillation Algorithms Subcategories
    KD2a["Supervised Fine-Tuning"]:::subcatDetail
    KD2b["Divergence_and_Similarity"]:::subcatDetail
    KD2c["Reinforcement Learning"]:::subcatDetail
    KD2d["Rank Optimization"]:::subcatDetail
    KD2 --> KD2a
    KD2 --> KD2b
    KD2 --> KD2c
    KD2 --> KD2d

    %% Skill Distillation Subgraph
    subgraph "Skill Distillation"
        SD1["Context Following"]:::subcat
        SD2["Alignment"]:::subcat
        SD3["Agent"]:::subcat
        SD4["NLP Task Specialization"]:::subcat
        SD5["Multi-Modality"]:::subcat
    end

    %% Verticalization Distillation Subgraph
    subgraph "Verticalization Distillation"
        V1["Law"]:::subcat
        V2["Medical & Healthcare"]:::subcat
        V3["Finance"]:::subcat
        V4["Science"]:::subcat
        V5["Misc"]:::subcat
    end

    %% Encoder-based KD Node
    E["Encoder-based KD (Related Work)"]:::category

    %% Side Nodes
    M["Metadata: Citation, UpdateNews, StarHistory"]:::side
    VAssets["Visual Assets (imgs)"]:::side
    UP["Update Process (News_Contributing)"]:::side
    ER["External Resources (arXiv_GitHub_DataLinks)"]:::external

    %% Flow from Introduction to Major Categories
    B -->|"navigates"| KD1
    B -->|"navigates"| KD2
    B -->|"navigates"| SD1
    B -->|"navigates"| V1
    B -->|"navigates"| E

    %% Connections from Central Repository to Side Nodes
    A --> UP
    A --> M
    A --> VAssets
    A --> ER

    %% External References from Categories
    KD1 -->|"references"| ER
    KD2 -->|"references"| ER
    SD1 -->|"references"| ER
    V1 -->|"references"| ER
    E  -->|"references"| ER

    %% Click Events for Component Mapping
    click A "https://github.com/tebmer/awesome-knowledge-distillation-of-llms/blob/main/README.md"
    click VAssets "https://github.com/tebmer/awesome-knowledge-distillation-of-llms/tree/main/imgs"

    %% Styles
    classDef center fill:#f9c,stroke:#333,stroke-width:2px;
    classDef section fill:#bbf,stroke:#333,stroke-width:2px;
    classDef category fill:#dfd,stroke:#333,stroke-width:2px;
    classDef subcat fill:#ffd,stroke:#333,stroke-width:2px;
    classDef subcatDetail fill:#fcf,stroke:#333,stroke-width:1px;
    classDef side fill:#cff,stroke:#333,stroke-width:2px;
    classDef external fill:#fee,stroke:#333,stroke-width:2px;
```
