How tebmer/awesome-knowledge-distillation-of-llms works
Below are the guidelines tailored for this project:
Main “Components” of the Repository
• Content Repository – The README file is the heart of the project. It acts as the landing “interface” that presents multiple research categories.
• Categories / Sections – The README is divided into sections such as:
– Introduction – Explaining knowledge distillation (KD) in LLMs and the survey’s purpose.
– KD Algorithms (further subdivided into Knowledge Elicitation and Distillation Algorithms with numerous subcategories like Labeling, Expansion, Curation, etc.)
– Skill Distillation – Covering aspects like Context Following, Alignment, Agent, NLP Task Specialization, and Multi-Modality.
– Verticalization Distillation – Covering domains like Law, Medical & Healthcare, Finance, Science, etc.
– Encoder-based KD – An extra note on related work.
– Citation, Update News, and Star History.
• Update Process – The repository is updated weekly, and the “News” and “Contributing” sections reflect a dynamic element (process of regularly feeding in new research).
Relationships and Flow Between Components
• Branch out into high-level groups:
– KD Algorithms, Skill Distillation, Verticalization Distillation, and Encoder-based KD.
• Under each group, further branch into the subcategories (for example, under KD Algorithms, include “Knowledge Elicitation” and “Distillation Algorithms” with their respective subnodes like Labeling, Feedback, etc.).
• Additional components for external elements:
– “External Links & Resources” associated with each paper (e.g., links to arXiv, GitHub, datasets).
• Continuous contribution/feedback loop with nodes for “News” and “Contributing.”
Architectural Patterns and Design Principles
• Modular Categorization: Each category (KD Algorithms, Skill Distillation, etc.) is treated as an independent module that itself may have submodules.
• Update Cycle: Emphasize that the repository is designed to be updated regularly (weekly updates) to incorporate new papers and changes in the taxonomy.
Relevant Technologies/Frameworks
– Use one style (e.g., rounded rectangles) for main text sections.
• “Central Repository / Landing Page” node (the README as the central hub).
• Branches representing the major sections:
– KD Algorithms (with sub-nodes for the various knowledge distillation methods).
– Skill Distillation (with branches like Instruction Following, Multi-turn Dialogue, RAG Capability, etc.)
– Verticalization Distillation (with domain-specific branches).
– Encoder-based KD.
• Side nodes for supplementary information:
– Visual Assets (imgs folder)
– Update Process (weekly contribution and update log)
– Citation and Star History sections.