How vietnamaihub/vietnamese_llms works
Main Components
• Data Generation & Translation Module:
• “Generate_and_Translate_Dataset” directory as a component that handles both dataset generation (using human and machine-generated instructions) and translation.
• Dataset Generation: Include “Generation_instruction_OpenAI_api.py” that leverages the Alpaca template.
• Translation: Show two distinct translation pipelines – one using “Using_OpenAI_Translate_API.py” and one using “Using_NLLB_MetaAI_Translate.py.” Also, illustrate the input/output flow associated with the Vietnamese instruction datasets stored under “Vietnamese_Instructions_datasets” (with separated subfolders for Human-generated and Translation outputs).
• Model Fine-Tuning & Training Module:
• “LLMs” directory as a component used for fine-tuning language models. Indicate submodules for:
• Finetuning using LoRA (“Finetune_llm_LoRA.py”)
• Finetuning using QLoRA (“Finetune_llm_QLoRA.py”)
• Inference of fine-tuned models (“inference_llm.py”)
• Mention the integration with optimization tools (e.g., references to DeepSpeed in the “Utils/Deepspeed_config” directory).
• Web UI / Frontend Interface:
• “WebUI” directory as a component that hosts the user interface. This module (with “assistant_gradio.py”) is responsible for serving a Gradio-based demonstration, enabling users to interact with the fine-tuned models.
• Utility & Configurations:
• A “Utils” component that holds shared configurations such as DeepSpeed JSON files and dataset templates (like “alpaca.json”).
• Also, display how the licensing, environment setup (environment.yml), and contribution guidelines (contribute.md) support the overall system.
Important Architectural Patterns/Design Principles
• Separation of Concerns: Different components handle dataset generation, model fine-tuning, and user interaction.
• Extensibility: The design is modular; new translation methods or training pipelines can be added to the corresponding modules.
• Pipeline Flow: Emphasize the sequential process – from data generation to translation, then fine-tuning, and finally serving via an interactive UI.
Incorporate Relevant Technologies
• Under Data Generation, mention Python scripts and API integrations (OpenAI and NLLB).
• Under Model Training, note the use of fine-tuning techniques (LoRA, QLoRA), potentially backed by DeepSpeed (from the provided configuration files).
• Under UI, indicate the use of the Gradio framework.
• Also, briefly mark integration with external datasets and model checkpoints (referenced in the README with Hugging Face links).
• Every element, such as the datasets directory with its separate sub-folders (Human-generated, Translation – divided into Translational outputs with GPT-3.5, GPT-4, NLLB, etc.), is clearly annotated.
• Where appropriate, put in brief tool names (e.g., OpenAI API, MetaAI’s NLLB) that inform readers of the methods used.
– Starting with dataset generation and translation pipelines,
– Flowing into the fine-tuning/training process of LLMs,
– And ending with the Gradio-based Web UI that allows for model inference and demoing.