How hungnlp/vibiogpt works
Project Type and Purpose
• The project focuses on fine-tuning a large language model for biomedical Q&A in Vietnamese.
• It provides both training and inference capabilities, with training scripts (training.py, train.sh, train_debug.sh) and an inference script (inference.py) tailored for adapting the base model to a specific medical Q&A dataset.
• The README clearly explains usage examples, dependency management, and training pipelines.
Analyzing the File Structure
• Data Layer:
• Data files (under the “data” directory, e.g., qa_medical_pairs.json, qa_pairs_edoctor.json, qa_vinmec.json) serve as the training dataset.
• Tools like dataset.py and data_collators.py are responsible for preparing and batching the data.
• Prompting Utilities:
• The “prompt” folder (with base.py, qa_prompt.py, role.py) defines the chat prompt structures and roles which are vital for interaction during inference.
• Training and Inference Pipelines:
• The training pipeline is orchestrated by training.py and the associated shell scripts (train.sh, train_debug.sh) which leverage settings from arguments.py, constants.py, and utils.py.
• The inference.py script provides command-line-driven inference to generate responses using the fine-tuned model.
• Tokenization:
• tokenizer.py configures and manages how text is tokenized, ensuring compatibility with both training and inference processes.
• Configuration & Environment:
• The .env.example and dependency files (requirements.txt) suggest environment configuration and dependency management.
How to Represent the System
a. Main Components:
• Data Ingestion & Preprocessing: (data files, dataset.py, data_collators.py)
• Prompt Infrastructure: (prompt/ folder, base and qa prompt logic)
• Training Pipeline: (training.py, train.sh/scripts, arguments.py, constants.py)
• Inference Pipeline: (inference.py, tokenizer.py, utils.py)
• External Dependencies/Models: (Pre-trained Vistral-Chat-7B model, the fine-tuned adapter, and external libraries like Transformers, PEFT, bitsandbytes, Flash Attention)
• Environment & Configuration: (.env, requirements.txt)
b. Relationships & Interactions:
• Data flows from the dataset (JSON files in the data folder) into the preprocessing modules (dataset.py and data_collators.py).
• The training pipeline fetches the pre-trained model, applies the QLora-based fine-tuning (using training.py and shell scripts), and incorporates prompt structures (via the prompt folder).
• The inference module leverages both the tokenizer and prompt templates to generate answers.
• Environment variables (from .env) and dependency configurations (requirements.txt) are used across the pipelines.
• Data flow: from raw dataset → preprocessing → training/inference.
• Control flow: from training scripts to model fine-tuning and then to inference.
• Dependency links: e.g., the model fetch from Hugging Face using tokens from the environment.
c. Important Architectural Patterns & Design Principles:
• Separation of concerns is clearly maintained between training and inference.
• Modularity through dedicated files for data management, prompting, tokenization, and utility functions.
• Extensibility in design, as evidenced by the clear demarcation of the prompt template logic (making it easier to adjust the conversational context).
d. Relevant Technologies and Frameworks:
• Transformer-based Models: Transformers, PEFT library.
• QLora Fine-Tuning Technique.
• Quantization libraries like bitsandbytes and Flash Attention.
• Python ecosystem with dependencies managed in requirements.txt.
• External model hosting from Hugging Face (both pre-trained and fine-tuned adapter).
• Use clear labels for each component (e.g., “Data Loader”, “Data Preprocessor”, “Training Pipeline”, “Prompt Templates”, “Inference Engine”).
▪ Rectangles for scripts/modules.
▪ Rounded rectangles for external services (Hugging Face model repository) and environment configuration.