How hysonlab/videberta works
Project type and purpose
• This is an ML research codebase for pre-training and fine-tuning a Vietnamese DeBERTa-v3 transformer (“ViDeBERTa”).
• The key workflows are:
a. Pre-training a tokenizer and a DeBERTa model on raw Vietnamese text.
b. Packaging the pre-trained artifacts into the “model” folder.
c. Fine-tuning on downstream tasks (POS/NER, QA, open-domain QA).
Top-level components
• pre-training/
– bash/ : orchestration scripts (pre_tokenizer.sh, pre-train_tokenizer.sh, pre-train_model.sh)
– config/: JSON model & tokenizer configs
– source/: Python modules implementing each step (data splitting, SentencePiece tokenization, model training)
• model/
– Outputs of pre-training: tokenizer files (.json), special_tokens_map, tokenizer_config
• fine-tuning/
– POS_NER/: scripts for part-of-speech tagging & NER
– QA/: scripts & data for machine reading comprehension
– ORQA/: open-retrieval QA notebooks and data
• README.md & LICENSE
High-level dataflow and relationships
1. Raw text datasets (external HDD/remote storage)
→ pre-training/source/pre_tokenizer.py
→ PyVi segmentation & split (90/5/5) → datasets saved to disk
2. Tokenizer training
bash/pre-train_tokenizer.sh
→ source/pre-train_tokenizer.py
→ SentencePiece models → tokenizer artifacts (model/tokenizer.json)
3. Model pre-training
bash/pre-train_model.sh
→ source/pre-training.py
→ Loads tokenized datasets + tokenizer + config.json
→ Trains DeBERTa-v3 on GPUs
→ Emits model checkpoints → model/
4. Fine-tuning
– POS_NER: bash/run.sh → token-classification.py
– QA: Machine_reading_comprehension.ipynb & utils/preprocess.py
– ORQA: ORQA_Evaluation.ipynb
Each uses the pre-trained model + task-specific data → fine-tuned models + metrics
Architectural patterns and principles
• Pipeline modularity: each stage (tokenizer, model training, evaluation) is its own script/module
• Config-driven: JSON & YAML configs control hyperparameters
• Separation of concerns: Bash scripts orchestrate; Python modules implement logic
• Experiment tracking: Weights & Biases (wandb/) for logs, metrics, artifacts
• Blue for data
• Green for processing modules
• Yellow for configs
• Orange for artifacts