# hysonlab/videberta architecture

> Architecture diagram of the GitHub repository hysonlab/videberta, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hysonlab/videberta
- Repository: https://github.com/hysonlab/videberta

## Overview

##### Project type and purpose
- This is an ML research codebase for pre-training and fine-tuning a Vietnamese DeBERTa-v3 transformer (“ViDeBERTa”).
- The key workflows are:
a. Pre-training a tokenizer and a DeBERTa model on raw Vietnamese text.
b. Packaging the pre-trained artifacts into the “model” folder.
c. Fine-tuning on downstream tasks (POS/NER, QA, open-domain QA).

##### Top-level components
- pre-training/
– bash/ : orchestration scripts (pre_tokenizer.sh, pre-train_tokenizer.sh, pre-train_model.sh)
– config/: JSON model & tokenizer configs
– source/: Python modules implementing each step (data splitting, SentencePiece tokenization, model training)
- model/
– Outputs of pre-training: tokenizer files (.json), special_tokens_map, tokenizer_config
- fine-tuning/
– POS_NER/: scripts for part-of-speech tagging & NER
– QA/: scripts & data for machine reading comprehension
– ORQA/: open-retrieval QA notebooks and data
- README.md & LICENSE

##### High-level dataflow and relationships
1. Raw text datasets (external HDD/remote storage)
→ pre-training/source/pre_tokenizer.py
→ PyVi segmentation & split (90/5/5) → datasets saved to disk
2. Tokenizer training
bash/pre-train_tokenizer.sh
→ source/pre-train_tokenizer.py
→ SentencePiece models → tokenizer artifacts (model/tokenizer.json)
3. Model pre-training
bash/pre-train_model.sh
→ source/pre-training.py
→ Loads tokenized datasets + tokenizer + config.json
→ Trains DeBERTa-v3 on GPUs
→ Emits model checkpoints → model/
4. Fine-tuning
– POS_NER: bash/run.sh → token-classification.py
– QA: Machine_reading_comprehension.ipynb & utils/preprocess.py
– ORQA: ORQA_Evaluation.ipynb
Each uses the pre-trained model + task-specific data → fine-tuned models + metrics

##### Architectural patterns and principles
- **Pipeline modularity**: each stage (tokenizer, model training, evaluation) is its own script/module
- **Config-driven**: JSON & YAML configs control hyperparameters
- **Separation of concerns**: Bash scripts orchestrate; Python modules implement logic
- **Experiment tracking**: Weights & Biases (wandb/) for logs, metrics, artifacts

- Blue for data
- Green for processing modules
- Yellow for configs
- Orange for artifacts

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Storage
    RD["Raw Text Datasets"]:::data
    DW["Processed Datasets"]:::data

    %% Pre-processing
    RD -->|"split & segment"| PT["Pre-tokenizer"]:::process
    click PT "https://github.com/hysonlab/videberta/blob/main/pre-training/source/pre_tokenizer.py"
    PT --> DW

    %% Tokenizer Training
    subgraph "Tokenizer Training"
        TTrainScript["pre-train_tokenizer.sh"]:::process
        click TTrainScript "https://github.com/hysonlab/videberta/blob/main/pre-training/bash/pre-train_tokenizer.sh"
        TTrainModule["SentencePiece Trainer"]:::process
        click TTrainModule "https://github.com/hysonlab/videberta/blob/main/pre-training/source/pre-train_tokenizer.py"
        TCfgTT["Tokenizer Configs (.json)"]:::config
        click TCfgTT "https://github.com/hysonlab/videberta/blob/main/pre-training/config/*.json"
        DW --> TTrainScript
        TTrainScript --> TTrainModule
        TTrainModule -->|"tokenizer.json"| TA["Tokenizer Artifact"]:::artifact
    end

    %% Model Pre-training
    subgraph "Model Training"
        MTrainScript["pre-train_model.sh"]:::process
        click MTrainScript "https://github.com/hysonlab/videberta/blob/main/pre-training/bash/pre-train_model.sh"
        MTrainModule["DeBERTa Trainer"]:::process
        click MTrainModule "https://github.com/hysonlab/videberta/blob/main/pre-training/source/pre-training.py"
        CfgPT["Model Configs (.json)"]:::config
        click CfgPT "https://github.com/hysonlab/videberta/blob/main/pre-training/config/*.json"
        DW --> MTrainScript
        MTrainScript --> MTrainModule
        MTrainModule -->|"checkpoint.pth"| MA["Model Artifacts"]:::artifact
    end

    %% Packaged Model Artifacts
    MA -->|"tokenizer.json"| ModelTok["tokenizer.json"]:::artifact
    click ModelTok "https://github.com/hysonlab/videberta/blob/main/model/tokenizer.json"
    MA -->|"tokenizer_config.json"| ModelTokCfg["tokenizer_config.json"]:::artifact
    click ModelTokCfg "https://github.com/hysonlab/videberta/blob/main/model/tokenizer_config.json"
    MA -->|"special_tokens_map.json"| ModelSTM["special_tokens_map.json"]:::artifact
    click ModelSTM "https://github.com/hysonlab/videberta/blob/main/model/special_tokens_map.json"

    %% Fine-tuning Stage
    subgraph "Fine-tuning"
        FW["Fine-tuning Inputs"]:::data
        ModelTok & ModelTokCfg & ModelSTM --> FW
        DW --> FW

        subgraph "POS/NER"
            PONRun["run.sh"]:::process
            click PONRun "https://github.com/hysonlab/videberta/blob/main/fine-tuning/POS_NER/bash/run.sh"
            PONScript["token-classification.py"]:::process
            click PONScript "https://github.com/hysonlab/videberta/blob/main/fine-tuning/POS_NER/source/token-classification.py"
            FW --> PONRun --> PONScript
        end

        subgraph "QA"
            QANb["Machine_reading_comprehension.ipynb"]:::process
            click QANb "https://github.com/hysonlab/videberta/blob/main/fine-tuning/QA/Machine_reading_comprehension.ipynb"
            QAPrep["preprocess.py"]:::process
            click QAPrep "https://github.com/hysonlab/videberta/blob/main/fine-tuning/QA/utils/preprocess.py"
            FW --> QANb --> QAPrep
        end

        subgraph "ORQA"
            ORQaNb["ORQA_Evaluation.ipynb"]:::process
            click ORQaNb "https://github.com/hysonlab/videberta/blob/main/fine-tuning/ORQA/ORQA_Evaluation.ipynb"
            ORQaData["ORQA Data (.json)"]:::data
            click ORQaData "https://github.com/hysonlab/videberta/blob/main/fine-tuning/ORQA/data/*.json"
            FW --> ORQaNb
        end
    end

    %% Experiment Tracking
    WBPT["W&B Pre-training"]:::external
    WBPON["W&B POS/NER"]:::external
    TTrainModule -.-> WBPT
    MTrainModule -.-> WBPT
    PONScript -.-> WBPON

    %% Styles
    classDef data fill:#add8e6,stroke:#000,color:#000
    classDef process fill:#98fb98,stroke:#000,color:#000
    classDef config fill:#ffff99,stroke:#000,color:#000
    classDef artifact fill:#ffa500,stroke:#000,color:#000
    classDef external fill:#d3d3d3,stroke:#000,color:#000
```
