# wissamantoun/camemberta architecture

> Architecture diagram of the GitHub repository wissamantoun/camemberta, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wissamantoun/camemberta
- Repository: https://github.com/wissamantoun/camemberta

## Overview

##### Project Type and Purpose
- This is an end-to-end machine-learning training pipeline for pretraining and fine-tuning French language models (CamemBERTa, CamemBERTv2, CamemBERTav2) using TensorFlow 2, Horovod, and optional TPU support.
- Key use cases:
- Extract and prepare massive French text corpora
- Tokenize and shard data into TFRecords
- Train a generator/discriminator (ELECTRA-style RTD) model with gradient-disentangled embedding sharing (GDES)
- Export checkpoints, convert to PyTorch, and serve via HuggingFace

##### File-Structure Analysis
- Top-level components:
· Dockerfile, Dockerfile.aws for containerized environments
· configs/ for JSON hyperparameter sets (phase 1/phase 2, local/GPU/TPU, Horovod)
· data/ for downloaders, corpus formatting, sharding, and `build_pretraining_dataset*.py` scripts
· fast_tokenizer/ for custom tokenizer conversions and implementations
· modeling_tf_*.py & configuration_*.py for model definitions (DeBERTaV2, RoBERTa) and GDES logic
· official_utils/ for TF2 training-loop abstractions, distribution strategies, file I/O, and export utilities
· run_pretraining.py, run_hvd.sh, run_pretraining.sbatch for launching distributed training
· conversion & postprocessing: `postprocess_pretrained_ckpt.py`, `convert_to_pt.py`
· vocab/ holds pretrained tokenizer files
- Patterns & choices:
- Modular separation of data-prep, tokenization, model, training-utils, export
- Config-driven orchestration (JSON files + `configuration_*.py`)
- Support for multiple backends (Horovod, TF Strategy, TPU)

##### README Insights
- Describes two-phase pretraining (seq-len 512 then 1024)
- Highlights GDES embedded in `TFDebertaV3Embeddings` and `PretrainingModel`
- Lists external dependencies (HuggingFace, NVIDIA TF2 examples) and compute environments (OAR/SLURM HPC, Docker/Singularity)
- Defines model components (generator vs discriminator, shared embeddings with GDES)

a. Main Components
1. Data Ingestion & Preparation
2. Tokenization & TFRecord Sharding
3. Model Definition
– Generator (MLM head)
– Discriminator (RTD head)
– GDES embedding layer
4. Training Orchestration
– Horovod / TF2 Strategy / TPU cluster
– Configuration loader
5. Checkpoint Storage & Export
– TF checkpoints → `postprocess_pretrained_ckpt.py` → PyTorch via `convert_to_pt.py`
6. Downstream Fine-tuning & Serving (HuggingFace)
b. Relationships & Data Flow
- Raw French corpora → data/ downloaders → corpus formatting → TextSharding → TFRecords
- TFRecords + configs → run_pretraining → distributed training → generator + discriminator iterate on batches
- Training jobs read model definitions (modeling_tf_*.py) & utility functions (model_training_utils, pretrain_utils)
- Checkpoints written to storage → postprocess → convert_to_pt → final model artifacts on disk or HuggingFace hub
c. Architectural Patterns
- Pipeline pattern: clear data-prep → train → export stages
- Modular code organization for easy extension (e.g. swap RoBERTa vs DeBERTaV2)
- Config-driven, containerized reproducible experiments
d. Key Technologies & Frameworks
- TensorFlow 2, Horovod, TPU support (official_utils/accelerator)
- HuggingFace Transformers for inference & fine-tuning
- Docker/Singularity for environment
- NVIDIA BERT/ELECTRA TF2 example code as upstream inspiration

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Ingestion & Preparation"
        direction TB
        RawCorpora["Raw French Corpora"]:::data
        DataMod["data/"]:::data
        BuildScripts["build_pretraining_dataset*.py"]:::data
    end

    %% Tokenization Layer
    subgraph "Tokenization & TFRecord Sharding"
        direction TB
        FastTokenizerDir["fast_tokenizer/"]:::tokenize
        TextShard["TextSharding (data/TextSharding.py)"]:::tokenize
    end

    %% Model Definition Layer
    subgraph "Model Definition"
        direction TB
        DebertaV2["modeling_tf_deberta_v2.py"]:::model
        Roberta["modeling_tf_roberta.py"]:::model
        Outputs["modeling_tf_outputs.py"]:::model
        Utils["modeling_tf_utils.py"]:::model
        ConfigDeB["configuration_deberta_v2.py"]:::model
        ConfigRob["configuration_roberta.py"]:::model
    end

    %% Training Orchestration
    subgraph "Training Orchestration"
        direction TB
        RunPre["run_pretraining.py"]:::compute
        RunHVD["run_hvd.sh"]:::compute
        RunSB["run_pretraining.sbatch"]:::compute
        ConfigsDir["configs/"]:::compute
        ConfigUtils["configuration_utils.py"]:::compute
        ModelTU["model_training_utils.py"]:::compute
        PreUtils["pretrain_utils.py"]:::compute
        TPUAcc["official_utils/accelerator/tpu.py"]:::compute
        DistFlags["official_utils/flags/"]:::compute
    end

    %% Checkpoint & Export
    subgraph "Checkpoint Storage & Export"
        direction TB
        PostProc["postprocess_pretrained_ckpt.py"]:::storage
        ConvertPT["convert_to_pt.py"]:::storage
        ExportUtil["official_utils/export/export.py"]:::storage
    end

    %% Downstream Serving
    subgraph "Downstream Fine-tuning & Serving"
        direction TB
        VocabDir["vocab/"]:::external
        ConvertSlow["fast_tokenizer/convert_slow_tokenizer.py"]:::external
        HuggingFace["HuggingFace Model Hub"]:::external
    end

    %% Flow
    RawCorpora -->|feeds| DataMod
    DataMod -->|prepares| BuildScripts
    BuildScripts -->|outputs| FastTokenizerDir
    BuildScripts -->|shards| TextShard
    TextShard -->|TFRecords| RunPre
    FastTokenizerDir -->|tokenizes| RunPre
    RunPre -->|loads| DebertaV2
    RunPre -->|loads| Roberta
    RunPre -->|uses| Outputs
    RunPre -->|uses| Utils
    RunPre -->|reads| ConfigDeB
    RunPre -->|reads| ConfigRob
    ConfigsDir -->|provides| RunPre
    RunHVD -->|wraps| RunPre
    RunSB -->|schedules| RunPre
    ConfigUtils -->|helpers| RunPre
    ModelTU -->|helpers| RunPre
    PreUtils -->|helpers| RunPre
    TPUAcc -->|enables| RunPre
    DistFlags -->|flags| RunPre
    RunPre -->|writes| PostProc
    PostProc -->|creates| ConvertPT
    ConvertPT -->|exports| ExportUtil
    ExportUtil -->|publishes| HuggingFace
    ConvertSlow -->|converts| HuggingFace
    VocabDir -->|provides| ConvertSlow

    %% Click events
    click DataMod "https://github.com/wissamantoun/camemberta/tree/main/data/"
    click BuildScripts "https://github.com/wissamantoun/camemberta/blob/main/build_pretraining_dataset.py"
    click BuildScripts "https://github.com/wissamantoun/camemberta/blob/main/build_pretraining_dataset_culturax.py"
    click BuildScripts "https://github.com/wissamantoun/camemberta/blob/main/build_pretraining_dataset_culturax_long.py"
    click BuildScripts "https://github.com/wissamantoun/camemberta/blob/main/build_pretraining_dataset_hf.py"
    click BuildScripts "https://github.com/wissamantoun/camemberta/blob/main/build_pretraining_dataset_hf_long.py"
    click FastTokenizerDir "https://github.com/wissamantoun/camemberta/tree/main/fast_tokenizer/"
    click TextShard "https://github.com/wissamantoun/camemberta/blob/main/data/TextSharding.py"
    click DebertaV2 "https://github.com/wissamantoun/camemberta/blob/main/modeling_tf_deberta_v2.py"
    click Roberta "https://github.com/wissamantoun/camemberta/blob/main/modeling_tf_roberta.py"
    click Outputs "https://github.com/wissamantoun/camemberta/blob/main/modeling_tf_outputs.py"
    click Utils "https://github.com/wissamantoun/camemberta/blob/main/modeling_tf_utils.py"
    click ConfigDeB "https://github.com/wissamantoun/camemberta/blob/main/configuration_deberta_v2.py"
    click ConfigRob "https://github.com/wissamantoun/camemberta/blob/main/configuration_roberta.py"
    click RunPre "https://github.com/wissamantoun/camemberta/blob/main/run_pretraining.py"
    click RunHVD "https://github.com/wissamantoun/camemberta/blob/main/run_hvd.sh"
    click RunSB "https://github.com/wissamantoun/camemberta/blob/main/run_pretraining.sbatch"
    click ConfigsDir "https://github.com/wissamantoun/camemberta/tree/main/configs/"
    click ConfigUtils "https://github.com/wissamantoun/camemberta/blob/main/configuration_utils.py"
    click ModelTU "https://github.com/wissamantoun/camemberta/blob/main/model_training_utils.py"
    click PreUtils "https://github.com/wissamantoun/camemberta/blob/main/pretrain_utils.py"
    click TPUAcc "https://github.com/wissamantoun/camemberta/blob/main/official_utils/accelerator/tpu.py"
    click DistFlags "https://github.com/wissamantoun/camemberta/tree/main/official_utils/flags/"
    click PostProc "https://github.com/wissamantoun/camemberta/blob/main/postprocess_pretrained_ckpt.py"
    click ConvertPT "https://github.com/wissamantoun/camemberta/blob/main/convert_to_pt.py"
    click ExportUtil "https://github.com/wissamantoun/camemberta/blob/main/official_utils/export/export.py"
    click VocabDir "https://github.com/wissamantoun/camemberta/tree/main/vocab/"
    click ConvertSlow "https://github.com/wissamantoun/camemberta/blob/main/fast_tokenizer/convert_slow_tokenizer.py"

    %% Styles
    classDef data fill:#BBDEFB,stroke:#0277BD,color:#000
    classDef tokenize fill:#B3E5FC,stroke:#0288D1,color:#000
    classDef model fill:#C8E6C9,stroke:#2E7D32,color:#000
    classDef compute fill:#C5E1A5,stroke:#558B2F,color:#000
    classDef storage fill:#FFE0B2,stroke:#EF6C00,color:#000
    classDef external fill:#CFD8DC,stroke:#455A64,color:#000
```
