How wissamantoun/camemberta works
Project Type and Purpose
• This is an end-to-end machine-learning training pipeline for pretraining and fine-tuning French language models (CamemBERTa, CamemBERTv2, CamemBERTav2) using TensorFlow 2, Horovod, and optional TPU support.
• Key use cases:
• Extract and prepare massive French text corpora
• Tokenize and shard data into TFRecords
• Train a generator/discriminator (ELECTRA-style RTD) model with gradient-disentangled embedding sharing (GDES)
• Export checkpoints, convert to PyTorch, and serve via HuggingFace
File-Structure Analysis
• Top-level components:
· Dockerfile, Dockerfile.aws for containerized environments
· configs/ for JSON hyperparameter sets (phase 1/phase 2, local/GPU/TPU, Horovod)
· data/ for downloaders, corpus formatting, sharding, and build_pretraining_dataset*.py scripts
· fast_tokenizer/ for custom tokenizer conversions and implementations
· modeling_tf_*.py & configuration_*.py for model definitions (DeBERTaV2, RoBERTa) and GDES logic
· official_utils/ for TF2 training-loop abstractions, distribution strategies, file I/O, and export utilities
· run_pretraining.py, run_hvd.sh, run_pretraining.sbatch for launching distributed training
· conversion & postprocessing: postprocess_pretrained_ckpt.py, convert_to_pt.py
· vocab/ holds pretrained tokenizer files
• Patterns & choices:
• Modular separation of data-prep, tokenization, model, training-utils, export
• Config-driven orchestration (JSON files + configuration_*.py)
• Support for multiple backends (Horovod, TF Strategy, TPU)
README Insights
• Describes two-phase pretraining (seq-len 512 then 1024)
• Highlights GDES embedded in TFDebertaV3Embeddings and PretrainingModel
• Lists external dependencies (HuggingFace, NVIDIA TF2 examples) and compute environments (OAR/SLURM HPC, Docker/Singularity)
• Defines model components (generator vs discriminator, shared embeddings with GDES)
a. Main Components
1. Data Ingestion & Preparation
2. Tokenization & TFRecord Sharding
3. Model Definition
– Generator (MLM head)
– Discriminator (RTD head)
– GDES embedding layer
4. Training Orchestration
– Horovod / TF2 Strategy / TPU cluster
– Configuration loader
5. Checkpoint Storage & Export
– TF checkpoints → postprocess_pretrained_ckpt.py → PyTorch via convert_to_pt.py
6. Downstream Fine-tuning & Serving (HuggingFace)
b. Relationships & Data Flow
• Raw French corpora → data/ downloaders → corpus formatting → TextSharding → TFRecords
• TFRecords + configs → run_pretraining → distributed training → generator + discriminator iterate on batches
• Training jobs read model definitions (modeling_tf_*.py) & utility functions (model_training_utils, pretrain_utils)
• Checkpoints written to storage → postprocess → convert_to_pt → final model artifacts on disk or HuggingFace hub
c. Architectural Patterns
• Pipeline pattern: clear data-prep → train → export stages
• Modular code organization for easy extension (e.g. swap RoBERTa vs DeBERTaV2)
• Config-driven, containerized reproducible experiments
d. Key Technologies & Frameworks
• TensorFlow 2, Horovod, TPU support (official_utils/accelerator)
• HuggingFace Transformers for inference & fine-tuning
• Docker/Singularity for environment
• NVIDIA BERT/ELECTRA TF2 example code as upstream inspiration