How scbdatax/genai-datax-language-aware-token-boosting works
Key Components to Include:
Core Components
• LATB Module (Main Logic)
• Adaptive-LATB Module
• Language Identification System
• Evaluation Pipeline
• Experiment Runner
External Dependencies
• FastText Language ID Model
• Baseline Models (Suzume 8B)
• XLSUM Dataset
Main Processing Flow Layer
• Input text flow
• Language identification process
• Token boosting logic (both vanilla and adaptive)
• Output generation
Core Modules Layer (from latb/)
• Language ID Module (lang_id.py)
• LLM Interface (llms.py)
• Logit Processors (logit_processors.py)
• Utility Functions (utils/)
Evaluation Pipeline (latb/evaluate/)
• Base Evaluator
• Different evaluation metrics:
• ROUGE Score
• Average Token Length
• Language Confusion
Experiment Framework
Experiment runner architecture:
• Baseline implementations
• Script runners
• Configuration management
To both the core modules and evaluation pipeline.
Specific Design Elements:
Component Relationships
• Double lines: API boundaries
Technical Details
• Model specifications (Llama3 8B, Suzume 8B)
• Key hyperparameters (α for LATB, β for Adaptive-LATB)
• Language identification confidence thresholds
Environment Context
• CUDA dependency
• Poetry package management
• FastText integration points
For implementation details, use UML-style notation where appropriate, especially for showing the relationship between the base evaluator and its specialized implementations.