# scbdatax/genai-datax-language-aware-token-boosting architecture

> Architecture diagram of the GitHub repository scbdatax/genai-datax-language-aware-token-boosting, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scbdatax/genai-datax-language-aware-token-boosting
- Repository: https://github.com/scbdatax/genai-datax-language-aware-token-boosting

## Overview

Key Components to Include:

##### Core Components
- LATB Module (Main Logic)
- Adaptive-LATB Module
- Language Identification System
- Evaluation Pipeline
- Experiment Runner

##### External Dependencies
- FastText Language ID Model
- Baseline Models (Suzume 8B)
- XLSUM Dataset

##### Main Processing Flow Layer
- Input text flow
- Language identification process
- Token boosting logic (both vanilla and adaptive)
- Output generation

##### Core Modules Layer (from latb/)
- Language ID Module (lang_id.py)
- LLM Interface (llms.py)
- Logit Processors (logit_processors.py)
- Utility Functions (utils/)

##### Evaluation Pipeline (latb/evaluate/)
- Base Evaluator
- Different evaluation metrics:
- ROUGE Score
- Average Token Length
- Language Confusion

##### Experiment Framework
Experiment runner architecture:
- Baseline implementations
- Script runners
- Configuration management
To both the core modules and evaluation pipeline.

Specific Design Elements:

##### Component Relationships
- Double lines: API boundaries

##### Technical Details
- Model specifications (Llama3 8B, Suzume 8B)
- Key hyperparameters (α for LATB, β for Adaptive-LATB)
- Language identification confidence thresholds

##### Environment Context
- CUDA dependency
- Poetry package management
- FastText integration points

For implementation details, use UML-style notation where appropriate, especially for showing the relationship between the base evaluator and its specialized implementations.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        input[Input Text]
    end

    subgraph "Core Processing"
        langid["Language ID Module"]:::core
        llm["LLM Interface"]:::core
        latb["LATB Module"]:::core
        alatb["Adaptive LATB"]:::core
    end

    subgraph "External Models"
        fasttext["FastText Model"]:::external
        llama["Llama3 8B"]:::external
        suzume["Suzume 8B"]:::external
    end

    subgraph "Evaluation System"
        evaluator["Base Evaluator"]:::eval
        rouge["ROUGE Score"]:::eval
        toklen["Token Length"]:::eval
        langconf["Language Confusion"]:::eval
    end

    subgraph "Experiment Framework"
        exprun["Experiment Runner"]:::exp
        baseline["Baseline Models"]:::exp
        respgen["Response Generator"]:::exp
    end

    subgraph "Analysis Tools"
        alpha["Alpha Analysis"]:::analysis
        beta["Beta Analysis"]:::analysis
    end

    subgraph "Utilities"
        langutil["Language Utils"]:::util
        statsutil["Statistics Utils"]:::util
    end

    %% Connections
    input --> langid
    langid --> latb
    latb --> llm
    llm --> alatb
    alatb --> respgen

    langid --> fasttext
    llm --> llama
    baseline --> suzume

    respgen --> evaluator
    evaluator --> rouge
    evaluator --> toklen
    evaluator --> langconf

    latb --> exprun
    alatb --> exprun
    exprun --> baseline

    latb --> alpha
    alatb --> beta

    langid --> langutil
    evaluator --> statsutil

    %% Click Events
    click langid "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/latb/lang_id.py"
    click llm "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/latb/llms.py"
    click latb "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/latb/logit_processors.py"
    click evaluator "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/latb/evaluate/base_evaluator.py"
    click rouge "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/latb/evaluate/rouge.py"
    click toklen "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/latb/evaluate/avg_token_length.py"
    click langconf "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/latb/evaluate/lang_confusion.py"
    click baseline "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/tree/main/experiments/sum/baselines/"
    click suzume "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/experiments/sum/baselines/suzume.py"
    click respgen "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/experiments/sum/gen_response.py"
    click alpha "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/analysis/alpha_varying.ipynb"
    click beta "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/analysis/beta_varying.ipynb"
    click langutil "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/latb/utils/lang.py"
    click statsutil "https://github.com/scbdatax/genai-datax-language-aware-token-boosting/blob/main/latb/utils/stats.py"

    %% Styles
    classDef core fill:#2196F3,stroke:#1565C0,color:white
    classDef external fill:#4CAF50,stroke:#2E7D32,color:white
    classDef eval fill:#FF9800,stroke:#EF6C00,color:white
    classDef exp fill:#9C27B0,stroke:#6A1B9A,color:white
    classDef analysis fill:#00BCD4,stroke:#00838F,color:white
    classDef util fill:#9E9E9E,stroke:#424242,color:white

    %% Legend
    subgraph Legend
        l1["Core Components"]:::core
        l2["External Models"]:::external
        l3["Evaluation"]:::eval
        l4["Experiments"]:::exp
        l5["Analysis"]:::analysis
        l6["Utilities"]:::util
    end
```
