# mbzuai-nlp/arabicmmlu architecture

> Architecture diagram of the GitHub repository mbzuai-nlp/arabicmmlu, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mbzuai-nlp/arabicmmlu
- Repository: https://github.com/mbzuai-nlp/arabicmmlu

## Overview

##### Project type and purpose
- This is a research‐oriented evaluation toolkit for assessing large language models on the ArabicMMLU benchmark. It’s not a web or end-user application but a command‐line driven evaluation pipeline.

##### File structure
- data/ArabicMMLU.csv – the raw benchmark questions and answers
- util_prompt.py – routines to assemble prompts for zero-shot or few-shot evaluation
- util_compute.py – functions to compute accuracy/metrics on model outputs
- evaluate.py – the main orchestrator: loads data, calls prompt routines, queries models, gathers outputs, and computes metrics
- run.sh – a shell script that sets environment variables, installs dependencies, and invokes evaluate.py for different models or configurations
- requirements.txt – lists Python dependencies (e.g., datasets, transformers, OpenAI client)
- README.md – documentation of dataset, evaluation procedure, and citation

##### README for additional insights
- The pipeline supports multiple open‐source and closed models, with prompts in English or Arabic.
- Output artifacts include numeric results and bar charts, but charting code is likely embedded in util_compute or post‐processing.

- Data Store – the CSV dataset under /data
- Prompt Builder (util_prompt.py)
- Evaluation Orchestrator (evaluate.py + run.sh)
- Model Inference Layer – huggingface/transformers clients or OpenAI API
- Metrics Computation (util_compute.py)
- Results & Visualization (console output, saved metrics, plots)
- External Services – Hugging Face Hub, OpenAI endpoints

1. Data Store → Evaluation Orchestrator (load CSV via datasets.load_dataset)
2. Evaluation Orchestrator → Prompt Builder (pass question + choices → formatted prompt)
3. Prompt Builder → Model Inference Layer (send HTTP/API or local model call)
4. Model Inference Layer → Evaluation Orchestrator (return raw model answer)
5. Evaluation Orchestrator → Metrics Computation (submit ground truth + predictions)
6. Metrics Computation → Results & Visualization (compute accuracy, generate bar charts)
7. run.sh wraps the orchestrator invocation and handles dependency installation

c. Architectural patterns & principles:
- Pipeline pattern – clear sequential stages (load → prompt → infer → compute → output)
- Separation of concerns – prompt logic separated from metric logic, orchestrator coordinates
- Pluggable models – easily swap out different model clients by configuring evaluate.py or run.sh

d. Key technologies & libraries:
- Python 3.x
- Hugging Face Datasets & Transformers
- OpenAI Python client (for closed‐source models)
- Shell scripting (run.sh)
- pandas or built-in CSV reader (implicit)
- matplotlib or seaborn for charting (if present in util_compute)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Components
    DataStore["Data Store\n(ArabicMMLU.csv)"]:::data
    PromptBuilder["Prompt Builder\n(util_prompt.py)"]:::processing
    EvalOrch["Evaluation Orchestrator\n(evaluate.py)"]:::orchestrator
    Bootstrap["Bootstrap & Env Setup\n(run.sh)"]:::orchestrator
    Metrics["Metrics Computation & Visualization\n(util_compute.py)"]:::processing
    Dependencies["Dependencies\n(requirements.txt)"]:::processing
    Docs["Documentation\n(README.md)"]:::data
    Results["Results & Visualization\n(Console output, charts)"]:::output

    subgraph "Model Inference Layer"
        HFHub["Hugging Face Hub"]:::external
        OpenAI["OpenAI API"]:::external
    end

    subgraph "Evaluation Pipeline"
        DataStore -->|"CSV rows"| EvalOrch
        EvalOrch -->|"questions + choices"| PromptBuilder
        PromptBuilder -->|"text prompt"| HFHub
        PromptBuilder -->|"text prompt"| OpenAI
        HFHub -->|"model response"| EvalOrch
        OpenAI -->|"model response"| EvalOrch
        EvalOrch -->|"predictions + ground truth"| Metrics
        Metrics -->|"accuracy metrics & charts"| Results
    end

    Dependencies --> Bootstrap
    Docs --> Bootstrap
    Bootstrap --> EvalOrch

    %% Click Events
    click DataStore "https://github.com/mbzuai-nlp/arabicmmlu/blob/main/data/ArabicMMLU.csv"
    click PromptBuilder "https://github.com/mbzuai-nlp/arabicmmlu/blob/main/util_prompt.py"
    click EvalOrch "https://github.com/mbzuai-nlp/arabicmmlu/blob/main/evaluate.py"
    click Bootstrap "https://github.com/mbzuai-nlp/arabicmmlu/blob/main/run.sh"
    click Metrics "https://github.com/mbzuai-nlp/arabicmmlu/blob/main/util_compute.py"
    click Dependencies "https://github.com/mbzuai-nlp/arabicmmlu/blob/main/requirements.txt"
    click Docs "https://github.com/mbzuai-nlp/arabicmmlu/blob/main/README.md"

    %% Styles
    classDef data fill:#D0E6FF,stroke:#0366d6,color:#0366d6;
    classDef processing fill:#DEFDE0,stroke:#28A745,color:#116530;
    classDef external fill:#FFE5B4,stroke:#D97706,color:#B45309;
    classDef orchestrator fill:#E9D5FF,stroke:#7E22CE,color:#5B21B6;
    classDef output fill:#F3F4F6,stroke:#6B7280,color:#374151;
```
