How mbzuai-nlp/arabicmmlu works
Project type and purpose
• This is a research‐oriented evaluation toolkit for assessing large language models on the ArabicMMLU benchmark. It’s not a web or end-user application but a command‐line driven evaluation pipeline.
File structure
• data/ArabicMMLU.csv – the raw benchmark questions and answers
• util_prompt.py – routines to assemble prompts for zero-shot or few-shot evaluation
• util_compute.py – functions to compute accuracy/metrics on model outputs
• evaluate.py – the main orchestrator: loads data, calls prompt routines, queries models, gathers outputs, and computes metrics
• run.sh – a shell script that sets environment variables, installs dependencies, and invokes evaluate.py for different models or configurations
• requirements.txt – lists Python dependencies (e.g., datasets, transformers, OpenAI client)
• README.md – documentation of dataset, evaluation procedure, and citation
README for additional insights
• The pipeline supports multiple open‐source and closed models, with prompts in English or Arabic.
• Output artifacts include numeric results and bar charts, but charting code is likely embedded in util_compute or post‐processing.
• Data Store – the CSV dataset under /data
• Prompt Builder (util_prompt.py)
• Evaluation Orchestrator (evaluate.py + run.sh)
• Model Inference Layer – huggingface/transformers clients or OpenAI API
• Metrics Computation (util_compute.py)
• Results & Visualization (console output, saved metrics, plots)
• External Services – Hugging Face Hub, OpenAI endpoints
1. Data Store → Evaluation Orchestrator (load CSV via datasets.load_dataset)
2. Evaluation Orchestrator → Prompt Builder (pass question + choices → formatted prompt)
3. Prompt Builder → Model Inference Layer (send HTTP/API or local model call)
4. Model Inference Layer → Evaluation Orchestrator (return raw model answer)
5. Evaluation Orchestrator → Metrics Computation (submit ground truth + predictions)
6. Metrics Computation → Results & Visualization (compute accuracy, generate bar charts)
7. run.sh wraps the orchestrator invocation and handles dependency installation
c. Architectural patterns & principles:
• Pipeline pattern – clear sequential stages (load → prompt → infer → compute → output)
• Separation of concerns – prompt logic separated from metric logic, orchestrator coordinates
• Pluggable models – easily swap out different model clients by configuring evaluate.py or run.sh
d. Key technologies & libraries:
• Python 3.x
• Hugging Face Datasets & Transformers
• OpenAI Python client (for closed‐source models)
• Shell scripting (run.sh)
• pandas or built-in CSV reader (implicit)
• matplotlib or seaborn for charting (if present in util_compute)