How mbzuai-oryx/kitab-bench works
Project Type & Purpose
• Project type: A Python-based, multi-module benchmarking toolkit for Arabic OCR and document-understanding tasks.
File Structure & README
• Root: README, LICENSE, static/ (documentation assets).
• Nine domain folders (e.g., ocr-eval, layout-eval, tables-eval, etc.). Each contains:
– eval.py / main.py: orchestrator script
– metrics.py (or metrics_utils.py): computes task-specific metrics
– models/: wrappers for each model (Tesseract, EasyOCR, Gemini, GPT-4o, Qwen, etc.)
– prompts.py (for LLM-based tasks)
– requirements.txt
– domain-specific data (e.g., pdfs-eval has pdfs/ + labels/)
• Key takeaway: Modular, repeatable pipeline per domain.
Main Components & Their Roles
A. CLI / Entry Point
– Single-command launch of each domain’s pipeline via Python scripts.
B. Data Loader
– Reads raw inputs (images, PDFs, markdown labels).
C. Model Interface Layer
– Abstracts multiple backends:
• Traditional OCR engines (Tesseract, EasyOCR)
• Vision-language and LLM APIs (GPT-4o, Gemini, Qwen-VL)
• Specialized models (RT-DETR, YOLO, Surya for layout, DocLayNet, Img2Table)
– Handles authentication (OPENAI_API_KEY, GEMINI_API_KEY)
D. Inference Engine
– Calls the chosen model wrapper, passing data & prompts, returns raw outputs.
E. Metrics Calculator
– Domain-specific metrics modules compute CER, WER, MARS, TEDS, SCRM, CODM, mAP, BLEU, etc.
F. Results Aggregator & Export
– Writes evaluation results (JSON, CSV, console logs).
G. Visualization / Reporting (optional)
– Static images (in docs) illustrate performance, but not part of runtime pipeline.
Interactions & Data Flow
1. User invokes python eval.py --model_name X in a domain folder.
2. CLI parses arguments → calls Data Loader.
3. Data Loader fetches input samples.
4. For each sample, pass to Model Interface → Inference Engine.
5. Collect predictions → feed into Metrics Calculator.
6. Metrics Calculator outputs scores → Results Aggregator logs them.
7. (Optional) Combine all domain results for summary performance table.
Architectural Patterns & Principles
• Modular Monorepo: each domain is a self-contained module.
• Strategy Pattern: Model wrappers implement a common interface (predict()), enabling swap-in of any engine.
• Pipeline Pattern: Clear sequence—Load → Infer → Evaluate → Report.
• Dependency Injection: Models and metrics passed as parameters to eval scripts.
• Separation of Concerns:
– Data loading, inference, and metrics are separated into distinct files/modules.
– Domain data (pdfs, labels) lives alongside its pipeline.