How fawad-laal/qalb-urdu works
This repository is an independent bilingual evaluation framework for Qalb, rather than the model itself. The primary workflow runs Urdu-script and Roman Urdu benchmark cases against Qalb, scores the responses, and saves results for analysis and reporting. Its documented scope includes eight task categories, iterative assessment rounds, performance analysis, and PDF and GPT-assisted Markdown reports. The README documents the runner’s role and outputs; the runner implementation was not sampled, so internal execution and scoring links are shown only at that documented level.
Main entry points: test_runner.py Test runner · generate_final_report.py GPT Markdown report · analyze_round4.py Round analysis · analyze_low_performers.py Failure analysis
Components 17
Evaluation workflow
- Test runner
scripts/test_runner.py - Qalb model
- Response scoring
scripts/test_runner.py - Run results
Benchmark inputs
- Urdu cases
tests/baseline/urdu_script_tests.json - Roman Urdu cases
tests/baseline/roman_urdu_tests.json - Task categories
tests/test_cases.py
Analysis
- Round analysis
scripts/analyze_round4.py - Failure analysis
scripts/analyze_low_performers.py - Round 2 analysis
scripts/analyze_round2.py
Reporting
- Data collector
scripts/data_collector.py - Report data
- PDF report
scripts/generate_pdf_report.py - Academic report
scripts/generate_academic_pdf.py - GPT Markdown report
scripts/generate_final_report.py
Other components
- Evaluator
- OpenAI service
Connections 15
- Evaluator → Test runner: starts run
- Test runner → Urdu cases: loads cases
- Test runner → Roman Urdu cases: loads cases
- Test runner → Task categories: uses categories
- Test runner → Qalb model: sends prompts
- Test runner → Response scoring: scores responses
- Response scoring → Run results: records scores
- Round analysis → Run results: analyzes results
- Failure analysis → Run results: analyzes failures
- Round 2 analysis → Run results: analyzes results
- Data collector → Report data: aggregates and saves
- Report data → PDF report: loads report data
- Run results → GPT Markdown report: provides results
- GPT Markdown report → OpenAI service: requests analysis
- Report data → Academic report: supplies evaluation content