How jeff-2000/kreyolbench works
KreyolBench is a Haitian Creole NLP benchmark ecosystem, not just a scoring library. Its main implemented workflow is CLI-driven: load JSONL predictions and references, extract task-specific values, compute metrics, and save reproducible result records. Separate capabilities cover dataset schemas and provenance, source planning and governance audits, preprocessing, baselines, annotation, and Hugging Face dataset compatibility. The README documents several ecosystem branches whose runtime wiring was not sampled; those are shown without asserted call paths. Source excerpts establish the CLI, evaluation, and governance links.
Main entry points: cli.py CLI commands · governance.py Governance audit · runner.py Evaluation runner · io.py JSONL and hashes
Components 18
User entry
- CLI commands
src/kreyolbench/cli.py
Data and governance
- JSONL and hashes
src/kreyolbench/io.py - Dataset schemas
src/kreyolbench/schemas.py - Config registry
src/kreyolbench/registry.py - Source plans
src/kreyolbench/sources.py - Governance audit
src/kreyolbench/governance.py - Text preprocessing
src/kreyolbench/preprocessing
Benchmark research
- Dataset loader
datasets/kreyolbench_dataset.py - Task and source configs
configs - Annotation resources
annotation - Model baselines
src/kreyolbench/baselines
Evaluation
- Evaluation runner
src/kreyolbench/evaluation/runner.py - Task metrics
src/kreyolbench/evaluation/metrics.py - Result reporting
src/kreyolbench/evaluation/reporting.py - Evaluation results
eval/results
Results ecosystem
- Leaderboard
leaderboard - Hugging Face
Other components
- Researcher
Connections 22
- Researcher → CLI commands: runs commands
README.md - CLI commands → Evaluation runner: runs evaluation
src/kreyolbench/cli.py - CLI commands → JSONL and hashes: reads rows
src/kreyolbench/cli.py - CLI commands → Dataset schemas: validates rows
src/kreyolbench/cli.py - CLI commands → Config registry: loads YAML
src/kreyolbench/cli.py - CLI commands → Source plans: builds plans
src/kreyolbench/cli.py - CLI commands → Governance audit: audits repository
src/kreyolbench/cli.py - Evaluation runner → JSONL and hashes: reads data
src/kreyolbench/evaluation/runner.py - Evaluation runner → Task metrics: computes scores
src/kreyolbench/evaluation/runner.py - Evaluation runner → Result reporting: builds results
src/kreyolbench/evaluation/runner.py - Result reporting → Evaluation results: writes JSON
src/kreyolbench/evaluation/reporting.py - CLI commands → Result reporting: exports leaderboard
src/kreyolbench/cli.py - Source plans → Governance audit: validates records
src/kreyolbench/sources.py - Governance audit → JSONL and hashes: uses helpers
src/kreyolbench/governance.py - Governance audit → Config registry: uses registry
src/kreyolbench/governance.py - Governance audit → Dataset schemas: uses records
src/kreyolbench/governance.py - Text preprocessing → JSONL and hashes: hashes text
src/kreyolbench/preprocessing/dedupe.py - Dataset loader → Hugging Face: supports compatibility
README.md - Task and source configs → Dataset schemas: defines task data
- Annotation resources → Dataset loader: annotates examples
- Model baselines → Evaluation runner: provides model outputs
- Leaderboard → Evaluation results: records submissions