How open-compass/lawbench works
LawBench is a benchmark for measuring large language models’ knowledge of Chinese law across memory, text understanding, and applied legal reasoning. Its user-to-result workflow is to provide task examples to a model, collect model predictions, then run task-specific evaluators and export scores and abstention rates. The repository also includes shared scoring and Chinese text-correction utilities. Evaluator dispatch and prediction loading are visible in source; the README documents datasets and prediction formats. The excerpts do not show how predictions are generated, so that link is documented at the product level.
Main entry points: main.py Evaluation Runner · annotator.py Edit Annotator · evaluation_functions Understanding Evaluators · merger.py Edit Merger
Components 19
Benchmark Data
- Task Examples
data - Model Predictions
predictions
Evaluation
- Evaluation Runner
evaluation/main.py - Memory Evaluators
evaluation/evaluation_functions - Understanding Evaluators
evaluation/evaluation_functions - Application Evaluators
evaluation/evaluation_functions - Score Results
evaluation/main.py
Scoring Utilities
- Basic Metrics
evaluation/utils/function_utils.py - Comprehension Metrics
evaluation/utils/comprehension_scores.py - Reading F1
evaluation/utils/rc_f1.py
Text Correction Tools
- Character Similarity
evaluation/utils/char_smi.py - Chinese Tokenizer
evaluation/utils/modules/tokenizer.py - BPE Tokenization
evaluation/utils/modules/tokenization.py - Token Alignment
evaluation/utils/modules/alignment.py - Edit Merger
evaluation/utils/modules/merger.py - Edit Annotator
evaluation/utils/modules/annotator.py - Edit Classifier
evaluation/utils/modules/classifier.py
Other components
- Benchmark User
- Language Model
Connections 22
- Benchmark User → Task Examples: selects tasks
README.md - Task Examples → Language Model: provides examples
README.md - Language Model → Model Predictions: produces predictions
README.md - Benchmark User → Evaluation Runner: runs evaluation
evaluation/main.py - Model Predictions → Evaluation Runner: loads predictions
evaluation/main.py - Evaluation Runner → Memory Evaluators: dispatches tasks
evaluation/main.py - Evaluation Runner → Understanding Evaluators: dispatches tasks
evaluation/main.py - Evaluation Runner → Application Evaluators: dispatches tasks
evaluation/main.py - Evaluation Runner → Score Results: writes scores
evaluation/main.py - Memory Evaluators → Basic Metrics: uses metrics
evaluation/evaluation_functions/jec_kd.py - Understanding Evaluators → Basic Metrics: uses metrics
evaluation/evaluation_functions/ljp_accusation.py - Understanding Evaluators → Comprehension Metrics: scores extraction
evaluation/evaluation_functions/xxcq.py - Application Evaluators → Reading F1: scores triggers
evaluation/evaluation_functions/sjjc.py - Comprehension Metrics → Reading F1: computes F1
evaluation/utils/comprehension_scores.py - Chinese Tokenizer → BPE Tokenization: uses BPE
evaluation/utils/modules/tokenizer.py - Token Alignment → Chinese Tokenizer: tokenizes text
evaluation/utils/modules/alignment.py - Edit Merger → Chinese Tokenizer: tokenizes edits
evaluation/utils/modules/merger.py - Edit Merger → Token Alignment: merges alignments
evaluation/utils/modules/merger.py - Edit Annotator → Token Alignment: aligns edits
evaluation/utils/modules/annotator.py - Edit Annotator → Edit Merger: merges edits
evaluation/utils/modules/annotator.py - Edit Annotator → Edit Classifier: classifies edits
evaluation/utils/modules/annotator.py - Edit Classifier → Character Similarity: uses similarity
evaluation/utils/modules/classifier.py