# open-compass/lawbench architecture

> Architecture diagram of the GitHub repository open-compass/lawbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/open-compass/lawbench
- Repository: https://github.com/open-compass/lawbench

## Overview

LawBench is a benchmark for measuring large language models’ knowledge of Chinese law across memory, text understanding, and applied legal reasoning. Its user-to-result workflow is to provide task examples to a model, collect model predictions, then run task-specific evaluators and export scores and abstention rates. The repository also includes shared scoring and Chinese text-correction utilities. Evaluator dispatch and prediction loading are visible in source; the README documents datasets and prediction formats. The excerpts do not show how predictions are generated, so that link is documented at the product level.

Main entry points: [`evaluation/main.py`](https://github.com/open-compass/lawbench/blob/main/evaluation/main.py) (Evaluation Runner), [`evaluation/utils/modules/annotator.py`](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/annotator.py) (Edit Annotator), [`evaluation/evaluation_functions`](https://github.com/open-compass/lawbench/tree/main/evaluation/evaluation_functions) (Understanding Evaluators), [`evaluation/utils/modules/merger.py`](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/merger.py) (Edit Merger)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_benchmark_data["Benchmark Data"]
  node_task_data[("Task Examples")]
  node_prediction_sets[("Model Predictions")]
end

subgraph group_evaluation["Evaluation"]
  node_runner["Evaluation Runner<br/>[main.py]"]
  node_memory_evaluators["Memory Evaluators"]
  node_understanding_evaluators["Understanding Evaluators"]
  node_application_evaluators["Application Evaluators"]
  node_score_output["Score Results<br/>[main.py]"]
end

subgraph group_scoring["Scoring Utilities"]
  node_basic_metrics["Basic Metrics<br/>[function_utils.py]"]
  node_comprehension_metrics["Comprehension Metrics"]
  node_reading_f1["Reading F1<br/>[rc_f1.py]"]
end

subgraph group_text_tools["Text Correction Tools"]
  node_char_similarity["Character Similarity<br/>[char_smi.py]"]
  node_tokenizer["Chinese Tokenizer<br/>[tokenizer.py]"]
  node_tokenization["BPE Tokenization<br/>[tokenization.py]"]
  node_alignment["Token Alignment<br/>[alignment.py]"]
  node_edit_merger["Edit Merger<br/>[merger.py]"]
  node_annotator["Edit Annotator<br/>[annotator.py]"]
  node_classifier["Edit Classifier<br/>[classifier.py]"]
end

node_user(("Benchmark User"))
node_model(("Language Model"))

node_user -->|"selects tasks"| node_task_data
node_task_data -.->|"provides examples"| node_model
node_model -->|"produces predictions"| node_prediction_sets
node_user -->|"runs evaluation"| node_runner
node_prediction_sets -->|"loads predictions"| node_runner
node_runner -->|"dispatches tasks"| node_memory_evaluators
node_runner -->|"dispatches tasks"| node_understanding_evaluators
node_runner -->|"dispatches tasks"| node_application_evaluators
node_runner -->|"writes scores"| node_score_output
node_memory_evaluators -->|"uses metrics"| node_basic_metrics
node_understanding_evaluators -->|"uses metrics"| node_basic_metrics
node_understanding_evaluators -->|"scores extraction"| node_comprehension_metrics
node_application_evaluators -->|"scores triggers"| node_reading_f1
node_comprehension_metrics -->|"computes F1"| node_reading_f1
node_tokenizer -->|"uses BPE"| node_tokenization
node_alignment -->|"tokenizes text"| node_tokenizer
node_edit_merger -->|"tokenizes edits"| node_tokenizer
node_edit_merger -->|"merges alignments"| node_alignment
node_annotator -->|"aligns edits"| node_alignment
node_annotator -->|"merges edits"| node_edit_merger
node_annotator -->|"classifies edits"| node_classifier
node_classifier -->|"uses similarity"| node_char_similarity

click node_task_data "https://github.com/open-compass/lawbench/tree/main/data"
click node_prediction_sets "https://github.com/open-compass/lawbench/tree/main/predictions"
click node_runner "https://github.com/open-compass/lawbench/blob/main/evaluation/main.py"
click node_memory_evaluators "https://github.com/open-compass/lawbench/tree/main/evaluation/evaluation_functions"
click node_understanding_evaluators "https://github.com/open-compass/lawbench/tree/main/evaluation/evaluation_functions"
click node_application_evaluators "https://github.com/open-compass/lawbench/tree/main/evaluation/evaluation_functions"
click node_score_output "https://github.com/open-compass/lawbench/blob/main/evaluation/main.py"
click node_basic_metrics "https://github.com/open-compass/lawbench/blob/main/evaluation/utils/function_utils.py"
click node_comprehension_metrics "https://github.com/open-compass/lawbench/blob/main/evaluation/utils/comprehension_scores.py"
click node_reading_f1 "https://github.com/open-compass/lawbench/blob/main/evaluation/utils/rc_f1.py"
click node_char_similarity "https://github.com/open-compass/lawbench/blob/main/evaluation/utils/char_smi.py"
click node_tokenizer "https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/tokenizer.py"
click node_tokenization "https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/tokenization.py"
click node_alignment "https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/alignment.py"
click node_edit_merger "https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/merger.py"
click node_annotator "https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/annotator.py"
click node_classifier "https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/classifier.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_task_data,node_prediction_sets,node_user toneBlue
class node_runner,node_memory_evaluators,node_understanding_evaluators,node_application_evaluators,node_score_output toneAmber
class node_basic_metrics,node_comprehension_metrics,node_reading_f1 toneMint
class node_char_similarity,node_tokenizer,node_tokenization,node_alignment,node_edit_merger,node_annotator,node_classifier toneRose
class node_model toneIndigo
```

## Components

### Benchmark Data

- [Task Examples](https://github.com/open-compass/lawbench/tree/main/data): `data`
- [Model Predictions](https://github.com/open-compass/lawbench/tree/main/predictions): `predictions`

### Evaluation

- [Evaluation Runner](https://github.com/open-compass/lawbench/blob/main/evaluation/main.py): `evaluation/main.py`
- [Memory Evaluators](https://github.com/open-compass/lawbench/tree/main/evaluation/evaluation_functions): `evaluation/evaluation_functions`
- [Understanding Evaluators](https://github.com/open-compass/lawbench/tree/main/evaluation/evaluation_functions): `evaluation/evaluation_functions`
- [Application Evaluators](https://github.com/open-compass/lawbench/tree/main/evaluation/evaluation_functions): `evaluation/evaluation_functions`
- [Score Results](https://github.com/open-compass/lawbench/blob/main/evaluation/main.py): `evaluation/main.py`

### Scoring Utilities

- [Basic Metrics](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/function_utils.py): `evaluation/utils/function_utils.py`
- [Comprehension Metrics](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/comprehension_scores.py): `evaluation/utils/comprehension_scores.py`
- [Reading F1](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/rc_f1.py): `evaluation/utils/rc_f1.py`

### Text Correction Tools

- [Character Similarity](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/char_smi.py): `evaluation/utils/char_smi.py`
- [Chinese Tokenizer](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/tokenizer.py): `evaluation/utils/modules/tokenizer.py`
- [BPE Tokenization](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/tokenization.py): `evaluation/utils/modules/tokenization.py`
- [Token Alignment](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/alignment.py): `evaluation/utils/modules/alignment.py`
- [Edit Merger](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/merger.py): `evaluation/utils/modules/merger.py`
- [Edit Annotator](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/annotator.py): `evaluation/utils/modules/annotator.py`
- [Edit Classifier](https://github.com/open-compass/lawbench/blob/main/evaluation/utils/modules/classifier.py): `evaluation/utils/modules/classifier.py`

### Other components

- **Benchmark User**
- **Language Model**

## Connections

- Benchmark User → Task Examples: selects tasks (evidence: `README.md`)
- Task Examples → Language Model: provides examples (evidence: `README.md`)
- Language Model → Model Predictions: produces predictions (evidence: `README.md`)
- Benchmark User → Evaluation Runner: runs evaluation (evidence: `evaluation/main.py`)
- Model Predictions → Evaluation Runner: loads predictions (evidence: `evaluation/main.py`)
- Evaluation Runner → Memory Evaluators: dispatches tasks (evidence: `evaluation/main.py`)
- Evaluation Runner → Understanding Evaluators: dispatches tasks (evidence: `evaluation/main.py`)
- Evaluation Runner → Application Evaluators: dispatches tasks (evidence: `evaluation/main.py`)
- Evaluation Runner → Score Results: writes scores (evidence: `evaluation/main.py`)
- Memory Evaluators → Basic Metrics: uses metrics (evidence: `evaluation/evaluation_functions/jec_kd.py`)
- Understanding Evaluators → Basic Metrics: uses metrics (evidence: `evaluation/evaluation_functions/ljp_accusation.py`)
- Understanding Evaluators → Comprehension Metrics: scores extraction (evidence: `evaluation/evaluation_functions/xxcq.py`)
- Application Evaluators → Reading F1: scores triggers (evidence: `evaluation/evaluation_functions/sjjc.py`)
- Comprehension Metrics → Reading F1: computes F1 (evidence: `evaluation/utils/comprehension_scores.py`)
- Chinese Tokenizer → BPE Tokenization: uses BPE (evidence: `evaluation/utils/modules/tokenizer.py`)
- Token Alignment → Chinese Tokenizer: tokenizes text (evidence: `evaluation/utils/modules/alignment.py`)
- Edit Merger → Chinese Tokenizer: tokenizes edits (evidence: `evaluation/utils/modules/merger.py`)
- Edit Merger → Token Alignment: merges alignments (evidence: `evaluation/utils/modules/merger.py`)
- Edit Annotator → Token Alignment: aligns edits (evidence: `evaluation/utils/modules/annotator.py`)
- Edit Annotator → Edit Merger: merges edits (evidence: `evaluation/utils/modules/annotator.py`)
- Edit Annotator → Edit Classifier: classifies edits (evidence: `evaluation/utils/modules/annotator.py`)
- Edit Classifier → Character Similarity: uses similarity (evidence: `evaluation/utils/modules/classifier.py`)
