How fudandisc/j1bench works
J1-Bench is an interactive legal-agent benchmark. Its runtime selects a scenario, constructs scenario-specific participants from case data, and runs legal dialogues; model engines provide language-model responses. The README also documents J1-Eval as a separate holistic evaluation capability, but its implementation is not present in the supplied tree. Scenario excerpts show consultation and dialogue logging; other scenario implementations and most agent modules were not inspected, so their internal call paths are not inferred.
Main entry points: run.py Scenario Runner · KQ.py Legal Consultation · base_agent.py Agent Memory · LC.py Legal Consultation LC
Components 27
Runtime Selection
- Scenario Runner
src/run.py - Runtime Options
src/utils/options.py - Class Registry
src/utils/register.py - Case Utilities
src/utils/utils_func.py
Legal Environments
- Civil Dialogue
src/scenario/CD.py - Civil Prediction
src/scenario/CI.py - Criminal Dialogue
src/scenario/CR.py - Document Generation
src/scenario/DD.py - Legal Consultation
src/scenario/KQ.py - Legal Consultation LC
src/scenario/LC.py
Legal Participants
- Agent Memory
src/agents/base_agent.py - Public Agents
src/agents/general_public_KQ.py - KQ Trainee
src/agents/trainee_KQ.py - LC Trainee
src/agents/trainee_LC.py - Civil Judge
src/agents/judge_CI.py - Criminal Judge
src/agents/judge_CR.py - Civil Plaintiff
src/agents/plaintiff_CI.py - Civil Defendant
src/agents/defendant_CI.py - Civil Lawyer
src/agents/lawyer_CD.py - Document Lawyer
src/agents/lawyer_DD.py - Scenario Characters
src/agents/specific_character_CD.py - Other Legal Roles
src/agents
Model Engines
- Engine Interface
src/engine/base_engine.py - Model Adapters
src/engine
Benchmark Evaluation
- J1-Eval
- J1-Eval Dataset
Other components
- Benchmark User
Connections 29
- Benchmark User → Scenario Runner: runs
README.md - Scenario Runner → Runtime Options: parses options
src/run.py - Scenario Runner → Class Registry: selects scenario
src/run.py - Scenario Runner → Civil Dialogue: runs selected
- Scenario Runner → Civil Prediction: runs selected
- Scenario Runner → Criminal Dialogue: runs selected
- Scenario Runner → Document Generation: runs selected
- Scenario Runner → Legal Consultation: runs selected
- Scenario Runner → Legal Consultation LC: runs selected
- Runtime Options → Class Registry: resolves classes
src/utils/options.py - Runtime Options → Legal Consultation: configures scenario
src/utils/options.py - Legal Consultation → Case Utilities: loads cases
src/scenario/KQ.py - Legal Consultation LC → Case Utilities: loads cases
src/scenario/LC.py - Legal Consultation LC → Case Utilities: requests completion
src/scenario/LC.py - Legal Consultation → Public Agents: constructs participants
- Legal Consultation → KQ Trainee: constructs participants
- Legal Consultation LC → Public Agents: constructs participants
- Legal Consultation LC → LC Trainee: constructs participants
- Document Lawyer → Class Registry: selects engine
src/agents/lawyer_DD.py - Civil Judge → Agent Memory: extends
src/agents/judge_CI.py - Criminal Judge → Agent Memory: extends
src/agents/judge_CR.py - Public Agents → Agent Memory: extends
src/agents/general_public_KQ.py - KQ Trainee → Agent Memory: extends
src/agents/trainee_KQ.py - LC Trainee → Agent Memory: extends
src/agents/trainee_LC.py - Engine Interface → Class Registry: registers interface
src/engine/base_engine.py - Agent Memory → Engine Interface: uses engine
- Model Adapters → Engine Interface: implements interface
src/engine/lawllm.py - Legal Consultation → J1-Eval: followed by evaluation
README.md - J1-Eval Dataset → J1-Eval: evaluated by
README.md