How aptus-ai/legalita works
LegalITA is a benchmark for Italian legal-reasoning language models. Its main workflow loads separately distributed tasks, queries evaluated models, judges criterion-level answers, and produces scores and reports. A separate adversarial benchmark checks whether models detect missing documents. Citation evaluation has two related paths: criterion judges can receive precomputed grounding context, while local citation services extract and resolve citations against a supplied registry and profiles. The README documents these capabilities; some runner internals and their wiring are not included in the sampled source.
Main entry points: run_benchmark.py Reasoning runner · bullshit.py Missing-doc runner · grounding/service.py Grounding pipeline · citations/service.py Citation existence
Components 19
Benchmark runs
- Reasoning runner
run_benchmark.py - Missing-doc runner
legal_ita/cli/bullshit.py - Model adapters
legal_ita/modeling/query.py
Judging and scoring
- Legal scoring
evaluation/scoring/service.py - Criterion judges
evaluation/judge.py - False-premise judge
evaluation/bullshit_judge.py
Citation grounding
- Citation existence
evaluation/citations/service.py - Citation parser
evaluation/citations/parser.py - LLM extractor
evaluation/citations/llm_extractor.py - Local ECLI registry
evaluation/citations/local_registry.py - Local resolver
evaluation/citations/local_resolver.py - Grounding pipeline
legal_ita/grounding/service.py
Results and reports
- Report generation
evaluation/reporting/cli.py
Other components
- Benchmark user
- Task bundle
- Model APIs
- Profiles and registry
- Run results
- Judge APIs
Connections 26
- Benchmark user → Reasoning runner: runs benchmark
- Benchmark user → Missing-doc runner: runs adversarial
- Benchmark user → Grounding pipeline: runs grounding
- Benchmark user → Report generation: requests reports
- Task bundle → Reasoning runner: provides tasks
- Task bundle → Missing-doc runner: provides tasks
- Reasoning runner → Model adapters: queries models
- Missing-doc runner → Model adapters: queries models
- Model adapters → Model APIs: sends prompts
- Reasoning runner → Legal scoring: scores answers
- Legal scoring → Criterion judges: judges criteria
- Legal scoring → Citation existence: checks citations
- Criterion judges → Judge APIs: requests verdicts
- Missing-doc runner → False-premise judge: scores answers
- False-premise judge → Judge APIs: requests verdicts
- Citation existence → Citation parser: parses citations
- Citation existence → LLM extractor: extracts citations
- Citation existence → Local resolver: resolves citations
- Local resolver → Local ECLI registry: looks up ECLI
- Profiles and registry → Grounding pipeline: supplies profiles
- Grounding pipeline → LLM extractor: extracts citations
- Grounding pipeline → Local ECLI registry: reads registry
- Reasoning runner → Run results: writes results
- Missing-doc runner → Run results: writes results
- Grounding pipeline → Run results: writes reports
- Run results → Report generation: loads results