# aptus-ai/legalita architecture

> Architecture diagram of the GitHub repository aptus-ai/legalita, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aptus-ai/legalita
- Repository: https://github.com/aptus-ai/legalita

## Overview

LegalITA is a benchmark for Italian legal-reasoning language models. Its main workflow loads separately distributed tasks, queries evaluated models, judges criterion-level answers, and produces scores and reports. A separate adversarial benchmark checks whether models detect missing documents. Citation evaluation has two related paths: criterion judges can receive precomputed grounding context, while local citation services extract and resolve citations against a supplied registry and profiles. The README documents these capabilities; some runner internals and their wiring are not included in the sampled source.

Main entry points: [`run_benchmark.py`](https://github.com/aptus-ai/legalita/blob/main/run_benchmark.py) (Reasoning runner), [`legal_ita/cli/bullshit.py`](https://github.com/aptus-ai/legalita/blob/main/legal_ita/cli/bullshit.py) (Missing-doc runner), [`legal_ita/grounding/service.py`](https://github.com/aptus-ai/legalita/blob/main/legal_ita/grounding/service.py) (Grounding pipeline), [`evaluation/citations/service.py`](https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/service.py) (Citation existence)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_benchmark["Benchmark runs"]
  node_benchmark_runner["Reasoning runner<br/>[run_benchmark.py]"]
  node_bullshit_runner["Missing-doc runner<br/>[bullshit.py]"]
  node_model_query["Model adapters<br/>[query.py]"]
end

subgraph group_evaluation["Judging and scoring"]
  node_legal_scoring["Legal scoring<br/>[service.py]"]
  node_legal_judges["Criterion judges<br/>[judge.py]"]
  node_bullshit_judging["False-premise judge<br/>[bullshit_judge.py]"]
end

subgraph group_citations["Citation grounding"]
  node_citation_service["Citation existence<br/>[service.py]"]
  node_citation_parser["Citation parser<br/>[parser.py]"]
  node_citation_extractor["LLM extractor<br/>[llm_extractor.py]"]
  node_local_registry[("Local ECLI registry<br/>[local_registry.py]")]
  node_local_resolver["Local resolver<br/>[local_resolver.py]"]
  node_grounding_runner["Grounding pipeline<br/>[service.py]"]
end

subgraph group_reporting["Results and reports"]
  node_reporting["Report generation<br/>[cli.py]"]
end

node_user(("Benchmark user"))
node_tasks["Task bundle"]
node_model_providers{{"Model APIs"}}
node_grounding_bundle["Profiles and registry"]
node_result_files["Run results"]
node_judge_apis{{"Judge APIs"}}

node_user -->|"runs benchmark"| node_benchmark_runner
node_user -->|"runs adversarial"| node_bullshit_runner
node_user -->|"runs grounding"| node_grounding_runner
node_user -->|"requests reports"| node_reporting
node_tasks -->|"provides tasks"| node_benchmark_runner
node_tasks -->|"provides tasks"| node_bullshit_runner
node_benchmark_runner -->|"queries models"| node_model_query
node_bullshit_runner -->|"queries models"| node_model_query
node_model_query -->|"sends prompts"| node_model_providers
node_benchmark_runner -->|"scores answers"| node_legal_scoring
node_legal_scoring -->|"judges criteria"| node_legal_judges
node_legal_scoring -->|"checks citations"| node_citation_service
node_legal_judges -->|"requests verdicts"| node_judge_apis
node_bullshit_runner -->|"scores answers"| node_bullshit_judging
node_bullshit_judging -->|"requests verdicts"| node_judge_apis
node_citation_service -->|"parses citations"| node_citation_parser
node_citation_service -->|"extracts citations"| node_citation_extractor
node_citation_service -->|"resolves citations"| node_local_resolver
node_local_resolver -->|"looks up ECLI"| node_local_registry
node_grounding_bundle -->|"supplies profiles"| node_grounding_runner
node_grounding_runner -->|"extracts citations"| node_citation_extractor
node_grounding_runner -->|"reads registry"| node_local_registry
node_benchmark_runner -->|"writes results"| node_result_files
node_bullshit_runner -->|"writes results"| node_result_files
node_grounding_runner -->|"writes reports"| node_result_files
node_result_files -->|"loads results"| node_reporting

click node_benchmark_runner "https://github.com/aptus-ai/legalita/blob/main/run_benchmark.py"
click node_bullshit_runner "https://github.com/aptus-ai/legalita/blob/main/legal_ita/cli/bullshit.py"
click node_model_query "https://github.com/aptus-ai/legalita/blob/main/legal_ita/modeling/query.py"
click node_legal_scoring "https://github.com/aptus-ai/legalita/blob/main/evaluation/scoring/service.py"
click node_legal_judges "https://github.com/aptus-ai/legalita/blob/main/evaluation/judge.py"
click node_bullshit_judging "https://github.com/aptus-ai/legalita/blob/main/evaluation/bullshit_judge.py"
click node_citation_service "https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/service.py"
click node_citation_parser "https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/parser.py"
click node_citation_extractor "https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/llm_extractor.py"
click node_local_registry "https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/local_registry.py"
click node_local_resolver "https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/local_resolver.py"
click node_grounding_runner "https://github.com/aptus-ai/legalita/blob/main/legal_ita/grounding/service.py"
click node_reporting "https://github.com/aptus-ai/legalita/blob/main/evaluation/reporting/cli.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_benchmark_runner,node_bullshit_runner,node_model_query,node_user toneBlue
class node_legal_scoring,node_legal_judges,node_bullshit_judging toneAmber
class node_citation_service,node_citation_parser,node_citation_extractor,node_local_registry,node_local_resolver,node_grounding_runner,node_model_providers,node_judge_apis toneMint
class node_reporting,node_tasks toneRose
class node_grounding_bundle,node_result_files toneIndigo
```

## Components

### Benchmark runs

- [Reasoning runner](https://github.com/aptus-ai/legalita/blob/main/run_benchmark.py): `run_benchmark.py`
- [Missing-doc runner](https://github.com/aptus-ai/legalita/blob/main/legal_ita/cli/bullshit.py): `legal_ita/cli/bullshit.py`
- [Model adapters](https://github.com/aptus-ai/legalita/blob/main/legal_ita/modeling/query.py): `legal_ita/modeling/query.py`

### Judging and scoring

- [Legal scoring](https://github.com/aptus-ai/legalita/blob/main/evaluation/scoring/service.py): `evaluation/scoring/service.py`
- [Criterion judges](https://github.com/aptus-ai/legalita/blob/main/evaluation/judge.py): `evaluation/judge.py`
- [False-premise judge](https://github.com/aptus-ai/legalita/blob/main/evaluation/bullshit_judge.py): `evaluation/bullshit_judge.py`

### Citation grounding

- [Citation existence](https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/service.py): `evaluation/citations/service.py`
- [Citation parser](https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/parser.py): `evaluation/citations/parser.py`
- [LLM extractor](https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/llm_extractor.py): `evaluation/citations/llm_extractor.py`
- [Local ECLI registry](https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/local_registry.py): `evaluation/citations/local_registry.py`
- [Local resolver](https://github.com/aptus-ai/legalita/blob/main/evaluation/citations/local_resolver.py): `evaluation/citations/local_resolver.py`
- [Grounding pipeline](https://github.com/aptus-ai/legalita/blob/main/legal_ita/grounding/service.py): `legal_ita/grounding/service.py`

### Results and reports

- [Report generation](https://github.com/aptus-ai/legalita/blob/main/evaluation/reporting/cli.py): `evaluation/reporting/cli.py`

### Other components

- **Benchmark user**
- **Task bundle**
- **Model APIs**
- **Profiles and registry**
- **Run results**
- **Judge APIs**

## Connections

- Benchmark user → Reasoning runner: runs benchmark
- Benchmark user → Missing-doc runner: runs adversarial
- Benchmark user → Grounding pipeline: runs grounding
- Benchmark user → Report generation: requests reports
- Task bundle → Reasoning runner: provides tasks
- Task bundle → Missing-doc runner: provides tasks
- Reasoning runner → Model adapters: queries models
- Missing-doc runner → Model adapters: queries models
- Model adapters → Model APIs: sends prompts
- Reasoning runner → Legal scoring: scores answers
- Legal scoring → Criterion judges: judges criteria
- Legal scoring → Citation existence: checks citations
- Criterion judges → Judge APIs: requests verdicts
- Missing-doc runner → False-premise judge: scores answers
- False-premise judge → Judge APIs: requests verdicts
- Citation existence → Citation parser: parses citations
- Citation existence → LLM extractor: extracts citations
- Citation existence → Local resolver: resolves citations
- Local resolver → Local ECLI registry: looks up ECLI
- Profiles and registry → Grounding pipeline: supplies profiles
- Grounding pipeline → LLM extractor: extracts citations
- Grounding pipeline → Local ECLI registry: reads registry
- Reasoning runner → Run results: writes results
- Missing-doc runner → Run results: writes results
- Grounding pipeline → Run results: writes reports
- Run results → Report generation: loads results