# machine-theory/lm-council architecture

> Architecture diagram of the GitHub repository machine-theory/lm-council, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/machine-theory/lm-council
- Repository: https://github.com/machine-theory/lm-council

## Overview

Language Model Council is a Python library for asking a group of language models to evaluate one another and aggregating their judgments. Its main workflow configures an evaluation, obtains model completions, gathers rubric or pairwise judgments, and presents agreement, rankings, and leaderboard results; runs can also be saved and loaded. The repository additionally includes research-analysis modules for bias, consistency, affinity, and separability. The graph reflects documented behavior and source-index references; unsampled implementation details and connections are omitted where their wiring is unclear.

Main entry points: [`lm_council/council.py`](https://github.com/machine-theory/lm-council/blob/main/lm_council/council.py) (Council Runner), [`lm_council/analysis/pairwise/agreement.py`](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/agreement.py) (Pairwise Agreement), [`lm_council/analysis/pairwise/affinity.py`](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/affinity.py) (Pairwise Affinity), [`lm_council/analysis/pairwise/separability.py`](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/separability.py) (Rankings & Separability)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_eval["Council Evaluation"]
  node_council["Council Runner<br/>[council.py]"]
end

subgraph group_judge["Judging Setup"]
  node_judgeconfig["Evaluation Config<br/>[config.py]"]
  node_promptbuilder["Prompt Builder<br/>[prompt_builder.py]"]
  node_presets["Preset Configs<br/>[__init__.py]"]
  node_prompts["Prompt Templates<br/>[prompts.py]"]
  node_structured["Structured Outputs"]
end

subgraph group_analysis["Research Analysis"]
  node_affinity["Pairwise Affinity<br/>[affinity.py]"]
  node_agreement["Pairwise Agreement<br/>[agreement.py]"]
  node_bradley["Bradley-Terry<br/>[bradley_terry.py]"]
  node_consistency["Vote Consistency<br/>[consistency.py]"]
  node_winrate["Explicit Win Rates"]
  node_lengthbias["Length Bias<br/>[length_bias.py]"]
  node_separability["Rankings &amp; Separability<br/>[separability.py]"]
  node_rubricaffinity["Rubric Affinity<br/>[affinity.py]"]
  node_rubricagreement["Rubric Agreement<br/>[agreement.py]"]
end

subgraph group_support["Shared Analysis"]
  node_pairutils["Pairwise Utilities<br/>[pairwise_utils.py]"]
  node_visualization["Visualization<br/>[visualization.py]"]
  node_constants["Analysis Constants<br/>[constants.py]"]
end

node_user(("Library User"))
node_models(("Language Models"))
node_openrouter["OpenRouter"]

node_user -->|"runs evaluation"| node_council
node_council -->|"requests completions"| node_models
node_council -->|"queries models"| node_openrouter
node_council -->|"uses config"| node_judgeconfig
node_council -->|"builds prompts"| node_promptbuilder
node_council -->|"requests structured output"| node_structured
node_council -->|"computes affinity"| node_affinity
node_council -->|"computes agreement"| node_agreement
node_council -->|"computes rankings"| node_bradley
node_council -->|"computes win rates"| node_winrate
node_council -->|"computes separability"| node_separability
node_council -->|"computes affinity"| node_rubricaffinity
node_council -->|"computes agreement"| node_rubricagreement
node_council -->|"plots results"| node_visualization
node_presets -->|"loads configs"| node_judgeconfig
node_judgeconfig -->|"uses template"| node_promptbuilder
node_affinity -->|"uses utilities"| node_pairutils
node_affinity -->|"uses rankings"| node_separability
node_affinity -->|"plots results"| node_visualization
node_agreement -->|"uses utilities"| node_pairutils
node_agreement -->|"sorts results"| node_visualization
node_agreement -->|"uses vote labels"| node_constants
node_consistency -->|"uses vote sides"| node_agreement
node_consistency -->|"aggregates votes"| node_pairutils
node_separability -->|"uses vote sides"| node_agreement
node_separability -->|"aggregates votes"| node_pairutils
node_rubricagreement -->|"plots results"| node_visualization

click node_council "https://github.com/machine-theory/lm-council/blob/main/lm_council/council.py"
click node_judgeconfig "https://github.com/machine-theory/lm-council/blob/main/lm_council/judging/config.py"
click node_promptbuilder "https://github.com/machine-theory/lm-council/blob/main/lm_council/judging/prompt_builder.py"
click node_presets "https://github.com/machine-theory/lm-council/blob/main/lm_council/judging/__init__.py"
click node_prompts "https://github.com/machine-theory/lm-council/blob/main/lm_council/prompts.py"
click node_structured "https://github.com/machine-theory/lm-council/blob/main/lm_council/structured_outputs.py"
click node_affinity "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/affinity.py"
click node_agreement "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/agreement.py"
click node_bradley "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/bradley_terry.py"
click node_consistency "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/consistency.py"
click node_winrate "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/explicit_win_rate.py"
click node_lengthbias "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/length_bias.py"
click node_separability "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/separability.py"
click node_rubricaffinity "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/rubric/affinity.py"
click node_rubricagreement "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/rubric/agreement.py"
click node_pairutils "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/pairwise_utils.py"
click node_visualization "https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/visualization.py"
click node_constants "https://github.com/machine-theory/lm-council/blob/main/lm_council/constants.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_council,node_user toneBlue
class node_judgeconfig,node_promptbuilder,node_presets,node_prompts,node_structured toneAmber
class node_affinity,node_agreement,node_bradley,node_consistency,node_winrate,node_lengthbias,node_separability,node_rubricaffinity,node_rubricagreement,node_openrouter toneMint
class node_pairutils,node_visualization,node_constants toneRose
class node_models toneIndigo
```

## Components

### Council Evaluation

- [Council Runner](https://github.com/machine-theory/lm-council/blob/main/lm_council/council.py): `lm_council/council.py`

### Judging Setup

- [Evaluation Config](https://github.com/machine-theory/lm-council/blob/main/lm_council/judging/config.py): `lm_council/judging/config.py`
- [Prompt Builder](https://github.com/machine-theory/lm-council/blob/main/lm_council/judging/prompt_builder.py): `lm_council/judging/prompt_builder.py`
- [Preset Configs](https://github.com/machine-theory/lm-council/blob/main/lm_council/judging/__init__.py): `lm_council/judging/__init__.py`
- [Prompt Templates](https://github.com/machine-theory/lm-council/blob/main/lm_council/prompts.py): `lm_council/prompts.py`
- [Structured Outputs](https://github.com/machine-theory/lm-council/blob/main/lm_council/structured_outputs.py): `lm_council/structured_outputs.py`

### Research Analysis

- [Pairwise Affinity](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/affinity.py): `lm_council/analysis/pairwise/affinity.py`
- [Pairwise Agreement](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/agreement.py): `lm_council/analysis/pairwise/agreement.py`
- [Bradley-Terry](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/bradley_terry.py): `lm_council/analysis/pairwise/bradley_terry.py`
- [Vote Consistency](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/consistency.py): `lm_council/analysis/pairwise/consistency.py`
- [Explicit Win Rates](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/explicit_win_rate.py): `lm_council/analysis/pairwise/explicit_win_rate.py`
- [Length Bias](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/length_bias.py): `lm_council/analysis/pairwise/length_bias.py`
- [Rankings & Separability](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/separability.py): `lm_council/analysis/pairwise/separability.py`
- [Rubric Affinity](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/rubric/affinity.py): `lm_council/analysis/rubric/affinity.py`
- [Rubric Agreement](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/rubric/agreement.py): `lm_council/analysis/rubric/agreement.py`

### Shared Analysis

- [Pairwise Utilities](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/pairwise/pairwise_utils.py): `lm_council/analysis/pairwise/pairwise_utils.py`
- [Visualization](https://github.com/machine-theory/lm-council/blob/main/lm_council/analysis/visualization.py): `lm_council/analysis/visualization.py`
- [Analysis Constants](https://github.com/machine-theory/lm-council/blob/main/lm_council/constants.py): `lm_council/constants.py`

### Other components

- **Library User**
- **Language Models**
- **OpenRouter**

## Connections

- Library User → Council Runner: runs evaluation (evidence: `README.md`)
- Council Runner → Language Models: requests completions (evidence: `README.md`)
- Council Runner → OpenRouter: queries models (evidence: `README.md`)
- Council Runner → Evaluation Config: uses config (evidence: `lm_council/council.py`)
- Council Runner → Prompt Builder: builds prompts (evidence: `lm_council/council.py`)
- Council Runner → Structured Outputs: requests structured output (evidence: `lm_council/council.py`)
- Council Runner → Pairwise Affinity: computes affinity (evidence: `lm_council/council.py`)
- Council Runner → Pairwise Agreement: computes agreement (evidence: `lm_council/council.py`)
- Council Runner → Bradley-Terry: computes rankings (evidence: `lm_council/council.py`)
- Council Runner → Explicit Win Rates: computes win rates (evidence: `lm_council/council.py`)
- Council Runner → Rankings & Separability: computes separability (evidence: `lm_council/council.py`)
- Council Runner → Rubric Affinity: computes affinity (evidence: `lm_council/council.py`)
- Council Runner → Rubric Agreement: computes agreement (evidence: `lm_council/council.py`)
- Council Runner → Visualization: plots results (evidence: `lm_council/council.py`)
- Preset Configs → Evaluation Config: loads configs (evidence: `lm_council/judging/__init__.py`)
- Evaluation Config → Prompt Builder: uses template (evidence: `lm_council/judging/config.py`)
- Pairwise Affinity → Pairwise Utilities: uses utilities (evidence: `lm_council/analysis/pairwise/affinity.py`)
- Pairwise Affinity → Rankings & Separability: uses rankings (evidence: `lm_council/analysis/pairwise/affinity.py`)
- Pairwise Affinity → Visualization: plots results (evidence: `lm_council/analysis/pairwise/affinity.py`)
- Pairwise Agreement → Pairwise Utilities: uses utilities (evidence: `lm_council/analysis/pairwise/agreement.py`)
- Pairwise Agreement → Visualization: sorts results (evidence: `lm_council/analysis/pairwise/agreement.py`)
- Pairwise Agreement → Analysis Constants: uses vote labels (evidence: `lm_council/analysis/pairwise/agreement.py`)
- Vote Consistency → Pairwise Agreement: uses vote sides (evidence: `lm_council/analysis/pairwise/consistency.py`)
- Vote Consistency → Pairwise Utilities: aggregates votes (evidence: `lm_council/analysis/pairwise/consistency.py`)
- Rankings & Separability → Pairwise Agreement: uses vote sides (evidence: `lm_council/analysis/pairwise/separability.py`)
- Rankings & Separability → Pairwise Utilities: aggregates votes (evidence: `lm_council/analysis/pairwise/separability.py`)
- Rubric Agreement → Visualization: plots results (evidence: `lm_council/analysis/rubric/agreement.py`)