How machine-theory/lm-council works
Language Model Council is a Python library for asking a group of language models to evaluate one another and aggregating their judgments. Its main workflow configures an evaluation, obtains model completions, gathers rubric or pairwise judgments, and presents agreement, rankings, and leaderboard results; runs can also be saved and loaded. The repository additionally includes research-analysis modules for bias, consistency, affinity, and separability. The graph reflects documented behavior and source-index references; unsampled implementation details and connections are omitted where their wiring is unclear.
Main entry points: council.py Council Runner · agreement.py Pairwise Agreement · affinity.py Pairwise Affinity · separability.py Rankings & Separability
Components 21
Council Evaluation
- Council Runner
lm_council/council.py
Judging Setup
- Evaluation Config
lm_council/judging/config.py - Prompt Builder
lm_council/judging/prompt_builder.py - Preset Configs
lm_council/judging/__init__.py - Prompt Templates
lm_council/prompts.py - Structured Outputs
lm_council/structured_outputs.py
Research Analysis
- Pairwise Affinity
lm_council/analysis/pairwise/affinity.py - Pairwise Agreement
lm_council/analysis/pairwise/agreement.py - Bradley-Terry
lm_council/analysis/pairwise/bradley_terry.py - Vote Consistency
lm_council/analysis/pairwise/consistency.py - Explicit Win Rates
lm_council/analysis/pairwise/explicit_win_rate.py - Length Bias
lm_council/analysis/pairwise/length_bias.py - Rankings & Separability
lm_council/analysis/pairwise/separability.py - Rubric Affinity
lm_council/analysis/rubric/affinity.py - Rubric Agreement
lm_council/analysis/rubric/agreement.py
Shared Analysis
- Pairwise Utilities
lm_council/analysis/pairwise/pairwise_utils.py - Visualization
lm_council/analysis/visualization.py - Analysis Constants
lm_council/constants.py
Other components
- Library User
- Language Models
- OpenRouter
Connections 27
- Library User → Council Runner: runs evaluation
README.md - Council Runner → Language Models: requests completions
README.md - Council Runner → OpenRouter: queries models
README.md - Council Runner → Evaluation Config: uses config
lm_council/council.py - Council Runner → Prompt Builder: builds prompts
lm_council/council.py - Council Runner → Structured Outputs: requests structured output
lm_council/council.py - Council Runner → Pairwise Affinity: computes affinity
lm_council/council.py - Council Runner → Pairwise Agreement: computes agreement
lm_council/council.py - Council Runner → Bradley-Terry: computes rankings
lm_council/council.py - Council Runner → Explicit Win Rates: computes win rates
lm_council/council.py - Council Runner → Rankings & Separability: computes separability
lm_council/council.py - Council Runner → Rubric Affinity: computes affinity
lm_council/council.py - Council Runner → Rubric Agreement: computes agreement
lm_council/council.py - Council Runner → Visualization: plots results
lm_council/council.py - Preset Configs → Evaluation Config: loads configs
lm_council/judging/__init__.py - Evaluation Config → Prompt Builder: uses template
lm_council/judging/config.py - Pairwise Affinity → Pairwise Utilities: uses utilities
lm_council/analysis/pairwise/affinity.py - Pairwise Affinity → Rankings & Separability: uses rankings
lm_council/analysis/pairwise/affinity.py - Pairwise Affinity → Visualization: plots results
lm_council/analysis/pairwise/affinity.py - Pairwise Agreement → Pairwise Utilities: uses utilities
lm_council/analysis/pairwise/agreement.py - Pairwise Agreement → Visualization: sorts results
lm_council/analysis/pairwise/agreement.py - Pairwise Agreement → Analysis Constants: uses vote labels
lm_council/analysis/pairwise/agreement.py - Vote Consistency → Pairwise Agreement: uses vote sides
lm_council/analysis/pairwise/consistency.py - Vote Consistency → Pairwise Utilities: aggregates votes
lm_council/analysis/pairwise/consistency.py - Rankings & Separability → Pairwise Agreement: uses vote sides
lm_council/analysis/pairwise/separability.py - Rankings & Separability → Pairwise Utilities: aggregates votes
lm_council/analysis/pairwise/separability.py - Rubric Agreement → Visualization: plots results
lm_council/analysis/rubric/agreement.py