How mayureshmohnalkar22/llm-output-arbitrator works
This is a local service for evaluating LLM outputs through three independent critics, routing disagreements or missing critiques to adjudication, and returning an auditable verdict. The main user path runs from the explorer or API through validation, parallel criticism, disagreement detection, synthesis or adjudication, and persistent history. The README documents batch evaluation, analytics, and live providers; those capabilities are included. Source excerpts show API, workflow, provider, storage, and explorer wiring; other README-described behavior is represented without claiming unsampled implementation details.
Main entry points: workflow.py Workflow engine · main.py Arbitration API · explorer.py Verdict explorer · providers.py Provider routing
Components 17
User interfaces
- Verdict explorer
dashboard/explorer.py - Arbitration API
app/main.py - Evidence annotations
app/annotations.py
Arbitration workflow
- Request models
app/models.py - Workflow engine
app/workflow.py - Collect disagreements
app/workflow.py - Adjudicator
app/workflow.py - Direct synthesis
app/workflow.py - Runtime settings
app/config.py
Evaluation providers
- Accuracy critic
app/workflow.py - Logic critic
app/workflow.py - Completeness critic
app/workflow.py - Provider routing
app/providers.py
Verdicts and history
- Verdict and telemetry
app/models.py - SQLite audit store
app/storage.py
Other components
- User
- Live model services
Connections 27
- User → Verdict explorer: submits request
README.md - Verdict explorer → Arbitration API: calls API
dashboard/explorer.py - User → Arbitration API: submits request
README.md - Arbitration API → Workflow engine: runs arbitration
app/main.py - Arbitration API → Request models: validates input
app/main.py - Workflow engine → Request models: parses request
app/workflow.py - Workflow engine → Accuracy critic: dispatches
app/workflow.py - Workflow engine → Logic critic: dispatches
app/workflow.py - Workflow engine → Completeness critic: dispatches
app/workflow.py - Accuracy critic → Provider routing: evaluates via
app/workflow.py - Logic critic → Provider routing: evaluates via
app/workflow.py - Completeness critic → Provider routing: evaluates via
app/workflow.py - Provider routing → Live model services: calls live models
README.md - Workflow engine → Collect disagreements: collects critiques
app/workflow.py - Collect disagreements → Adjudicator: routes conflicts
app/workflow.py - Collect disagreements → Direct synthesis: routes agreement
app/workflow.py - Adjudicator → Provider routing: requests decision
app/workflow.py - Adjudicator → Verdict and telemetry: produces verdict
app/workflow.py - Direct synthesis → Verdict and telemetry: produces verdict
app/workflow.py - Workflow engine → Runtime settings: uses settings
app/workflow.py - Workflow engine → Verdict and telemetry: returns result
app/workflow.py - Arbitration API → SQLite audit store: saves verdict
app/main.py - Arbitration API → SQLite audit store: reads history and analytics
app/main.py - SQLite audit store → Verdict and telemetry: stores verdict
app/storage.py - Arbitration API → Verdict explorer: serves API responses
README.md - Verdict explorer → Evidence annotations: renders evidence
dashboard/explorer.py - Verdict explorer → Verdict and telemetry: displays verdict
dashboard/explorer.py