How dinobby/reconcile works
Project Type and Purpose
• This is a Python-based research framework (no GUI/frontend) for multi-round discussion and consensus answering across diverse LLMs (OpenAI/GPT, PaLM, Claude).
• It takes benchmark question sets, runs each question through a round-table process among the LLMs, aggregates a consensus answer, and saves both raw and consensus answers for later analysis.
File Structure Analysis
Top level:
• run.py – CLI entry point that orchestrates the experiment.
• generation.py – Core “round-table” discussion logic and API orchestration.
• data_utils.py – Loading/parsing datasets.
• utils.py – Utility helpers (e.g., consensus scoring, JSON read/write).
• claude.py – Wrapper around Claude-API authentication & request logic.
• requirements.txt/.env – Dependencies and credential placeholders.
• convincing/ – Output storage: per-dataset, per-LLM JSON results and consensus files.
• dataset/ – Input test/development data for each benchmark.
Key README Insights
• Uses Python 3.10, external LLM services (OpenAI, PaLM, Claude via Claude-API).
• Expects an .env with API keys/cookies.
• CLI flags: --num_samples and --dataset to choose subset and benchmark.
a. Main Components
1. CLI Runner (run.py)
2. Dataset Loader (data_utils.py)
3. Discussion Coordinator (generation.py)
4. LLM Service Clients
• OpenAI Client (via openai library)
• PaLM Client (via Google PaLM API)
• Claude Client (claude.py + Claude-API wrapper)
5. Consensus & Scoring Utils (utils.py)
6. Results Store (convincing/ folder)
b. Relationships & Data Flow
1. User invokes run.py → parses arguments
2. run.py calls data_utils to load questions
3. For each question, run.py invokes generation.py
4. generation.py loops through discussion rounds:
– Sends prompts to each LLM client → receives individual responses
– Applies utilities to compare, score, and vote
– Builds a consensus answer after X rounds
5. generation.py calls utils to serialize both raw and consensus outputs to JSON under convincing/<Dataset>/
6. run.py optionally aggregates final metrics
c. Architectural Patterns & Principles
– Pipeline/Scripting Architecture: clear sequential stages (load → discuss → aggregate → store)
– Modularity & Single Responsibility: each .py handles a dedicated concern
– Adapter Pattern: separate client wrappers for each external LLM API
– Pluggability: adding another model requires minimal changes (new client + config)
d. Key Technologies & Libraries
– Python 3.10
– openai (OpenAI API)
– google-generativeai (PaLM)
– Claude-API wrapper (claude.py)
– pandas or built-in json/csv modules for data handling