How lean-96/sos-mad works
Project Type & Purpose
• Type: Python-based CLI pipeline for research experiments (no UI).
• Purpose: Implement and evaluate a progressive scaling multi-agent debate system on reasoning tasks, explore upscaling vs. downscaling, and perform post-hoc analysis.
Key Architectural Components
Break the system into six high-level components:
A. Data Layer
• BIG-Bench-Hard dataset (data/bbh)
• Manifests (data/manifests/*.json)
B. Configuration
• .env files (DEBATER_A_…, DEBATER_B_…, JUDGE_…)
• src/config.py & scripts/setup.py
C. Orchestration Scripts (CLI entry-points)
• Phase 0: scripts/01_Setup_&_Dataset_Loading
• download_bbh.py → populates data/bbh
• setup.py → installs deps & checks config
• Phase 1: scripts/02_Data_Collection/run_single_agent.py
• Phase 2: scripts/02_Data_Collection/run_debate.py
• Phase 3: scripts/03_Analysis/run_sos_mad_analysis.py
D. Core Library (src/)
• Orchestration & Processing
• src/debate_processor.py → debate rounds logic
• src/data.py → loading tasks & manifests
• src/models.py → model management (load, switch)
• src/prompts.py → prompt templates & format standardization
• Providers Abstraction
• src/providers/ollama.py (local calls)
• src/providers/openrouter.py (cloud calls)
• Analysis Modules (src/analysis/)
• agreement_metrics.py, evolution_metrics.py, performance_metrics.py, visualization.py, etc.
E. Results Storage
• results/single_agent/
• model_responses/*.json
• model_results.csv, model_results_verified.csv
• results/multi-agent/
• debate_history/{upscale, downscale}/*.json
• debate_results_{upscale,downscale}_*.csv
F. External Services
• Ollama (local LLMs)
• OpenRouter (cloud LLMs)
Data Flow & Interactions
1. Setup Phase
• User runs setup.py → installs libs, reads .env → verifies model availability.
• download_bbh.py → fetches dataset → data/bbh.
2. Manifest Generation
• generate_manifest.py (02_Data_Collection) → reads dataset via src/data.py → writes data/manifests/*.json.
3. Single-Agent Evaluation
• run_single_agent.py → loads manifest → for each task/example:
– src/debate_processor or dedicated single-agent logic calls src/models + src/prompts → loops over providers → gets JSON responses → writes to results/single_agent/model_responses/*.json → aggregates CSV.
4. Multi-Agent Debate
• run_debate.py → loads manifest + scaling direction → for each task:
– Round 1/2/3: src/debate_processor spins up two debaters (A & B) with size progression, uses src/providers to call models → captures transcripts → writes detailed JSON to results/multi-agent/debate_history/{upscale,downscale}/
– Judges (3 models) called sequentially via providers → evaluations appended → final CSV aggregated in results/multi-agent.
5. Analysis
• run_sos_mad_analysis.py → reads both single_agent & multi-agent CSV/JSON → uses src/analysis modules → computes RQ1–RQ5 metrics → generates charts (PNG/CSV/MD) in results/analysis_output/.
Architectural Patterns & Principles
• Pipeline-oriented CLI design: clear phases, sequential execution.
• Dependency injection via .env and src/config.py.
• Provider abstraction: swap Ollama/OpenRouter without code changes.
• Modular analysis: each metric lives in its own module.
• Data persistence in JSON/CSV for reproducibility.
• Setup → Data
• Manifest Generation → Single-Agent & Debate
• Single-Agent → Results
• Debate → Detailed History → Judges → Results
• Analysis → Final Reports