# lean-96/sos-mad architecture

> Architecture diagram of the GitHub repository lean-96/sos-mad, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lean-96/sos-mad
- Repository: https://github.com/lean-96/sos-mad

## Overview

##### Project Type & Purpose
- Type: Python-based CLI pipeline for research experiments (no UI).
- Purpose: Implement and evaluate a progressive scaling multi-agent debate system on reasoning tasks, explore upscaling vs. downscaling, and perform post-hoc analysis.

##### Key Architectural Components
Break the system into six high-level components:
A. Data Layer
- BIG-Bench-Hard dataset (data/bbh)
- Manifests (data/manifests/*.json)
B. Configuration
- .env files (DEBATER_A_…, DEBATER_B_…, JUDGE_…)
- src/config.py & scripts/setup.py
C. Orchestration Scripts (CLI entry-points)
- Phase 0: scripts/01_Setup_&_Dataset_Loading
- download_bbh.py → populates data/bbh
- setup.py → installs deps & checks config
- Phase 1: scripts/02_Data_Collection/run_single_agent.py
- Phase 2: scripts/02_Data_Collection/run_debate.py
- Phase 3: scripts/03_Analysis/run_sos_mad_analysis.py
D. Core Library (src/)
- Orchestration & Processing
- src/debate_processor.py → debate rounds logic
- src/data.py → loading tasks & manifests
- src/models.py → model management (load, switch)
- src/prompts.py → prompt templates & format standardization
- Providers Abstraction
- src/providers/ollama.py (local calls)
- src/providers/openrouter.py (cloud calls)
- Analysis Modules (src/analysis/)
- agreement_metrics.py, evolution_metrics.py, performance_metrics.py, visualization.py, etc.
E. Results Storage
- results/single_agent/
- model_responses/*.json
- model_results.csv, model_results_verified.csv
- results/multi-agent/
- debate_history/{upscale, downscale}/*.json
- debate_results_{upscale,downscale}_*.csv
F. External Services
- Ollama (local LLMs)
- OpenRouter (cloud LLMs)

##### Data Flow & Interactions
1. **Setup Phase**
- User runs setup.py → installs libs, reads .env → verifies model availability.
- download_bbh.py → fetches dataset → data/bbh.
2. **Manifest Generation**
- generate_manifest.py (02_Data_Collection) → reads dataset via src/data.py → writes data/manifests/*.json.
3. **Single-Agent Evaluation**
- run_single_agent.py → loads manifest → for each task/example:
– src/debate_processor or dedicated single-agent logic calls src/models + src/prompts → loops over providers → gets JSON responses → writes to results/single_agent/model_responses/*.json → aggregates CSV.
4. **Multi-Agent Debate**
- run_debate.py → loads manifest + scaling direction → for each task:
– Round 1/2/3: src/debate_processor spins up two debaters (A & B) with size progression, uses src/providers to call models → captures transcripts → writes detailed JSON to results/multi-agent/debate_history/{upscale,downscale}/
– Judges (3 models) called sequentially via providers → evaluations appended → final CSV aggregated in results/multi-agent.
5. **Analysis**
- run_sos_mad_analysis.py → reads both single_agent & multi-agent CSV/JSON → uses src/analysis modules → computes RQ1–RQ5 metrics → generates charts (PNG/CSV/MD) in results/analysis_output/.

##### Architectural Patterns & Principles
- Pipeline-oriented CLI design: clear phases, sequential execution.
- Dependency injection via .env and src/config.py.
- Provider abstraction: swap Ollama/OpenRouter without code changes.
- Modular analysis: each metric lives in its own module.
- Data persistence in JSON/CSV for reproducibility.

- Setup → Data
- Manifest Generation → Single-Agent & Debate
- Single-Agent → Results
- Debate → Detailed History → Judges → Results
- Analysis → Final Reports

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data & Config" 
        style DataConfig fill:#cce5ff,stroke:#004085,stroke-width:2px
        DL1["Dataset Directory"]:::data
        click DL1 "https://github.com/lean-96/sos-mad/tree/main/data/bbh"
        ML["Manifests Directory"]:::data
        click ML "https://github.com/lean-96/sos-mad/tree/main/data/manifests/"
        Env[".env Files"]:::config
        click Env "https://github.com/lean-96/sos-mad/blob/main/.env.example"
        click Env "https://github.com/lean-96/sos-mad/blob/main/.env.downscale"
        CFG["src/config.py"]:::config
        click CFG "https://github.com/lean-96/sos-mad/blob/main/src/config.py"
        SETUP_S["setup.py"]:::config
        click SETUP_S "https://github.com/lean-96/sos-mad/blob/main/scripts/01_Setup_&_Dataset_Loading/setup.py"
        DOC["docs/configuration.md"]:::config
        click DOC "https://github.com/lean-96/sos-mad/blob/main/docs/configuration.md"
    end

    subgraph "Orchestration Scripts" 
        style Orchestration fill:#d4edda,stroke:#155724,stroke-width:2px
        S0b["Phase 0: setup.py"]:::orchestration
        click S0b "https://github.com/lean-96/sos-mad/blob/main/scripts/01_Setup_&_Dataset_Loading/setup.py"
        S0["Phase 0: download_bbh.py"]:::orchestration
        click S0 "https://github.com/lean-96/sos-mad/blob/main/scripts/01_Setup_&_Dataset_Loading/download_bbh.py"
        GM["Manifest Generation\ngenerate_manifest.py"]:::orchestration
        click GM "https://github.com/lean-96/sos-mad/blob/main/scripts/02_Data_Collection/generate_manifest.py"
        SA["Phase 1:\nrun_single_agent.py"]:::orchestration
        click SA "https://github.com/lean-96/sos-mad/blob/main/scripts/02_Data_Collection/run_single_agent.py"
        MA["Phase 2:\nrun_debate.py"]:::orchestration
        click MA "https://github.com/lean-96/sos-mad/blob/main/scripts/02_Data_Collection/run_debate.py"
        UTS["run_all_scaling_experiments_robust.sh"]:::orchestration
        click UTS "https://github.com/lean-96/sos-mad/blob/main/scripts/02_Data_Collection/run_all_scaling_experiments_robust.sh"
        AN["Phase 3:\nrun_sos_mad_analysis.py"]:::orchestration
        click AN "https://github.com/lean-96/sos-mad/blob/main/scripts/03_Analysis/run_sos_mad_analysis.py"
    end

    subgraph "Core Library" 
        style Core fill:#ffe5b4,stroke:#854d00,stroke-width:2px
        DP["debate_processor.py"]:::core
        click DP "https://github.com/lean-96/sos-mad/blob/main/src/debate_processor.py"
        DA["data.py"]:::core
        click DA "https://github.com/lean-96/sos-mad/blob/main/src/data.py"
        MO["models.py"]:::core
        click MO "https://github.com/lean-96/sos-mad/blob/main/src/models.py"
        PR["prompts.py"]:::core
        click PR "https://github.com/lean-96/sos-mad/blob/main/src/prompts.py"
        subgraph "Providers Abstraction"
            style Prov fill:#ffe5b4,stroke:#854d00,stroke-width:1px
            OLL["ollama.py"]:::core
            click OLL "https://github.com/lean-96/sos-mad/blob/main/src/providers/ollama.py"
            ORT["openrouter.py"]:::core
            click ORT "https://github.com/lean-96/sos-mad/blob/main/src/providers/openrouter.py"
        end
        subgraph "Analysis Modules"
            style AnaMod fill:#fff3cd,stroke:#856404,stroke-width:1px
            AM1["agreement_metrics.py"]:::analysis
            click AM1 "https://github.com/lean-96/sos-mad/blob/main/src/analysis/agreement_metrics.py"
            AM2["evolution_metrics.py"]:::analysis
            click AM2 "https://github.com/lean-96/sos-mad/blob/main/src/analysis/evolution_metrics.py"
            AM3["performance_metrics.py"]:::analysis
            click AM3 "https://github.com/lean-96/sos-mad/blob/main/src/analysis/performance_metrics.py"
            AM4["visualization.py"]:::analysis
            click AM4 "https://github.com/lean-96/sos-mad/blob/main/src/analysis/visualization.py"
            AM5["answer_extraction.py"]:::analysis
            click AM5 "https://github.com/lean-96/sos-mad/blob/main/src/analysis/answer_extraction.py"
            AM6["performance_bar_chart.py"]:::analysis
            click AM6 "https://github.com/lean-96/sos-mad/blob/main/src/analysis/performance_bar_chart.py"
            AM7["manifest.py"]:::analysis
            click AM7 "https://github.com/lean-96/sos-mad/blob/main/src/analysis/manifest.py"
            AM8["sos_mad_analysis.py"]:::analysis
            click AM8 "https://github.com/lean-96/sos-mad/blob/main/src/analysis/sos_mad_analysis.py"
            AM9["utils.py"]:::analysis
            click AM9 "https://github.com/lean-96/sos-mad/blob/main/src/analysis/utils.py"
            AMC["config.py"]:::analysis
            click AMC "https://github.com/lean-96/sos-mad/blob/main/src/analysis/config.py"
        end
    end

    subgraph "Results Storage" 
        style Storage fill:#fff3cd,stroke:#856404,stroke-width:2px
        RSA1[/"Single-Agent JSON"/]:::storage
        click RSA1 "https://github.com/lean-96/sos-mad/tree/main/results/single_agent/model_responses/"
        RSA2["model_results.csv"]:::storage
        click RSA2 "https://github.com/lean-96/sos-mad/blob/main/results/single_agent/model_results.csv"
        RSA3["model_results_verified.csv"]:::storage
        click RSA3 "https://github.com/lean-96/sos-mad/blob/main/results/single_agent/model_results_verified.csv"
        RMB1[/"Debate Histories Downscale"/]:::storage
        click RMB1 "https://github.com/lean-96/sos-mad/tree/main/results/multi-agent/debate_history/downscale/"
        RMB2[/"Debate Histories Upscale"/]:::storage
        click RMB2 "https://github.com/lean-96/sos-mad/tree/main/results/multi-agent/debate_history/upscale/"
        RMB3["debate_results_downscale_merged.csv"]:::storage
        click RMB3 "https://github.com/lean-96/sos-mad/blob/main/results/multi-agent/debate_results_downscale_merged.csv"
        RMB4["debate_results_downscale_verified.csv"]:::storage
        click RMB4 "https://github.com/lean-96/sos-mad/blob/main/results/multi-agent/debate_results_downscale_verified.csv"
        RMB5["debate_results_upscale_merged.csv"]:::storage
        click RMB5 "https://github.com/lean-96/sos-mad/blob/main/results/multi-agent/debate_results_upscale_merged.csv"
        RMB6["debate_results_upscale_verified.csv"]:::storage
        click RMB6 "https://github.com/lean-96/sos-mad/blob/main/results/multi-agent/debate_results_upscale_verified.csv"
        HVR[/"Human-Verified Results"/]:::storage
        click HVR "https://github.com/lean-96/sos-mad/tree/main/results/human_verified/"
    end

    subgraph "External Services" 
        style External fill:#e2e3f3,stroke:#383d7e,stroke-width:2px
        EOS((Ollama)):::external
        ORS((OpenRouter)):::external
    end

    %% Connections
    S0b -->|"reads .env & verifies"| CFG
    S0b -->|"depends on"| Env
    S0b -->|"installs deps"| S0
    S0 -->|"writes dataset"| DL1
    S0 -->|"populates manifests"| ML

    ML -->|"input for"| GM
    DA -->|"load tasks"| GM

    GM -->|"writes manifests"| ML
    ML -->|"feeds"| SA
    DA -->|"Load Manifests"| SA
    SA -->|"calls"| DP
    SA -->|"uses"| MO
    SA -->|"formats prompts"| PR
    DP -->|"calls API"| OLL
    DP -->|"calls API"| ORT
    SA -->|"writes JSON"| RSA1
    SA -->|"writes CSV"| RSA2

    ML -->|"feeds"| MA
    MA -->|"calls"| DP
    MA -->|"uses"| MO
    MA -->|"formats prompts"| PR
    MA -->|"calls API"| OLL
    MA -->|"calls API"| ORT
    MA -->|"writes history"| RMB1
    MA -->|"writes history"| RMB2
    MA -->|"aggregates CSVs"| RMB3
    MA -->|"aggregates CSVs"| RMB5

    SA -->|"verified by"| RSA3
    MA -->|"verified by"| RMB4
    MA -->|"verified by"| RMB6

    RSA2 -->|"used by"| AN
    RSA3 -->|"used by"| AN
    RMB3 -->|"used by"| AN
    RMB4 -->|"used by"| AN
    RMB5 -->|"used by"| AN
    RMB6 -->|"used by"| AN

    AN -->|"computes metrics"| AM1
    AN -->|"computes metrics"| AM2
    AN -->|"computes metrics"| AM3
    AN -->|"generates visuals"| AM4
    AN -->|"uses utils"| AM9
    AN -->|"writes reports"| HVR

    %% Styles
    classDef data fill:#cce5ff,stroke:#004085;
    classDef config fill:#cce5ff,stroke:#004085;
    classDef orchestration fill:#d4edda,stroke:#155724;
    classDef core fill:#ffe5b4,stroke:#854d00;
    classDef analysis fill:#fff3cd,stroke:#856404;
    classDef storage fill:#fff3cd,stroke:#856404;
    classDef external fill:#e2e3f3,stroke:#383d7e;
```
