# skytliang/multi-agents-debate architecture

> Architecture diagram of the GitHub repository skytliang/multi-agents-debate, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skytliang/multi-agents-debate
- Repository: https://github.com/skytliang/multi-agents-debate

## Overview

This repo is a **Python research/experiment framework** for **MAD (Multi-Agent Debate)** over LLMs, used to improve outcomes on:
- **Commonsense Machine Translation** (CommonMT / Lexical Ambiguity subset)
- **Counterintuitive QA** (CIAR dataset)

It is not a “service” with a backend/frontend; it’s primarily a **batch evaluation pipeline** plus a **local interactive runner** that orchestrates **multiple LLM “agents” (devil/angel/judge)** via the **OpenAI API**, reads **task datasets** from `data/`, and writes **outputs/logs/results** back to the filesystem under `data/**/output/*`.

- **Orchestration flow** (runner → agents → debate rounds → judge → final answer)
- **External dependency** (OpenAI API)
- **Dataflow** (dataset JSON/TXT → prompts → model calls → outputs)
- **Config-driven behavior** (JSON configs controlling agents/models/prompts)

##### File-structure-driven architecture (map folders/files to components)
From the tree, the core runtime components are:

##### Entry points / Runners
- `debate4tran.sh`
Shell entrypoint for the translation MAD run (sets API key, launches Python).
- `interactive.py`
A local interactive script (likely single-instance “demo mode” to try debate manually).
- `code/debate4tran.py`
The main Python orchestration for the translation pipeline (dataset iteration, debate loop, output writing).

##### Core library / MAD engine pieces
- `code/utils/agent.py`
Defines the “Agent” abstraction(s): devil/angel/judge behavior, message format, debate step logic.
- `code/utils/openai_utils.py`
Wrapper around OpenAI calls (request formatting, retries, rate limits, parsing, etc.).
- `code/utils/config4tran.json`, `code/utils/config4all.json`
Configuration: model names, temperatures, prompts/roles, debate rounds, paths, etc.
- `code/__init__.py`, `code/utils/__init__.py`
Package plumbing.

##### Data / Artifacts
- `data/CommonMT/**`
Includes raw parallel text (`raw/lexical.zh-en.*`) and a large set of **per-example JSON debate processes** under `MAD_Debate_Process/*.json`, plus outputs grouped by method:
- `output/baseline`, `output/gpt4`, `output/mad`, `output/maps`, `output/rerank`, `output/self_reflect`
- `data/CounterintuitiveQA/CIAR.json`
QA dataset.
- `data/CommonMT/input.example.txt`
Example input.
- `imgs/*`

##### Dependency management
- `requirements.txt`
Captures Python deps (OpenAI SDK, etc.).

##### Dataset Loader
- CommonMT raw inputs (`data/CommonMT/.../raw/*.zh|*.en`)
- Or CIAR (`data/CounterintuitiveQA/CIAR.json`) if supported by other scripts (README indicates experiments on it even if the main runner shown is translation-focused)
6. **MAD Engine (Debate Loop Coordinator)** (conceptual component; implemented across orchestrator + `agent.py`)
##### Agents (separate nodes)
- **Affirmative Agent (Devil)**
- **Negative Agent (Angel)**
- **Judge Agent**
##### Output Writer / Artifact Store (Filesystem)
- Debate traces: `data/**/MAD_Debate_Process/*.json`
- Predictions/results: `data/**/output/{mad,baseline,self_reflect,...}`

- Dataset → Orchestrator → Agents → OpenAI → Agents → Judge → Final Answer → Output
- Config → Orchestrator & Agent instantiation
- Output includes both **final predictions** and **debate transcripts**

##### Offline evaluation / methods comparison (because outputs are method-labeled)
- **Baseline pipeline**
- **Self-reflection pipeline**
- **MAD pipeline**
- Possibly **MAPS/rerank** pipelines (or “external results imported” if not produced here)

This clarifies that MAD is part of a broader experimental harness.

- `debate4tran.sh` → `python code/debate4tran.py`
- `debate4tran.py` → `config4tran.json`
- `debate4tran.py` → `Dataset files`
- `debate4tran.py` → `Agent(devil/angel/judge)`
- `Agent` → `openai_utils.py` → `OpenAI API`
- `Agents` ↔ `Debate State / Transcript` (conceptual store in-memory)
- `Judge` → `Final decision`
- `Orchestrator` → `Filesystem outputs`

An explicit boundary:
- Everything in this repo runs on “**Local machine / single process**”
- OpenAI API is “**External managed service**”

##### Architectural patterns / principles visible here

- **Orchestrator + strategy objects**: `debate4tran.py` coordinates; `Agent` encapsulates role-specific prompting/behavior.
- **Config-driven experimentation**: JSON configs define behavior without code changes.
- **Pipeline architecture**: load → generate (multi-step) → judge → persist artifacts.
- **Human-readable traceability**: debate transcripts persisted as JSON per example (reproducibility, audit).

##### What “very detailed” means here (don’t overthink; split aggressively)
To match this repo, it’s better to over-separate than to compress:
- Separate **each agent role** as its own node.
- Separate **dataset input** vs **output artifacts**.
- Separate **config** from **code**.
- **multiple output modes** (baseline/mad/self_reflect/etc.) as parallel pipelines or as a “mode” selector feeding the same output writer.

## Diagram (Mermaid)

```mermaid
flowchart TD

%% =======================
%% Context / Actors
%% =======================
R["Researcher / User"]:::actor
OA["OpenAI API (External Managed Service)"]:::external

%% =======================
%% Local Machine Boundary
%% =======================
subgraph "Local Machine (Single Process) — This Repo"
direction TB

    %% ---- Entry Points ----
    subgraph "Entry Points / Runners"
    direction TB
        SH["Shell Runner: debate4tran.sh"]:::entry
        INT["Interactive Runner: interactive.py"]:::entry
    end

    %% ---- Core Orchestration ----
    subgraph "Orchestration Layer"
    direction TB
        ORCH["Batch Orchestrator: code/debate4tran.py"]:::orchestrator
        LOOP["MAD Engine: Debate Loop Coordinator"]:::engine
        MODE["Method / Mode Selector\n(baseline | gpt4 | mad | maps | rerank | self_reflect)"]:::engine
    end

    %% ---- Config & Library ----
    subgraph "Core Library (code/utils)"
    direction TB
        CFGT["Config: config4tran.json"]:::config
        CFGA["Config: config4all.json"]:::config
        AGENT["Agent Abstraction: agent.py"]:::core
        OAIU["OpenAI Client Wrapper: openai_utils.py"]:::core
        INIT1["Package Init: code/__init__.py"]:::plumbing
        INIT2["Package Init: code/utils/__init__.py"]:::plumbing
    end

    %% ---- Agents ----
    subgraph "Agents (Role-driven Strategy Objects)"
    direction TB
        DEV["Devil (Affirmative Agent)"]:::agent
        ANG["Angel (Negative Agent)"]:::agent
        JUD["Judge Agent"]:::judge
    end

    %% ---- Data Sources ----
    subgraph "Datasets (Filesystem Inputs)"
    direction TB
        ZH[("CommonMT Raw Source\nlexical.zh-en.zh")]:::data
        EN[("CommonMT Raw Target\nlexical.zh-en.en")]:::data
        CIAR[("CounterintuitiveQA Dataset\nCIAR.json")]:::data
        EXIN[("Example Input\ninput.example.txt")]:::data
        DOC[("Dataset Notes\nCommonMT README.md")]:::doc
    end

    %% ---- Artifact Stores ----
    subgraph "Artifacts (Filesystem Outputs)"
    direction TB
        TRACE[("Debate Trace Store\nMAD_Debate_Process/*.json")]:::store

        subgraph "Predictions / Results (Method-labeled Outputs)"
        direction TB
            OUTB[("output/baseline/")]:::store
            OUTG[("output/gpt4/")]:::store
            OUTM[("output/mad/")]:::store
            OUTMAPS[("output/maps/")]:::store
            OUTR[("output/rerank/")]:::store
            OUTSR[("output/self_reflect/")]:::store
        end
    end

    %% ---- Dependencies ----
    REQ["requirements.txt"]:::deps
    FIGS[("Figures (imgs/)")]:::doc
end

%% =======================
%% Control Flow (Solid)
%% =======================
R -->|"runs_batch"| SH
R -->|"runs_interactive"| INT

SH -->|"launches_python;injects_OPENAI_API_KEY"| ORCH
INT -->|"starts_local_session"| LOOP

ORCH -->|"loads_config"| CFGT
ORCH -->|"loads_config"| CFGA
ORCH -->|"uses_library"| INIT1
ORCH -->|"uses_library"| INIT2

ORCH -->|"creates_mode_pipeline"| MODE
MODE -->|"selects_flow"| LOOP

LOOP -->|"instantiates_agents(role,prompt,params)"| AGENT
AGENT -->|"creates_role_instances"| DEV
AGENT -->|"creates_role_instances"| ANG
AGENT -->|"creates_role_instances"| JUD

DEV -->|"calls_llm_via_wrapper"| OAIU
ANG -->|"calls_llm_via_wrapper"| OAIU
JUD -->|"calls_llm_via_wrapper"| OAIU
OAIU -->|"api_request(messages,temp,max_tokens)"| OA
OA -->|"api_response(text,usage)"| OAIU
OAIU -->|"parsed_response"| DEV
OAIU -->|"parsed_response"| ANG
OAIU -->|"parsed_response"| JUD

%% Debate rounds loop annotation (control)
DEV -->|"Round_1..N:propose"| ANG
ANG -->|"Round_1..N:rebut"| DEV
DEV -->|"Round_1..N:revise"| JUD
ANG -->|"Round_1..N:critique"| JUD
JUD -->|"final_decision(best_candidate)"| LOOP

LOOP -->|"writes_transcripts_and_predictions"| TRACE
LOOP -->|"writes_baseline_results"| OUTB
LOOP -->|"writes_gpt4_results"| OUTG
LOOP -->|"writes_mad_results"| OUTM
LOOP -->|"writes_maps_results_or_imports"| OUTMAPS
LOOP -->|"writes_rerank_results_or_imports"| OUTR
LOOP -->|"writes_self_reflect_results"| OUTSR

REQ -->|"pins_deps_for"| ORCH
REQ -->|"pins_deps_for"| OAIU

%% =======================
%% Data Flow (Dashed)
%% =======================
ORCH -.->|"reads_examples"| ZH
ORCH -.->|"reads_examples"| EN
ORCH -.->|"reads_examples_optional"| CIAR
INT -.->|"may_use_example_input"| EXIN
DOC -.->|"documents_dataset_and_outputs"| ORCH
FIGS -.->|"conceptual_reference"| R

DEV -.->|"appends_message_to_transcript"| TRACE
ANG -.->|"appends_message_to_transcript"| TRACE
JUD -.->|"appends_judgement_to_transcript"| TRACE

%% =======================
%% Click Events (Component Mapping)
%% =======================
click SH "https://github.com/skytliang/multi-agents-debate/blob/main/debate4tran.sh"
click INT "https://github.com/skytliang/multi-agents-debate/blob/main/interactive.py"
click ORCH "https://github.com/skytliang/multi-agents-debate/blob/main/code/debate4tran.py"
click LOOP "https://github.com/skytliang/multi-agents-debate/blob/main/code/debate4tran.py"
click AGENT "https://github.com/skytliang/multi-agents-debate/blob/main/code/utils/agent.py"
click OAIU "https://github.com/skytliang/multi-agents-debate/blob/main/code/utils/openai_utils.py"
click CFGT "https://github.com/skytliang/multi-agents-debate/blob/main/code/utils/config4tran.json"
click CFGA "https://github.com/skytliang/multi-agents-debate/blob/main/code/utils/config4all.json"
click INIT1 "https://github.com/skytliang/multi-agents-debate/blob/main/code/__init__.py"
click INIT2 "https://github.com/skytliang/multi-agents-debate/blob/main/code/utils/__init__.py"
click ZH "https://github.com/skytliang/multi-agents-debate/blob/main/data/CommonMT/Lexical_Ambiguity/raw/lexical.zh-en.zh"
click EN "https://github.com/skytliang/multi-agents-debate/blob/main/data/CommonMT/Lexical_Ambiguity/raw/lexical.zh-en.en"
click CIAR "https://github.com/skytliang/multi-agents-debate/blob/main/data/CounterintuitiveQA/CIAR.json"
click EXIN "https://github.com/skytliang/multi-agents-debate/blob/main/data/CommonMT/input.example.txt"
click TRACE "https://github.com/skytliang/multi-agents-debate/tree/main/data/CommonMT/Lexical_Ambiguity/MAD_Debate_Process/"
click OUTB "https://github.com/skytliang/multi-agents-debate/tree/main/data/CommonMT/Lexical_Ambiguity/output/baseline/"
click OUTG "https://github.com/skytliang/multi-agents-debate/tree/main/data/CommonMT/Lexical_Ambiguity/output/gpt4/"
click OUTM "https://github.com/skytliang/multi-agents-debate/tree/main/data/CommonMT/Lexical_Ambiguity/output/mad/"
click OUTMAPS "https://github.com/skytliang/multi-agents-debate/tree/main/data/CommonMT/Lexical_Ambiguity/output/maps/"
click OUTR "https://github.com/skytliang/multi-agents-debate/tree/main/data/CommonMT/Lexical_Ambiguity/output/rerank/"
click OUTSR "https://github.com/skytliang/multi-agents-debate/tree/main/data/CommonMT/Lexical_Ambiguity/output/self_reflect/"
click REQ "https://github.com/skytliang/multi-agents-debate/blob/main/requirements.txt"
click DOC "https://github.com/skytliang/multi-agents-debate/blob/main/data/CommonMT/README.md"
click FIGS "https://github.com/skytliang/multi-agents-debate/tree/main/imgs/"

%% =======================
%% Styles
%% =======================
classDef actor fill:#0b7285,stroke:#063b44,color:#ffffff,stroke-width:1.5px
classDef entry fill:#2f9e44,stroke:#1b5e20,color:#ffffff,stroke-width:1.5px
classDef orchestrator fill:#1864ab,stroke:#0b3d66,color:#ffffff,stroke-width:1.5px
classDef engine fill:#1971c2,stroke:#0b3d66,color:#ffffff,stroke-width:1.5px
classDef core fill:#364fc7,stroke:#1b2a80,color:#ffffff,stroke-width:1.5px
classDef agent fill:#5f3dc4,stroke:#321b82,color:#ffffff,stroke-width:1.5px
classDef judge fill:#7048e8,stroke:#3b1fa6,color:#ffffff,stroke-width:1.5px
classDef config fill:#845ef7,stroke:#4c2fb0,color:#ffffff,stroke-width:1.5px
classDef data fill:#f08c00,stroke:#8a4b00,color:#1f1300,stroke-width:1.5px
classDef store fill:#ffd43b,stroke:#8a6b00,color:#1f1a00,stroke-width:1.5px
classDef external fill:#e03131,stroke:#7a1a1a,color:#ffffff,stroke-width:1.5px
classDef deps fill:#495057,stroke:#212529,color:#ffffff,stroke-width:1.5px
classDef plumbing fill:#868e96,stroke:#343a40,color:#ffffff,stroke-width:1.5px
classDef doc fill:#ffe8cc,stroke:#b36b00,color:#3a2a00,stroke-width:1.5px
```
