How skytliang/multi-agents-debate works
This repo is a Python research/experiment framework for MAD (Multi-Agent Debate) over LLMs, used to improve outcomes on:
• Commonsense Machine Translation (CommonMT / Lexical Ambiguity subset)
• Counterintuitive QA (CIAR dataset)
It is not a “service” with a backend/frontend; it’s primarily a batch evaluation pipeline plus a local interactive runner that orchestrates multiple LLM “agents” (devil/angel/judge) via the OpenAI API, reads task datasets from data/, and writes outputs/logs/results back to the filesystem under data/**/output/*.
• Orchestration flow (runner → agents → debate rounds → judge → final answer)
• External dependency (OpenAI API)
• Dataflow (dataset JSON/TXT → prompts → model calls → outputs)
• Config-driven behavior (JSON configs controlling agents/models/prompts)
File-structure-driven architecture (map folders/files to components)
From the tree, the core runtime components are:
Entry points / Runners
• debate4tran.sh
Shell entrypoint for the translation MAD run (sets API key, launches Python).
• interactive.py
A local interactive script (likely single-instance “demo mode” to try debate manually).
• code/debate4tran.py
The main Python orchestration for the translation pipeline (dataset iteration, debate loop, output writing).
Core library / MAD engine pieces
• code/utils/agent.py
Defines the “Agent” abstraction(s): devil/angel/judge behavior, message format, debate step logic.
• code/utils/openai_utils.py
Wrapper around OpenAI calls (request formatting, retries, rate limits, parsing, etc.).
• code/utils/config4tran.json, code/utils/config4all.json
Configuration: model names, temperatures, prompts/roles, debate rounds, paths, etc.
• code/__init__.py, code/utils/__init__.py
Package plumbing.
Data / Artifacts
• data/CommonMT/**
Includes raw parallel text (raw/lexical.zh-en.*) and a large set of per-example JSON debate processes under MAD_Debate_Process/*.json, plus outputs grouped by method:
• output/baseline, output/gpt4, output/mad, output/maps, output/rerank, output/self_reflect
• data/CounterintuitiveQA/CIAR.json
QA dataset.
• data/CommonMT/input.example.txt
Example input.
• imgs/*
Dependency management
• requirements.txt
Captures Python deps (OpenAI SDK, etc.).
Dataset Loader
• CommonMT raw inputs (data/CommonMT/.../raw/*.zh|*.en)
• Or CIAR (data/CounterintuitiveQA/CIAR.json) if supported by other scripts (README indicates experiments on it even if the main runner shown is translation-focused)
6. MAD Engine (Debate Loop Coordinator) (conceptual component; implemented across orchestrator + agent.py)
Agents (separate nodes)
• Affirmative Agent (Devil)
• Negative Agent (Angel)
• Judge Agent
Output Writer / Artifact Store (Filesystem)
• Debate traces: data/**/MAD_Debate_Process/*.json
• Predictions/results: data/**/output/{mad,baseline,self_reflect,...}
• Dataset → Orchestrator → Agents → OpenAI → Agents → Judge → Final Answer → Output
• Config → Orchestrator & Agent instantiation
• Output includes both final predictions and debate transcripts
Offline evaluation / methods comparison (because outputs are method-labeled)
• Baseline pipeline
• Self-reflection pipeline
• MAD pipeline
• Possibly MAPS/rerank pipelines (or “external results imported” if not produced here)
This clarifies that MAD is part of a broader experimental harness.
• debate4tran.sh → python code/debate4tran.py
• debate4tran.py → config4tran.json
• debate4tran.py → Dataset files
• debate4tran.py → Agent(devil/angel/judge)
• Agent → openai_utils.py → OpenAI API
• Agents ↔ Debate State / Transcript (conceptual store in-memory)
• Judge → Final decision
• Orchestrator → Filesystem outputs
An explicit boundary:
• Everything in this repo runs on “Local machine / single process”
• OpenAI API is “External managed service”
Architectural patterns / principles visible here
• Orchestrator + strategy objects: debate4tran.py coordinates; Agent encapsulates role-specific prompting/behavior.
• Config-driven experimentation: JSON configs define behavior without code changes.
• Pipeline architecture: load → generate (multi-step) → judge → persist artifacts.
• Human-readable traceability: debate transcripts persisted as JSON per example (reproducibility, audit).
What “very detailed” means here (don’t overthink; split aggressively)
To match this repo, it’s better to over-separate than to compress:
• Separate each agent role as its own node.
• Separate dataset input vs output artifacts.
• Separate config from code.
• multiple output modes (baseline/mad/self_reflect/etc.) as parallel pipelines or as a “mode” selector feeding the same output writer.