How togethercomputer/moa works
Project Type and Purpose
• This is an open-source research toolkit and CLI library (Mixture-of-Agents, MoA) for orchestrating multiple LLMs in layered “mixture” setups, plus evaluation pipelines integrating three external benchmarks (AlpacaEval, MT-Bench/FastChat, and FLASK).
• The core library provides a layered agent orchestration (moa.py, advanced-moa.py), an interactive CLI demo (bot.py), and scripts to generate predictions and evaluate on the three benchmarks.
Key Directories and Components
A. Core MoA Engine
• moa.py / advanced-moa.py: define and run 2-layer or multi-layer mixtures of LLM agents.
• utils.py: shared helper functions (prompt formatting, API wrappers).
• openai_concurrent.py, openai_info/api_info.json: wrappers around OpenAI API for parallel calls.
• Together SDK dependency: external LLM provider (via Together API).
B. Interactive CLI
• bot.py: command-line interface, maintains conversation state and calls the core MoA engine each turn.
C. Evaluation Pipelines
1. FLASK (folder FLASK/)
– metadata_annotation: scripts to annotate skill, domain, difficulty on FLASK data.
– gpt_review: scripts to aggregate GPT-4 evaluations.
– model_output: code to load models and run inference for FLASK.
– evaluation_set & input_data: JSONL test sets for FLASK.
– run_eval_flask.sh: orchestrates generation and scoring.
2. FastChat (folder FastChat/)
– serve/: microservices architecture (API gateway, model workers, monitors, remote logger).
– llm_judge/: modules to run pairwise, GPT-4-based judgments on MT-Bench questions.
– model/: adapters and converters for different LLM backends (LLaMA, Falcon, etc.).
– train/: training scripts (not needed for inference).
– docker/: Dockerfile & compose for spinning up FastChat cluster.
– run_eval_mt_bench.sh: minimal example of MT-Bench evaluation.
3. AlpacaEval (folder alpaca_eval/)
– src/alpaca_eval/: core evaluation library (decoders, annotators, metrics).
– scripts/ and docs/: helpers to prepare data and format leaderboards.
– run_eval_alpaca_eval.sh: drives generation via MoA and then runs AlpacaEval.
D. Top-Level Utilities
• generate_for_*.py: glue scripts to call MoA engine and dump outputs for each benchmark.
• tests.py, tests/: unit and integration tests for MoA and evaluation scripts.
• requirements.txt, pyproject.toml: dependency manifests.
Architectural Patterns and Design Principles
• Layered Pipeline for MoA: each MoA run is a multi-layer pipeline—reference LLM agents → aggregator LLM → final output.
• Modular Evaluation Integration: each benchmark lives in its own directory, with a standard generate→evaluate script pattern.
• Microservices for FastChat: FastChat serve/ is a microservices-based inference cluster (API gateway, model workers, monitoring).
• Extensible Decoupling: decoders, annotators, specs in AlpacaEval and FLASK are pluggable via JSON/YAML config files.
Relationships and Data Flows
A. User Input (CLI or script)
→ Core MoA Engine (moa.py / advanced-moa.py)
→ Concurrent calls to Reference LLM Agents (Together API or OpenAI API)
→ Aggregator LLM call
→ Aggregated Response → CLI or output JSONL file
B. Evaluation Flow (for each benchmark):
1. generate_for_<benchmark>.py → calls MoA engine → dump predictions.json
2. Pass predictions.json + benchmark test set → evaluation library (alpaca_eval/, FastChat llm_judge, FLASK scripts)
3. Compute metrics → write leaderboard CSV / JSON → assets/graphs