# togethercomputer/moa architecture

> Architecture diagram of the GitHub repository togethercomputer/moa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/togethercomputer/moa
- Repository: https://github.com/togethercomputer/moa

## Overview

##### Project Type and Purpose
- This is an open-source research toolkit and CLI library (Mixture-of-Agents, MoA) for orchestrating multiple LLMs in layered “mixture” setups, plus evaluation pipelines integrating three external benchmarks (AlpacaEval, MT-Bench/FastChat, and FLASK).
- The core library provides a layered agent orchestration (moa.py, advanced-moa.py), an interactive CLI demo (bot.py), and scripts to generate predictions and evaluate on the three benchmarks.

##### Key Directories and Components
A. Core MoA Engine
- moa.py / advanced-moa.py: define and run 2-layer or multi-layer mixtures of LLM agents.
- utils.py: shared helper functions (prompt formatting, API wrappers).
- openai_concurrent.py, openai_info/api_info.json: wrappers around OpenAI API for parallel calls.
- Together SDK dependency: external LLM provider (via Together API).
B. Interactive CLI
- bot.py: command-line interface, maintains conversation state and calls the core MoA engine each turn.
C. Evaluation Pipelines
1. FLASK (folder FLASK/)
– metadata_annotation: scripts to annotate skill, domain, difficulty on FLASK data.
– gpt_review: scripts to aggregate GPT-4 evaluations.
– model_output: code to load models and run inference for FLASK.
– evaluation_set & input_data: JSONL test sets for FLASK.
– run_eval_flask.sh: orchestrates generation and scoring.
2. FastChat (folder FastChat/)
– serve/: microservices architecture (API gateway, model workers, monitors, remote logger).
– llm_judge/: modules to run pairwise, GPT-4-based judgments on MT-Bench questions.
– model/: adapters and converters for different LLM backends (LLaMA, Falcon, etc.).
– train/: training scripts (not needed for inference).
– docker/: Dockerfile & compose for spinning up FastChat cluster.
– run_eval_mt_bench.sh: minimal example of MT-Bench evaluation.
3. AlpacaEval (folder alpaca_eval/)
– src/alpaca_eval/: core evaluation library (decoders, annotators, metrics).
– scripts/ and docs/: helpers to prepare data and format leaderboards.
– run_eval_alpaca_eval.sh: drives generation via MoA and then runs AlpacaEval.
D. Top-Level Utilities
- generate_for_*.py: glue scripts to call MoA engine and dump outputs for each benchmark.
- tests.py, tests/: unit and integration tests for MoA and evaluation scripts.
- requirements.txt, pyproject.toml: dependency manifests.

##### Architectural Patterns and Design Principles
- Layered Pipeline for MoA: each MoA run is a multi-layer pipeline—reference LLM agents → aggregator LLM → final output.
- Modular Evaluation Integration: each benchmark lives in its own directory, with a standard generate→evaluate script pattern.
- Microservices for FastChat: FastChat serve/ is a microservices-based inference cluster (API gateway, model workers, monitoring).
- Extensible Decoupling: decoders, annotators, specs in AlpacaEval and FLASK are pluggable via JSON/YAML config files.

##### Relationships and Data Flows
A. User Input (CLI or script)
→ Core MoA Engine (moa.py / advanced-moa.py)
→ Concurrent calls to Reference LLM Agents (Together API or OpenAI API)
→ Aggregator LLM call
→ Aggregated Response → CLI or output JSONL file
B. Evaluation Flow (for each benchmark):
1. generate_for_<benchmark>.py → calls MoA engine → dump predictions.json
2. Pass predictions.json + benchmark test set → evaluation library (alpaca_eval/, FastChat llm_judge, FLASK scripts)
3. Compute metrics → write leaderboard CSV / JSON → assets/graphs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Interface
    subgraph "CLI Interface"
        CLI["bot.py"]:::script
    end

    %% Core MoA Engine
    subgraph "Core MoA Engine"
        moa["moa.py"]:::core
        adv["advanced-moa.py"]:::core
        utils["utils.py"]:::core
        pipeline["Layered MoA Pipeline"]:::note
        moa --> pipeline
        adv --> pipeline
    end

    %% External LLM Providers
    Together(("Together API")):::ext
    OpenAI(("OpenAI API")):::ext

    %% Generation Scripts
    subgraph "Generation Scripts"
        genF["generate_for_flask.py"]:::script
        genM["generate_for_mt_bench.py"]:::script
        genA["generate_for_alpaca_eval.py"]:::script
    end

    %% FLASK Evaluation Pipeline
    subgraph "FLASK Pipeline"
        ann["metadata_annotation/"]:::eval
        review["gpt_review/"]:::eval
        modelo["model_output/"]:::eval
        sets["evaluation_set/ & input_data/"]:::data
        runF["run_eval_flask.sh"]:::script
        ann --> modello
        modello --> review
        sets --> runF
        review --> runF
        runF -->|"metrics"| assetsF[(FLASK assets)]:::data
    end

    %% FastChat (MT-Bench) Evaluation Pipeline
    subgraph "FastChat Pipeline"
        docker["docker/"]:::eval
        serve["serve/"]:::eval
        judge["llm_judge/"]:::eval
        modeld["model/"]:::eval
        runM["run_eval_mt_bench.sh"]:::script
        docker --> serve
        serve --> judge
        judge --> runM
        modeld --> runM
        runM -->|"metrics"| assetsM[(MT-Bench assets)]:::data
    end

    %% AlpacaEval Evaluation Pipeline
    subgraph "AlpacaEval Pipeline"
        corelib["src/alpaca_eval/"]:::eval
        prep["scripts/ & docs/"]:::eval
        runA["run_eval_alpaca_eval.sh"]:::script
        corelib --> prep
        prep --> runA
        runA -->|"metrics"| assetsA[(AlpacaEval assets)]:::data
    end

    %% Data Flows
    CLI -->|"user input"| pipeline
    pipeline -->|"parallel LLM calls"| Together
    pipeline -->|"parallel LLM calls"| OpenAI
    pipeline --> genF
    pipeline --> genM
    pipeline --> genA
    genF --> ann
    genM --> serve
    genA --> corelib

    %% Click Events
    click moa "https://github.com/togethercomputer/moa/blob/main//moa.py"
    click adv "https://github.com/togethercomputer/moa/blob/main//advanced-moa.py"
    click utils "https://github.com/togethercomputer/moa/blob/main//utils.py"
    click CLI "https://github.com/togethercomputer/moa/blob/main//bot.py"
    click ann "https://github.com/togethercomputer/moa/tree/main/FLASK/metadata_annotation/"
    click review "https://github.com/togethercomputer/moa/tree/main/FLASK/gpt_review/"
    click modello "https://github.com/togethercomputer/moa/tree/main/FLASK/model_output/"
    click sets "https://github.com/togethercomputer/moa/tree/main/FLASK/evaluation_set/"
    click runF "https://github.com/togethercomputer/moa/blob/main//run_eval_flask.sh"
    click docker "https://github.com/togethercomputer/moa/tree/main/FastChat/docker/"
    click serve "https://github.com/togethercomputer/moa/tree/main/FastChat/fastchat/serve/"
    click judge "https://github.com/togethercomputer/moa/tree/main/FastChat/fastchat/llm_judge/"
    click modeld "https://github.com/togethercomputer/moa/tree/main/FastChat/fastchat/model/"
    click runM "https://github.com/togethercomputer/moa/blob/main//run_eval_mt_bench.sh"
    click corelib "https://github.com/togethercomputer/moa/tree/main/alpaca_eval/src/alpaca_eval/"
    click prep "https://github.com/togethercomputer/moa/tree/main/alpaca_eval/scripts/"
    click runA "https://github.com/togethercomputer/moa/blob/main//run_eval_alpaca_eval.sh"
    click genF "https://github.com/togethercomputer/moa/blob/main//generate_for_flask.py"
    click genM "https://github.com/togethercomputer/moa/blob/main//generate_for_mt_bench.py"
    click genA "https://github.com/togethercomputer/moa/blob/main//generate_for_alpaca_eval.py"

    %% Styling
    classDef core fill:#D0E8FF,stroke:#0366d6,stroke-width:1px;
    classDef ext fill:#FFE0B2,stroke:#FF9800,stroke-width:1px;
    classDef eval fill:#E8F5E9,stroke:#388E3C,stroke-width:1px;
    classDef script fill:#FFF3E0,stroke:#FB8C00,stroke-width:1px;
    classDef data fill:#F5F5F5,stroke:#9E9E9E,stroke-width:1px;
    classDef note fill:#FFFFFF,stroke-dasharray: 5 5,stroke:#757575;
    class pipeline,note note
```
