How moreagentsisallyouneed/agentforest works
Project type and purpose
• Type: A research-oriented CLI framework for running multi-agent experiments with Large Language Models (LLMs).
• Purpose: Orchestrate N LLM agents to solve tasks (MATH, GSM, MMLU, Chess, HumanEval), collect and evaluate their responses, and compare performance across models (OpenAI GPT on Azure or self-hosted LLM).
File structure
• Top level
– README.md: Project overview and setup
– dataset/: Raw problem sets (Chess, GSM, Math, MMLU)
– img/: Illustrative algorithm PDF
– script/: Shell wrappers (run.sh, run_generation_task.sh, run_reasoning_task.sh)
– src/: Core Python modules
• src/ layout
– main.py: Entry point—parses CLI args, dispatches tasks
– dataloader.py: Loads JSON/CSV into in-memory examples
– {chess, gsm, math, mmlu}_task.py: Task-specific prompt construction and parsing
– agent.py & more_agent.py: Abstractions for single-agent vs. multi-agent orchestration
– prompt_lib.py: Reusable prompt templates and system messages
– evaluation.py & utils.py: Scoring and helper routines
– categories.py: Defines question categories or labels
• script/
– run.sh: Wrapper that accepts {AGENT_NUM} {MODEL} {QTYPE} and invokes the correct sub-script
– run_generation_task.sh vs. run_reasoning_task.sh: Shell variants based on question type
README for additional insights
• External dependencies:
– human-eval framework (for HumanEval tasks)
– Environment variables for API endpoints and keys (OPENAI_KEY, OPENAI_IP, LLM_IP)
• Modes of operation: Multi-agent (“More Agents”) vs. single-agent runs
• Supported models: Azure-hosted GPT series or open-source LLM via HTTP
a. Main components
1. CLI / Shell Scripts (script/run.sh → run_generation_task.sh or run_reasoning_task.sh)
2. Orchestrator (src/main.py)
3. Data Layer (src/dataloader.py + dataset/*)
4. Task Modules (src/*_task.py)
5. Agent Layer (src/agent.py, src/more_agent.py)
6. Prompt Library (src/prompt_lib.py)
7. LLM Interfaces (src/utils.py handling API calls to Azure or self-hosted LLM)
8. Evaluation Engine (src/evaluation.py + human-eval integration)
9. Output Store (file system or JSON/CSV results)
b. Relationships and interactions
– Shell Scripts invoke the Orchestrator with parameters
– Orchestrator calls Data Layer to load problems
– For each example, Orchestrator uses Task Modules + Prompt Library to build prompts
– Agent Layer spins up N agents; each agent calls the LLM Interface in parallel or sequence
– LLM Interface issues HTTP requests to external LLM services (Azure OpenAI or LLM_IP)
– Responses routed back to Orchestrator, then passed to Evaluation Engine
– Evaluation Engine uses metrics or human-eval scripts, writes out results
c. Architectural patterns
– Modular “Task” plugins: each question type lives in its own module, enabling easy extension
– Agent abstraction: single vs. multi-agent interchangeable via strategy pattern
– Clear separation of concerns: data ingestion, prompt generation, agent orchestration, evaluation
d. Key technologies and frameworks
– Python 3.x for orchestration and logic
– Shell scripting for experiment launch
– HTTP/REST calls to LLM endpoints
– JSON/CSV data formats for datasets and results
– human-eval framework for standard evaluation