How salesforce/bolaa works
Project Type and Purpose
• This repo implements a benchmarking framework (“BOLAA”) for LLM-augmented autonomous agents.
• It orchestrates six agent architectures (Zeroshot, ReAct, PlanAct, etc.) paired with different LLMs and evaluates them in two environments:
– HotPotQA question-answering
– Webshop navigation
Top-Level Components (based on file tree)
A. Orchestrators (CLI entrypoints)
• run_hotpotqaagent.py
• run_webagent.py
B. Agent Frameworks
• hotpotqa_run/ (agent definitions, environment wrappers, evaluation, data)
• web_run/ (same structure, specialized for web tasks)
C. Environment Implementations
• hotpotqa_run/hotpotqa_env.py + wikienv.py
• web_run/web_env.py
• webshop/ (full Flask-based webshop simulation, Selenium env, shop_agent)
D. LLM Interface
• hotpotqa_run/llms.py & web_run/llms.py (wrap calls to OpenAI or FastChat)
E. Utilities & Configuration
• config.py in each run folder
• pre_prompt.py, utils.py, wrappers.py
F. Test Suites & Data
• webshop/tests, hotpotqa_run/data/*.joblib
Interactions and Data Flow
1. CLI invokes the appropriate “run_*” script.
2. Script loads config → registers agent architecture from agent_arch.py.
3. Orchestrator loops through episodes:
a. Calls environment.reset() → returns initial observation.
b. Agent receives observation → generates action via llm interface.
c. Environment.step(action) → returns next observation & reward.
d. Repeat until done; record metrics.
4. Evaluator collects per-episode rewards, logs, and aggregates.
5. (Webshop only) Agent actions trigger browser interactions via Selenium in webshop/web_agent_site/envs.
Architectural Patterns and Design Principles
• Modular Environment Abstraction (Env API similar to OpenAI Gym)
• Strategy Pattern for agent architectures (each variant as a class in agent_arch.py)
• Adapter Pattern for LLM layer (llms.py abstracts OpenAI vs. local models)
• Separation of Concerns:
– CLI/orchestrator vs. agent logic vs. environment simulation vs. evaluation
Technologies & Frameworks to Highlight
• Python 3.10, Conda environment
• LangChain (for prompt orchestration)
• OpenAI API / FastChat
• Selenium + ChromeDriver (webshop env)
• Flask / Jinja2 (web_agent_site)
• Joblib (caching HotPotQA data)