How stonybrooknlp/appworld works
AppWorld is a simulated multi-app environment and benchmark for agents that solve natural-language tasks through interactive code and API calls. The main workflow is task selection, agent interaction with a task world and its app APIs, then evaluation against task ground truth. The README also documents serving the environment through APIs or MCP, plus task generation and agent experiments. Those latter areas are represented at their documented directory or entry-point level; source excerpts were not provided for their internal wiring.
Main entry points: appworld/environment.py AppWorld · evaluator.py Evaluator · serve/environment.py Environment Server · apis App APIs
Components 21
Task Runtime
- CLI
src/appworld/cli.py - Task
src/appworld/task.py - AppWorld
src/appworld/environment.py - Evaluator
src/appworld/evaluator.py - Ground Truth
src/appworld/ground_truth.py
Simulated Apps
- App APIs
src/appworld/apps/lib/apis - API Authentication
src/appworld/apps/lib/apis/authentication.py - App Models
src/appworld/apps/lib/models - App Databases
src/appworld/apps/lib/models/db.py - API Catalog
src/appworld/collections/apis.py - API Docs
src/appworld/collections/api_docs.py
Serving Interfaces
- API Server
src/appworld/serve/apis.py - Environment Server
src/appworld/serve/environment.py - MCP Server
src/appworld/serve/_mcp.py - UI Server
src/appworld/serve/uis.py - API Requester
src/appworld/requester.py
Benchmark Research
- Task Generation
generate/tasks - Task Validation
generate/tasks/generate_and_validate_tasks.py - Agent Experiments
experiments/code - Leaderboard
src/appworld/leaderboard.py
Other components
- Agent
Connections 19
- Agent → AppWorld: interacts with
- AppWorld → Task: loads task
- AppWorld → App APIs: executes calls
- App APIs → API Authentication: checks identity
- App APIs → App Models: uses models
- App Models → App Databases: reads and writes
- AppWorld → Evaluator: evaluates run
- Evaluator → Ground Truth: checks against
- CLI → AppWorld: starts world
- CLI → Evaluator: starts evaluation
- API Server → Environment Server: serves world
- Environment Server → AppWorld: exposes runtime
- MCP Server → Environment Server: exposes tools
- UI Server → Environment Server: serves interface
- API Requester → API Server: sends requests
- API Catalog → API Docs: documents APIs
- Task Generation → Task Validation: validated by
- Agent Experiments → CLI: runs experiments
- Evaluator → Leaderboard: reports results