How shishirpatil/gorilla works
Project Type and Purpose
• This is a multi-component research and benchmarking platform for evaluating large-language-model (LLM) function-calling capabilities and agent workflows.
• Benchmarking is handled by three main subsystems:
• Berkeley Function-Calling Leaderboard (BFCL)
• Agent Arena
• Gorilla Evaluation Scripts (AST-based, CodeBLEU)
• Data catalogs (APIBench / API Zoo) provide the API definitions and test questions.
Key Benchmarking Components (from file tree)
• data/ → APIBench and APIZoo datasets (question/ground-truth definitions).
• gorilla/eval → Retrieval modules, LLM-inference scripts, AST and CodeBLEU evaluators.
• berkeley-function-call-leaderboard/
• bfcl/ → Benchmark runner, model handlers, eval checker, metrics.
• data/ → BFCL test sets and possible answers.
• utils/ → Compilation helpers, format checkers, visualization tools.
• agent-arena/ → UI client, evaluation notebooks, ELO rating engine.
• gorilla/inference & openfunctions/ → LLM serving endpoints used by the benchmarks.
Relationships & Data Flow
1. Test Definition
• Source: data/apizoo, data/BFCL_*.json, agent-arena/client/public/…
↓
2. Retrieval & Prompt Assembly
• Modules: gorilla/eval/retrievers (BM25, GPT)
↓
3. LLM Inference
• gorilla/inference (finite model), openfunctions (function calling endpoints), goex exec engine (optional)
↓
4. Response Collection
• Output: JSONL responses under gorilla/eval/responses or BFCL outputs
↓
5. Evaluation
• AST Eval (gorilla/eval), CodeBLEU (gorilla/eval/codebleu), function-calling correctness (bfcl/eval_checker)
↓
6. Metrics Aggregation & Storage
• BFCL computes accuracy, latency, cost; Agent Arena computes Elo scores
↓
7. Leaderboard UI
• agent-arena/client (React front end) connects via REST/GraphQL to metrics store
• BFCL includes an interactive explorer (notebook or web UI)
Architectural Patterns & Design Principles
• Pipeline Architecture: clearly separated stages (retrieve → infer → evaluate → aggregate → display)
• Modular Services:
• Retrieval (BM25, GPT-based)
• Inference (Gorilla, OpenFunctions, GoEx)
• Evaluation (AST, CodeBLEU, state-based eval)
• Data-Driven: test definitions and ground truth are pure data artifacts
• Extensibility: plug new LLMs into bfcl/model_handler or new tasks into agent-arena