# shishirpatil/gorilla architecture

> Architecture diagram of the GitHub repository shishirpatil/gorilla, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shishirpatil/gorilla
- Repository: https://github.com/shishirpatil/gorilla

## Overview

##### Project Type and Purpose
- This is a multi-component research and benchmarking platform for evaluating large-language-model (LLM) function-calling capabilities and agent workflows.
- Benchmarking is handled by three main subsystems:
- Berkeley Function-Calling Leaderboard (BFCL)
- Agent Arena
- Gorilla Evaluation Scripts (AST-based, CodeBLEU)
- Data catalogs (APIBench / API Zoo) provide the API definitions and test questions.

##### Key Benchmarking Components (from file tree)
- data/         → APIBench and APIZoo datasets (question/ground-truth definitions).
- gorilla/eval  → Retrieval modules, LLM-inference scripts, AST and CodeBLEU evaluators.
- berkeley-function-call-leaderboard/
- bfcl/    → Benchmark runner, model handlers, eval checker, metrics.
- data/    → BFCL test sets and possible answers.
- utils/   → Compilation helpers, format checkers, visualization tools.
- agent-arena/  → UI client, evaluation notebooks, ELO rating engine.
- gorilla/inference & openfunctions/ → LLM serving endpoints used by the benchmarks.

##### Relationships & Data Flow
1. Test Definition
- Source: data/apizoo, data/BFCL_*.json, agent-arena/client/public/…
↓
2. Retrieval & Prompt Assembly
- Modules: gorilla/eval/retrievers (BM25, GPT)
↓
3. LLM Inference
- gorilla/inference (finite model), openfunctions (function calling endpoints), goex exec engine (optional)
↓
4. Response Collection
- Output: JSONL responses under gorilla/eval/responses or BFCL outputs
↓
5. Evaluation
- AST Eval (gorilla/eval), CodeBLEU (gorilla/eval/codebleu), function-calling correctness (bfcl/eval_checker)
↓
6. Metrics Aggregation & Storage
- BFCL computes accuracy, latency, cost; Agent Arena computes Elo scores
↓
7. Leaderboard UI
- agent-arena/client (React front end) connects via REST/GraphQL to metrics store
- BFCL includes an interactive explorer (notebook or web UI)

##### Architectural Patterns & Design Principles
- Pipeline Architecture: clearly separated stages (retrieve → infer → evaluate → aggregate → display)
- Modular Services:
- Retrieval (BM25, GPT-based)
- Inference (Gorilla, OpenFunctions, GoEx)
- Evaluation (AST, CodeBLEU, state-based eval)
- Data-Driven: test definitions and ground truth are pure data artifacts
- Extensibility: plug new LLMs into bfcl/model_handler or new tasks into agent-arena

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Store
    subgraph "Data Store"
        ds1["APIBench dataset"]:::data
        ds2["APIZoo dataset"]:::data
        ds3["BFCL Test Data"]:::data
    end

    %% Retrieval Service
    subgraph "Retrieval Service"
        r1(("BM25 Retriever")):::compute
        r2(("GPT Retriever")):::compute
    end

    %% Inference Service
    subgraph "Inference Service"
        inf1(("Gorilla LLM Endpoint")):::compute
        inf2(("OpenFunctions Endpoint")):::compute
        inf3(("GoEx Execution Engine")):::compute
    end

    %% BFCL Benchmarking Orchestrator
    subgraph "BFCL Benchmarking"
        bfcl_runner["BFCL Benchmark Runner"]:::compute
        bfcl_models["BFCL Model Handlers"]:::compute
        bfcl_checker["BFCL Eval Checker"]:::compute
    end

    %% Gorilla Evaluation
    subgraph "Gorilla Evaluation Service"
        gorilla_prompt["Gorilla Prompt Assembly"]:::compute
        eval_ast(("AST Evaluator")):::compute
        eval_codebleu(("CodeBLEU Evaluator")):::compute
    end

    %% Metrics Store
    subgraph "Metrics Store"
        metrics["Metrics Artifacts (JSON/DB)"]:::storage
    end

    %% Dashboard UI
    subgraph "Dashboard UI"
        ui1["Agent Arena React UI"]:::ui
        ui2["BFCL Explorer UI"]:::ui
    end

    %% Data Flow
    ds1 -->|"questions.jsonν"| r1
    ds2 -->|"questions.jsonν"| r1
    ds3 -->|"benchmark sets"| bfcl_runner

    r1 -->|"retrieved docs"| gorilla_prompt
    r2 -->|"retrieved docs"| gorilla_prompt

    gorilla_prompt -->|"prompt"| inf1
    gorilla_prompt -->|"prompt"| inf2
    gorilla_prompt -->|"prompt"| inf3

    inf1 -->|"responses.jsonl"| eval_ast
    inf1 -->|"responses.jsonl"| eval_codebleu
    inf1 -->|"responses.jsonl"| bfcl_runner

    inf2 -->|"responses.jsonl"| eval_ast
    inf2 -->|"responses.jsonl"| eval_codebleu
    inf2 -->|"responses.jsonl"| bfcl_runner

    inf3 -->|"responses.jsonl"| eval_ast
    inf3 -->|"responses.jsonl"| eval_codebleu
    inf3 -->|"responses.jsonl"| bfcl_runner

    eval_ast -->|"ast_scores"| metrics
    eval_codebleu -->|"bleu_scores"| metrics

    bfcl_runner -->|"uses handlers"| bfcl_models
    bfcl_runner -->|"runs eval"| bfcl_checker
    bfcl_checker -->|"fc_results"| metrics

    metrics -->|"serves data"| ui1
    metrics -->|"serves data"| ui2

    %% Click Events
    click ds1 "https://github.com/shishirpatil/gorilla/tree/main/data/apibench"
    click ds2 "https://github.com/shishirpatil/gorilla/tree/main/data/apizoo"
    click ds3 "https://github.com/shishirpatil/gorilla/tree/main/berkeley-function-call-leaderboard/data"
    click r1 "https://github.com/shishirpatil/gorilla/blob/main/gorilla/eval/retrievers/bm25.py"
    click r2 "https://github.com/shishirpatil/gorilla/blob/main/gorilla/eval/retrievers/gpt.py"
    click gorilla_prompt "https://github.com/shishirpatil/gorilla/blob/main/gorilla/eval/get_llm_responses.py"
    click inf1 "https://github.com/shishirpatil/gorilla/blob/main/gorilla/inference/serve/gorilla_cli.py"
    click inf2 "https://github.com/shishirpatil/gorilla/blob/main/openfunctions/inference_hosted.py"
    click inf3 "https://github.com/shishirpatil/gorilla/blob/main/goex/exec_engine/pipeline.py"
    click bfcl_runner "https://github.com/shishirpatil/gorilla/blob/main/berkeley-function-call-leaderboard/bfcl/__main__.py"
    click bfcl_models "https://github.com/shishirpatil/gorilla/tree/main/berkeley-function-call-leaderboard/bfcl/model_handler"
    click bfcl_checker "https://github.com/shishirpatil/gorilla/tree/main/berkeley-function-call-leaderboard/bfcl/eval_checker"
    click ui1 "https://github.com/shishirpatil/gorilla/tree/main/agent-arena/client"
    click metrics "https://github.com/shishirpatil/gorilla/blob/main/agent-arena/evalutation/agent_ratings_V0.json"

    %% Styles
    classDef data fill:#9f6,stroke:#333,stroke-width:1px;
    classDef compute fill:#6cf,stroke:#333,stroke-width:1px;
    classDef storage fill:#f96,stroke:#333,stroke-width:1px;
    classDef ui fill:#c9f,stroke:#333,stroke-width:1px;
```
