How eduardotrevino/fail-talms works
Project Type and Purpose
• This is a research benchmark codebase (Fail‑TaLMs) for evaluating Tool‑Augmented Language Models (TaLMs) under realistic failure scenarios. It’s not a web app but a set of Python pipelines, data artifacts, and evaluation scripts focused on:
• Generating query and tool‑environment inputs
• Running inference with various LLMs (OpenAI, Claude, LLaMA) in multiple data settings
• Storing outputs
• Computing evaluation metrics
High‑Level Components
• api_docs/: Raw API documentation per tool (text files)
• tools/: Python “api.py” endpoints + JSON schemas for each external tool + autogenerated test logs
• benchmark/: JSON query sets for 5 data settings (original, no‑tools, replaceable, non‑replaceable, underspecified)
• generation/: Scripts to build
• Query JSONs (query_generation.py)
• Tool‑environment schemas from api_docs (tool_env_generation.py)
• inference/: Model‑specific pipelines that:
• Load queries & tool schemas
• Call LLM APIs (OpenAI, Claude, LLaMA) with function‑calling
• Invoke local tool endpoints for tool calls
• Optionally use Ask‑and‑Help (AAH) “human tool”
• Emit answers.json into outputs/
• outputs/: Model outputs organized by data setting, model, and AAH flag
• evaluation/: evaluation.py loads outputs/ and computes pass rates, awareness metrics, etc.
• runners/: Shell/Batch scripts orchestrating end‑to‑end inference
Relationships & Data Flow
a. generation/tool_env_generation.py reads api_docs/ → emits JSON schemas → placed in tools/
b. generation/query_generation.py reads benchmark/ → possibly filters by data setting → emits query JSON
c. inference pipeline (e.g., pipeline_openai.py) reads:
• Query JSON
• Tool schemas (tools/*.json)
• Loads appropriate LLM client (OpenAI/Anthropic/local LLaMA)
• Sends prompt + function schema to LLM
• Receives response specifying which tool & parameters → calls the local Python endpoint (tools/*/api.py)
• Aggregates tool results + final answer → writes answers.json under outputs/{setting}/{model}[AAH]
d. runners/* simply shell out to “python pipeline_*” with flags (–data_setting, –aah, –api_key)
e. evaluation/evaluation.py traverses outputs/ → loads answers.json from every model/setting → computes metrics → prints/exports results
Architectural Patterns & Design Principles
• Modular pipelines: separate “generation → inference → evaluation” stages
• Data‑centric architecture: JSON artifacts flow through each stage
• Plugin pattern for tools: each external API has its own directory with api.py and a JSON schema → easily extendable
• Strategy pattern for models: each LLM (OpenAI, Claude, LLaMA) has its own pipeline_xxx.py implementing a common interface
• Clear separation of concerns:
• Data definition (benchmark/, api_docs/)
• Code generation (generation/)
• Execution (inference/, runners/)
• Storage (outputs/)
• Evaluation (evaluation/)