# eduardotrevino/fail-talms architecture

> Architecture diagram of the GitHub repository eduardotrevino/fail-talms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eduardotrevino/fail-talms
- Repository: https://github.com/eduardotrevino/fail-talms

## Overview

##### Project Type and Purpose
- This is a research benchmark codebase (Fail‑TaLMs) for evaluating Tool‑Augmented Language Models (TaLMs) under realistic failure scenarios. It’s not a web app but a set of Python pipelines, data artifacts, and evaluation scripts focused on:
- Generating query and tool‑environment inputs
- Running inference with various LLMs (OpenAI, Claude, LLaMA) in multiple data settings
- Storing outputs
- Computing evaluation metrics

##### High‑Level Components
- api_docs/: Raw API documentation per tool (text files)
- tools/: Python “api.py” endpoints + JSON schemas for each external tool + autogenerated test logs
- benchmark/: JSON query sets for 5 data settings (original, no‑tools, replaceable, non‑replaceable, underspecified)
- generation/: Scripts to build
- Query JSONs (query_generation.py)
- Tool‑environment schemas from api_docs (tool_env_generation.py)
- inference/: Model‑specific pipelines that:
- Load queries & tool schemas
- Call LLM APIs (OpenAI, Claude, LLaMA) with function‑calling
- Invoke local tool endpoints for tool calls
- Optionally use Ask‑and‑Help (AAH) “human tool”
- Emit answers.json into outputs/
- outputs/: Model outputs organized by data setting, model, and AAH flag
- evaluation/: evaluation.py loads outputs/ and computes pass rates, awareness metrics, etc.
- runners/: Shell/Batch scripts orchestrating end‑to‑end inference

##### Relationships & Data Flow
a. generation/tool_env_generation.py reads api_docs/ → emits JSON schemas → placed in tools/
b. generation/query_generation.py reads benchmark/ → possibly filters by data setting → emits query JSON
c. inference pipeline (e.g., pipeline_openai.py) reads:
- Query JSON
- Tool schemas (tools/*.json)
- Loads appropriate LLM client (OpenAI/Anthropic/local LLaMA)
- Sends prompt + function schema to LLM
- Receives response specifying which tool & parameters → calls the local Python endpoint (tools/*/api.py)
- Aggregates tool results + final answer → writes answers.json under outputs/{setting}/{model}[AAH]
d. runners/* simply shell out to “python pipeline_*” with flags (–data_setting, –aah, –api_key)
e. evaluation/evaluation.py traverses outputs/ → loads answers.json from every model/setting → computes metrics → prints/exports results

##### Architectural Patterns & Design Principles
- Modular pipelines: separate “generation → inference → evaluation” stages
- Data‑centric architecture: JSON artifacts flow through each stage
- Plugin pattern for tools: each external API has its own directory with api.py and a JSON schema → easily extendable
- Strategy pattern for models: each LLM (OpenAI, Claude, LLaMA) has its own pipeline_xxx.py implementing a common interface
- Clear separation of concerns:
- Data definition (benchmark/, api_docs/)
- Code generation (generation/)
- Execution (inference/, runners/)
- Storage (outputs/)
- Evaluation (evaluation/)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        APIDOCS["api_docs/"]:::data
        BENCHMARK["benchmark/"]:::data
    end

    %% Generation Layer
    subgraph "Generation" 
        direction TB
        QGEN["query_generation.py"]:::code
        TGEN["tool_env_generation.py"]:::code
    end

    %% Tool Registry
    subgraph "Tool Registry"
        direction TB
        TOOLS["tools/"]:::data
    end

    %% Inference Layer
    subgraph "Inference"
        direction TB
        PIPE_OPENAI["pipeline_openai.py"]:::code
        PIPE_CLAUDE["pipeline_claude-3-5-sonnet.py"]:::code
        PIPE_LLAMA["pipeline_llama.py"]:::code
        PIPE_OPENAI_AAH["pipeline_openai_human_eval.py"]:::code
        subgraph "External LLMs & Human"
            direction TB
            OPENAI_API["OpenAI API"]:::external
            CLAUDE_API["Claude API"]:::external
            LLAMA_LOCAL["Local LLaMA"]:::external
            HUMAN_TOOL["Human Tool"]:::external
        end
    end

    %% Outputs Store
    OUTPUTS["outputs/"]:::data

    %% Evaluation Module
    EVAL["evaluation.py"]:::code

    %% Orchestration Scripts
    subgraph "Runners"
        direction TB
        SH["inference.sh"]:::code
        BAT["inference.bat"]:::code
    end

    %% Connections
    APIDOCS -->|reads| TGEN
    TGEN -->|writes schemas| TOOLS
    BENCHMARK -->|reads| QGEN
    QGEN -->|writes queries| TOOLS
    QGEN -->|writes queries| PIPE_OPENAI
    QGEN -->|writes queries| PIPE_CLAUDE
    QGEN -->|writes queries| PIPE_LLAMA
    QGEN -->|writes queries| PIPE_OPENAI_AAH

    TOOLS -->|reads schemas| PIPE_OPENAI
    TOOLS -->|reads schemas| PIPE_CLAUDE
    TOOLS -->|reads schemas| PIPE_LLAMA
    TOOLS -->|reads schemas| PIPE_OPENAI_AAH

    PIPE_OPENAI -->|calls| OPENAI_API
    PIPE_CLAUDE -->|calls| CLAUDE_API
    PIPE_LLAMA -->|calls| LLAMA_LOCAL
    PIPE_OPENAI_AAH -->|calls| OPENAI_API
    PIPE_OPENAI_AAH -->|optionally calls| HUMAN_TOOL

    PIPE_OPENAI -->|writes answers.json| OUTPUTS
    PIPE_CLAUDE -->|writes answers.json| OUTPUTS
    PIPE_LLAMA -->|writes answers.json| OUTPUTS
    PIPE_OPENAI_AAH -->|writes answers.json| OUTPUTS

    OUTPUTS -->|reads answers| EVAL
    EVAL -->|computes metrics| OUTPUTS

    SH -->|executes| PIPE_OPENAI
    SH -->|executes| PIPE_CLAUDE
    SH -->|executes| PIPE_LLAMA
    SH -->|executes| PIPE_OPENAI_AAH

    BAT -->|executes| PIPE_OPENAI
    BAT -->|executes| PIPE_CLAUDE
    BAT -->|executes| PIPE_LLAMA
    BAT -->|executes| PIPE_OPENAI_AAH

    %% Click Events
    click APIDOCS "https://github.com/eduardotrevino/fail-talms/tree/main/api_docs/"
    click BENCHMARK "https://github.com/eduardotrevino/fail-talms/tree/main/benchmark/"
    click QGEN "https://github.com/eduardotrevino/fail-talms/blob/main/generation/query_generation.py"
    click TGEN "https://github.com/eduardotrevino/fail-talms/blob/main/generation/tool_env_generation.py"
    click TOOLS "https://github.com/eduardotrevino/fail-talms/tree/main/tools/"
    click PIPE_OPENAI "https://github.com/eduardotrevino/fail-talms/blob/main/inference/pipeline_openai.py"
    click PIPE_CLAUDE "https://github.com/eduardotrevino/fail-talms/blob/main/inference/pipeline_claude-3-5-sonnet.py"
    click PIPE_LLAMA "https://github.com/eduardotrevino/fail-talms/blob/main/inference/pipeline_llama.py"
    click PIPE_OPENAI_AAH "https://github.com/eduardotrevino/fail-talms/blob/main/inference/pipeline_openai_human_eval.py"
    click HUMAN_TOOL "https://github.com/eduardotrevino/fail-talms/blob/main/tools/HumanInteract/humaninteract/api.py"
    click OUTPUTS "https://github.com/eduardotrevino/fail-talms/tree/main/outputs/"
    click EVAL "https://github.com/eduardotrevino/fail-talms/blob/main/evaluation/evaluation.py"
    click SH "https://github.com/eduardotrevino/fail-talms/blob/main/runners/inference.sh"
    click BAT "https://github.com/eduardotrevino/fail-talms/blob/main/runners/inference.bat"

    %% Styles
    classDef code fill:#bbdefb,stroke:#4285f4,color:#000
    classDef data fill:#c8e6c9,stroke:#2e7d32,color:#000
    classDef external fill:#ffcc80,stroke:#ef6c00,color:#000
```
