# stanford-futuredata/frugalgpt architecture

> Architecture diagram of the GitHub repository stanford-futuredata/frugalgpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stanford-futuredata/frugalgpt
- Repository: https://github.com/stanford-futuredata/frugalgpt

## Overview

##### Project Type & Purpose
- Type: A Python-based ML framework/library for building cost-aware LLM applications.
- Purpose: Provide modular components (dataloader, LLM chains, caching, cascade, optimization, scoring) plus a “ModelService” abstraction to manage calls to external LLM APIs, all aimed at reducing cost and improving performance.

##### File Structure
Top‐level directories reveal separation of concerns:
- src/FrugalGPT – Core library modules (dataloader, optimize, scoring, multiple LLM chain implementations, utilities).
- src/service – ModelService interface, utilities, and tests that wrap external LLM API calls and caching.
- config/ – Prompts and service-metadata (serviceidmap.json, serviceinfo.json).
- data/ – Dataset CSVs for COQA, HEADLINES, etc.
- examples/, analysis/, intro.ipynb – Jupyter notebooks demonstrating generation, evaluation, and analysis workflows.
- figures/, summary/ – Generated output and summary CSVs for reporting results.
- setup.py – Packaging entry point.

##### Extract Insights from README
- External dependencies: various LLM APIs (OpenAI, Claude, Gemini, Llama, etc.), pandas, sqlite (for cache).

##### Main Components & Their Interactions
a. Core Library (src/FrugalGPT)
– Dataloader
– LLM Chain modules (vanilla, cache, cascade, chain)
– Optimizer & Scoring
– Utilities
b. ModelService (src/service)
– API adapters for each LLM provider
– Local sqlite cache
c. Configuration & Metadata (config/)
– Prompt templates
– serviceinfo/serviceidmap
d. Data Storage (data/)
– Raw datasets (CSV)
e. Notebooks & Example Workflows (examples/, analysis/, intro.ipynb)

1. Data Storage → Core Library (Dataloader)
2. Core Library → ModelService
3. ModelService ↔ External LLM APIs (and Cache)
4. ModelService → Core Library (returns generations)
5. Core Library (Optimizer/Scoring) → Results
6. Results → Visualization & Notebooks → Figures/Summaries

##### Architectural Patterns & Principles
- Modular Layers: clear separation of core logic vs. service adapters vs. visualization.
- Strategy Pattern: different chain implementations plugged into a common interface.
- Decorator/Cache Pattern: llmcache wraps calls to avoid redundant requests.
- Pipeline Pattern: dataloader → chain → optimizer → scorer.
- Plugin-style Service Abstraction: one unified ModelService interface for multiple LLM backends.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Storage
    DATA["Data Storage (CSV)"]:::datastore
    click DATA "https://github.com/stanford-futuredata/frugalgpt/tree/main/data/COQA/"
    %% Assume click for directory covers all CSV dirs
    %% Configuration
    PROMPT["Prompt Templates"]:::config
    click PROMPT "https://github.com/stanford-futuredata/frugalgpt/tree/main/config/prompt/"
    SMETA["Service Metadata"]:::config
    click SMETA "https://github.com/stanford-futuredata/frugalgpt/blob/main/config/serviceidmap.json"
    click SMETA "https://github.com/stanford-futuredata/frugalgpt/blob/main/config/serviceinfo.json"

    %% Core Library Modules
    subgraph "Core Library"
        FRUGALGPT["Core API / Orchestrator"]:::module
        click FRUGALGPT "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/FrugalGPT/frugalgpt.py"
        DATALOADER["Dataloader (pandas)"]:::module
        click DATALOADER "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/FrugalGPT/dataloader.py"
        EVALUATE["Evaluation Module"]:::module
        click EVALUATE "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/FrugalGPT/evaluate.py"
        LVANILLA["Vanilla LLM Chain"]:::module
        click LVANILLA "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/FrugalGPT/llmvanilla.py"
        LCACHE["Cached LLM Chain"]:::module
        click LCACHE "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/FrugalGPT/llmcache.py"
        LCASCADE["Cascade LLM Chain"]:::module
        click LCASCADE "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/FrugalGPT/llmcascade.py"
        LCHAIN["Chained LLM Pipeline"]:::module
        click LCHAIN "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/FrugalGPT/llmchain.py"
        OPTIM["Optimizer"]:::module
        click OPTIM "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/FrugalGPT/optimizer.py"
        SCORE["Scoring"]:::module
        click SCORE "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/FrugalGPT/scoring.py"
        UTILS["Core Utilities"]:::module
        click UTILS "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/FrugalGPT/utils.py"
    end

    %% Service Layer
    subgraph "Service Layer"
        MS["ModelService Interface & Adapters"]:::service
        click MS "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/service/modelservice.py"
        SUTIL["Service Layer Utilities"]:::module
        click SUTIL "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/service/utils.py"
        CACHE["Cache (sqlite3)"]:::datastore
        EXT["External LLM APIs"]:::external
    end

    %% Results and Visualization
    RESULTS["Results"]:::module
    subgraph "Visualization & Examples"
        EX1["Example Workflows (Notebooks)"]:::notebook
        click EX1 "https://github.com/stanford-futuredata/frugalgpt/tree/main/examples/"
        EX2["Generation Analysis Notebook"]:::notebook
        click EX2 "https://github.com/stanford-futuredata/frugalgpt/blob/main/analysis/FrugalGPT_GenerationAnalysis.ipynb"
        EX3["Intro Notebook"]:::notebook
        click EX3 "https://github.com/stanford-futuredata/frugalgpt/blob/main/intro.ipynb"
        VIS["Visualization Notebook"]:::notebook
        click VIS "https://github.com/stanford-futuredata/frugalgpt/blob/main/src/visualize/visualize_tradeoffs_newmodels.ipynb"
        FIG["Output Figures"]:::output
        click FIG "https://github.com/stanford-futuredata/frugalgpt/tree/main/figures/"
        SUM["Summary Results"]:::output
        click SUM "https://github.com/stanford-futuredata/frugalgpt/tree/main/summary/"
    end

    %% Data and Config flows
    DATA --> DATALOADER
    PROMPT --> DATALOADER
    PROMPT --> FRUGALGPT
    SMETA --> MS

    %% Core workflow
    DATALOADER --> FRUGALGPT
    FRUGALGPT --> LVANILLA
    FRUGALGPT --> LCACHE
    FRUGALGPT --> LCASCADE
    FRUGALGPT --> LCHAIN
    FRUGALGPT --> MS
    MS --> CACHE
    MS --> EXT
    CACHE --> MS
    EXT --> MS
    MS --> FRUGALGPT
    FRUGALGPT --> EVALUATE
    EVALUATE --> OPTIM
    OPTIM --> SCORE
    SCORE --> RESULTS

    %% Results to Visualization
    RESULTS --> EX1
    RESULTS --> EX2
    RESULTS --> EX3
    RESULTS --> VIS
    VIS --> FIG
    VIS --> SUM

    %% Styles
    classDef datastore fill:#f9f,stroke:#333,stroke-width:1px;
    classDef module fill:#bbf,stroke:#333,stroke-width:1px;
    classDef service fill:#bfb,stroke:#333,stroke-width:1px;
    classDef external fill:#fbf,stroke:#333,stroke-width:1px;
    classDef notebook fill:#ffb,stroke:#333,stroke-width:1px;
    classDef output fill:#bff,stroke:#333,stroke-width:1px;
    classDef config fill:#ffd,stroke:#333,stroke-width:1px;
```
