How au-clan/cachesaver works
Project type and purpose
• Cachesaver is a Python-based, open-source caching library specialized for large language model (LLM) experiments.
• Its goal is to reduce cost, improve reproducibility, and streamline debugging of batch LLM calls by providing multi-response caching, usage tracking, deduplication, and race-condition prevention.
File structure
• Top-level directories:
• src/ – core library code
– algorithms/ (FOA, GOT, RAP, React, ToT)
– models/ (API wrappers for OpenAI, Together, Groq, “online” dummy, QROQ)
– tasks/ (task definitions for game24, hle, hotpotqa, humaneval, logiqa, matharena, scibench, sonnetwriting)
– typedefs.py, utils.py (shared types and utilities)
• scripts/ – experiment drivers and pipelines
– ablations/, correctness/, frameworks/, hyper_search/, motivation/, namespace/, old/, perms/, single/
• datasets/ – compressed datasets for each task
• examples/ – Jupyter notebooks demonstrating usage
• figures/ & prompts.md – experiment artifacts and prompt templates
• tests/ – pytest suite covering models, environments, benchmarks
• Config & build: requirements.txt, pytest.ini, API_proposal.md
README for additional insights
• Describes async, batching, deduplication, multi-response per query, statistical integrity.
• Mentions an external “cachesaver-core” package dependency.
• Emphasizes extensibility (multiple LLM API plugins) and experiment reproducibility.
Main components & their relationships
a. Core library (src/)
– Cache Manager: orchestrates storing/fetching of query–response pairs, enforces deduplication and sampling policies
– Algorithms: modules (foa.py, got.py, rap.py, react.py, tot_*.py) implementing different retrieval-oriented strategies
– Task Definitions: environment, state, prompts, agents, and benchmark logic for each supported task
– Model Adapters: asynchronous API wrappers to external LLM services (OpenAI, Together, Groq, QROQ, dummy online)
– Utilities & Typedefs: shared helper functions, type definitions, exception handling
b. Experiment runner (scripts/)
– Drives a sequence: load dataset → choose task → select algorithm → invoke library → collect & dump results
– Subfolders for different experiment types (ablations, correctness checks, hyperparameter search, etc.)
c. Data storage
– datasets/: raw inputs
– local cache (configured by user or default file-based/DB backend via cachesaver-core)
– output artifacts (figures/, examples/)
d. External services
– LLM providers (via network calls)
– User’s local filesystem / optional DB for persistent caching
Architectural patterns & design principles
• Modular plugin architecture: easily add new tasks, algorithms, or LLM adapters
• Async event loop: high-throughput, non-blocking API calls and batching
• Separation of concerns: caching engine decoupled from algorithm logic and from model interfaces
• Experiment reproducibility: scripts parameterize datasets, algorithms, and sampling seeds