How future-house/llm-client works
Project Type and Purpose
• This is a Python client library (llm-client) providing a unified, asynchronous interface to multiple Large Language Model (LLM) services (e.g., OpenAI, Anthropic) and embedding models.
• It’s not a web app or microservice suite, but a reusable SDK-style package: users import llmclient, instantiate models, and call methods to get completions or embeddings.
High-Level File Structure Analysis
• Top-level directories/files:
• .github/ – CI/publishing workflows (codeflash, test, publish).
• llmclient/ – the core package (modules under constants, exceptions, types, utils, cost_tracker, rate_limiter, llms, embeddings).
• tests/ – pytest test suite with recorded HTTP interactions (cassettes).
• pyproject.toml – build & dependency configuration.
• Key architectural clues:
• Separation of concerns via modules: LLM abstraction vs. embedding abstraction vs. cross-cutting concerns (rate limiting, cost tracking).
• Abstract base classes (LLMModel, EmbeddingModel) define the interface; concrete subclasses implement provider-specific logic.
Components
a. Public API Layer
• LiteLLMModel, LLMModel (abstract), LiteLLMEmbeddingModel, SparseEmbeddingModel, HybridEmbeddingModel, SentenceTransformerEmbeddingModel
• Utilities: call, call_single, embed_documents
b. Core Infrastructure
• Cost Tracker (cost_tracker.py) – collects token usage and cost metrics, exposes GLOBAL_COST_TRACKER
• Rate Limiter (rate_limiter.py) – implements GlobalRateLimiter with in-memory or Redis backend
• Common definitions: constants.py, types.py (e.g., LLMResult), exceptions.py, utils.py
c. External Services & Dependencies
• LiteLLM API (third-party routing layer for LLMs)
• OpenAI/Anthropic/etc. endpoints (via LiteLLM or direct SDK calls inside concrete LLMModel subclasses)
• Redis (optional, for cross-process rate limiting)
• Local libraries (e.g., sentence-transformers) for SentenceTransformerEmbeddingModel
d. Test Harness (optional to include)
• VCR cassettes under tests/cassettes/ to mock HTTP calls
Relationships and Data Flow
• The client application invokes LiteLLMModel.call(...) or embed_documents(...).
• Flow through layers:
1. Public API → 2. Rate Limiter (acquire permission, using GLOBAL_LIMITER.try_acquire) → 3. Cost Tracker (records pre-call state) → 4. Concrete LLM/Embedding Model (acompletion or embed_documents) → 5. External Service (LiteLLM, OpenAI, Anthropic, local model) → 6. Response bubbles back, cost usage captured, result wrapped in LLMResult → 7. Public API returns to caller.
Architectural Patterns & Principles
• Decorator/Cross-Cutting Concerns: Rate limiting and cost tracking wrap calls without mixing concerns.
• Async/Await Concurrency: All I/O operations are async for non-blocking calls.