# future-house/llm-client architecture

> Architecture diagram of the GitHub repository future-house/llm-client, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/future-house/llm-client
- Repository: https://github.com/future-house/llm-client

## Overview

##### Project Type and Purpose
- This is a Python client library (`llm-client`) providing a unified, asynchronous interface to multiple Large Language Model (LLM) services (e.g., OpenAI, Anthropic) and embedding models.
- It’s not a web app or microservice suite, but a reusable SDK-style package: users import `llmclient`, instantiate models, and call methods to get completions or embeddings.

##### High-Level File Structure Analysis
- Top-level directories/files:
- `.github/` – CI/publishing workflows (codeflash, test, publish).
- `llmclient/` – the core package (modules under constants, exceptions, types, utils, cost_tracker, rate_limiter, llms, embeddings).
- `tests/` – pytest test suite with recorded HTTP interactions (cassettes).
- `pyproject.toml` – build & dependency configuration.
- Key architectural clues:
- Separation of concerns via modules: LLM abstraction vs. embedding abstraction vs. cross-cutting concerns (rate limiting, cost tracking).
- Abstract base classes (`LLMModel`, `EmbeddingModel`) define the interface; concrete subclasses implement provider-specific logic.

##### Components
a. Public API Layer
- `LiteLLMModel`, `LLMModel` (abstract), `LiteLLMEmbeddingModel`, `SparseEmbeddingModel`, `HybridEmbeddingModel`, `SentenceTransformerEmbeddingModel`
- Utilities: `call`, `call_single`, `embed_documents`
b. Core Infrastructure
- Cost Tracker (`cost_tracker.py`) – collects token usage and cost metrics, exposes `GLOBAL_COST_TRACKER`
- Rate Limiter (`rate_limiter.py`) – implements `GlobalRateLimiter` with in-memory or Redis backend
- Common definitions: `constants.py`, `types.py` (e.g., `LLMResult`), `exceptions.py`, `utils.py`
c. External Services & Dependencies
- LiteLLM API (third-party routing layer for LLMs)
- OpenAI/Anthropic/etc. endpoints (via LiteLLM or direct SDK calls inside concrete LLMModel subclasses)
- Redis (optional, for cross-process rate limiting)
- Local libraries (e.g., `sentence-transformers`) for `SentenceTransformerEmbeddingModel`
d. Test Harness (optional to include)
- VCR cassettes under `tests/cassettes/` to mock HTTP calls

##### Relationships and Data Flow
- The client application invokes `LiteLLMModel.call(...)` or `embed_documents(...)`.
- Flow through layers:
1. **Public API** → 2. **Rate Limiter** (acquire permission, using `GLOBAL_LIMITER.try_acquire`) → 3. **Cost Tracker** (records pre-call state) → 4. **Concrete LLM/Embedding Model** (`acompletion` or `embed_documents`) → 5. **External Service** (LiteLLM, OpenAI, Anthropic, local model) → 6. **Response** bubbles back, cost usage captured, result wrapped in `LLMResult` → 7. **Public API** returns to caller.

##### Architectural Patterns & Principles
- **Decorator/Cross-Cutting Concerns**: Rate limiting and cost tracking wrap calls without mixing concerns.
- **Async/Await Concurrency**: All I/O operations are `async` for non-blocking calls.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client
    ClientApp["Client Application"]:::external

    %% Public API Layer
    subgraph "Public API" 
        direction TB
        LiteLLMModel["LiteLLMModel"]:::public
        LLMModel["LLMModel (abstract)"]:::public
        LiteLLMEmbeddingModel["LiteLLMEmbeddingModel"]:::public
        SparseEmbeddingModel["SparseEmbeddingModel"]:::public
        HybridEmbeddingModel["HybridEmbeddingModel"]:::public
        SentenceTransformerEmbeddingModel["SentenceTransformerEmbeddingModel"]:::public
        CallUtil["call / call_single"]:::public
        EmbedUtil["embed_documents"]:::public
    end

    %% Core Infrastructure
    subgraph "Core Infrastructure"
        direction TB
        RateLimiter["GlobalRateLimiter"]:::infra
        InMemoryStore["In-Memory Store"]:::infra
        RedisStore["Redis Store (optional)"]:::infra
        CostTracker["GLOBAL_COST_TRACKER"]:::infra
        Constants["constants.py"]:::infra
        Types["types.py"]:::infra
        Exceptions["exceptions.py"]:::infra
    end

    %% External Services
    subgraph "External Services"
        direction TB
        LiteLLMAPI["LiteLLM API"]:::external
        OpenAI["OpenAI/Anthropic Endpoints"]:::external
        LocalST["Sentence-Transformers Local Engine"]:::external
        RedisExt["Redis"]:::external
    end

    %% Test Harness
    subgraph "Test Harness"
        direction TB
        VCRCassettes["VCR Cassettes"]:::test
    end

    %% Flows: LLM
    ClientApp -->|"call()"| LiteLLMModel
    LiteLLMModel -->|"try_acquire() async"| RateLimiter
    RateLimiter -->|uses| InMemoryStore
    RateLimiter -.->|optional| RedisStore
    RateLimiter -->|"record pre-call"| CostTracker
    LiteLLMModel -->|"acompletion() async"| LLMModel
    LLMModel -->|"invoke provider"| LiteLLMAPI
    LiteLLMAPI -->|"response"| LLMModel
    LLMModel -->|"wrap in LLMResult"| CostTracker
    CostTracker -->|"record post-call"| LiteLLMModel
    LiteLLMModel -->|returns| ClientApp

    %% Flows: Embeddings
    ClientApp -->|"embed_documents()"| LiteLLMEmbeddingModel
    LiteLLMEmbeddingModel -->|"try_acquire() async"| RateLimiter
    LiteLLMEmbeddingModel -->|"record pre-call"| CostTracker
    LiteLLMEmbeddingModel -->|"embed_documents() async"| SparseEmbeddingModel
    LiteLLMEmbeddingModel -->|"embed_documents() async"| HybridEmbeddingModel
    LiteLLMEmbeddingModel -->|"embed_documents() async"| SentenceTransformerEmbeddingModel
    SparseEmbeddingModel -->|"invoke provider"| LiteLLMAPI
    HybridEmbeddingModel -->|"invoke provider"| LiteLLMAPI
    SentenceTransformerEmbeddingModel -->|"invoke local"| LocalST
    SparseEmbeddingModel -->|"response"| LiteLLMEmbeddingModel
    HybridEmbeddingModel -->|"response"| LiteLLMEmbeddingModel
    SentenceTransformerEmbeddingModel -->|"response"| LiteLLMEmbeddingModel
    LiteLLMEmbeddingModel -->|"wrap in LLMResult"| CostTracker
    CostTracker -->|"record post-call"| LiteLLMEmbeddingModel
    LiteLLMEmbeddingModel -->|returns| ClientApp

    %% Utilities
    ClientApp --> CallUtil
    ClientApp --> EmbedUtil

    %% Shared definitions
    ClientApp --> Constants
    ClientApp --> Types
    ClientApp --> Exceptions

    %% Test harness mocking
    LiteLLMAPI -->|mocked by| VCRCassettes
    OpenAI -->|mocked by| VCRCassettes

    %% Click Events
    click LiteLLMModel "https://github.com/future-house/llm-client/blob/main/llmclient/llms.py"
    click LLMModel "https://github.com/future-house/llm-client/blob/main/llmclient/llms.py"
    click LiteLLMEmbeddingModel "https://github.com/future-house/llm-client/blob/main/llmclient/embeddings.py"
    click SparseEmbeddingModel "https://github.com/future-house/llm-client/blob/main/llmclient/embeddings.py"
    click HybridEmbeddingModel "https://github.com/future-house/llm-client/blob/main/llmclient/embeddings.py"
    click SentenceTransformerEmbeddingModel "https://github.com/future-house/llm-client/blob/main/llmclient/embeddings.py"
    click CallUtil "https://github.com/future-house/llm-client/blob/main/llmclient/utils.py"
    click EmbedUtil "https://github.com/future-house/llm-client/blob/main/llmclient/utils.py"
    click CostTracker "https://github.com/future-house/llm-client/blob/main/llmclient/cost_tracker.py"
    click RateLimiter "https://github.com/future-house/llm-client/blob/main/llmclient/rate_limiter.py"
    click Constants "https://github.com/future-house/llm-client/blob/main/llmclient/constants.py"
    click Types "https://github.com/future-house/llm-client/blob/main/llmclient/types.py"
    click Exceptions "https://github.com/future-house/llm-client/blob/main/llmclient/exceptions.py"
    click VCRCassettes "https://github.com/future-house/llm-client/tree/main/tests/cassettes/"

    %% Styles
    classDef external fill:#ffe4b5,stroke:#d2691e,color:#000
    classDef public fill:#c6f6d5,stroke:#2f855a,color:#000
    classDef infra fill:#bee3f8,stroke:#2b6cb0,color:#000
    classDef test fill:#faf089,stroke:#d69e2e,color:#000
```
