# tensorzero/tensorzero architecture

> Architecture diagram of the GitHub repository tensorzero/tensorzero, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tensorzero/tensorzero
- Repository: https://github.com/tensorzero/tensorzero

## Overview

##### Main Components

a. Gateway Layer (Core Component)
- Written in Rust for high performance
- Handles API interfaces for multiple LLM providers
- Manages structured schema-based inference
- Implements experimentation and routing logic

b. Provider Integration Layer
- Multiple LLM provider connections (OpenAI, Anthropic, AWS Bedrock, etc.)
- Fallback mechanisms
- Load balancing and routing

c. Observability & Data Collection Layer
- Metrics collection
- Feedback processing
- Performance monitoring
- ClickHouse integration for data warehousing

d. Optimization Layer
- Recipe execution engine
- Fine-tuning workflows
- Dynamic In-Context Learning (DICL)
- Experimentation framework

e. Client Interface Layer
- Python SDK
- REST API endpoints
- WebSocket connections for streaming
- Dashboard UI

##### Component Relationships

- Client requests → Gateway → Provider Integration → LLM Providers
- Gateway → Observability Layer → ClickHouse
- ClickHouse → Optimization Layer → Gateway (feedback loop)
- Dashboard → Gateway (configuration and monitoring)

##### Specific Implementation Details

Technical annotations for:
- Rust-based gateway implementation
- <1ms P99 latency overhead
- GitOps orchestration
- Schema-based validation
- Real-time analytics capabilities

##### Key Architectural Patterns

Highlight:
- Microservices architecture (gateway, dashboard, optimization services)
- Event-driven feedback collection
- Pipeline pattern for optimization workflows
- Circuit breaker pattern for provider fallbacks

##### Detailed Component Breakdown

Gateway:
**gateway/**
- src/
- inference/
- providers/
- endpoints/
- clickhouse/

Dashboard:
**dashboard/**
- app/
- components/
- routes/
- utils/

##### Additional Elements

Include:
- Configuration management
- Security boundaries
- Scaling components
- Monitoring points
- Data flow directions

##### Improvements are deployed back to production

This system design should clearly illustrate TensorZero's unique value proposition as a learning flywheel for LLMs, emphasizing both its high-performance characteristics and its ability to continuously improve LLM applications through real-world usage data.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Client["Client Interface Layer"]
        PythonSDK["Python SDK"]:::client
        OpenAISDK["OpenAI SDK"]:::client
        Dashboard["Dashboard UI"]:::client
        REST["REST API"]:::client
        WS["WebSocket"]:::client
    end

    subgraph Gateway["Gateway Layer"]
        APIHandler["API Handler"]:::gateway
        SchemaValidator["Schema Validator"]:::gateway
        Router["Router & Load Balancer"]:::gateway
        FallbackMech["Fallback Mechanism"]:::gateway
    end

    subgraph Providers["Provider Integration Layer"]
        OpenAI["OpenAI"]:::provider
        Anthropic["Anthropic"]:::provider
        Bedrock["AWS Bedrock"]:::provider
        OtherLLMs["Other LLMs"]:::provider
    end

    subgraph Observability["Observability & Data Collection"]
        Metrics["Metrics Collector"]:::observability
        FeedbackProc["Feedback Processor"]:::observability
        PerfMon["Performance Monitor"]:::observability
        ClickHouse[(ClickHouse DB)]:::storage
    end

    subgraph Optimization["Optimization Layer"]
        RecipeEngine["Recipe Engine"]:::optimization
        FineTuning["Fine-tuning Workflows"]:::optimization
        DICL["Dynamic ICL"]:::optimization
        ExpFramework["Experimentation Framework"]:::optimization
    end

    %% Connections
    Client --> Gateway
    Gateway --> Providers
    Gateway --> Observability
    Observability --> ClickHouse
    ClickHouse --> Optimization
    Optimization --> Gateway

    %% Click Events
    click PythonSDK "https://github.com/tensorzero/tensorzero/tree/main/clients/python/"
    click OpenAISDK "https://github.com/tensorzero/tensorzero/tree/main/clients/openai-python/"
    click Dashboard "https://github.com/tensorzero/tensorzero/tree/main/dashboard/"
    click APIHandler "https://github.com/tensorzero/tensorzero/tree/main/gateway/src/endpoints/"
    click SchemaValidator "https://github.com/tensorzero/tensorzero/blob/main/gateway/src/jsonschema_util.rs"
    click Router "https://github.com/tensorzero/tensorzero/blob/main/gateway/src/lib.rs"
    click OpenAI "https://github.com/tensorzero/tensorzero/blob/main/gateway/src/inference/providers/openai.rs"
    click Anthropic "https://github.com/tensorzero/tensorzero/blob/main/gateway/src/inference/providers/anthropic.rs"
    click Bedrock "https://github.com/tensorzero/tensorzero/blob/main/gateway/src/inference/providers/aws_bedrock.rs"
    click ClickHouse "https://github.com/tensorzero/tensorzero/blob/main/gateway/src/clickhouse.rs"
    click Metrics "https://github.com/tensorzero/tensorzero/blob/main/gateway/src/observability.rs"
    click DICL "https://github.com/tensorzero/tensorzero/blob/main/gateway/src/variant/dicl.rs"
    click FineTuning "https://github.com/tensorzero/tensorzero/tree/main/recipes/supervised_fine_tuning/"
    click ExpFramework "https://github.com/tensorzero/tensorzero/blob/main/gateway/src/variant/best_of_n_sampling.rs"

    %% Styling
    classDef client fill:#9370DB,stroke:#333,stroke-width:2px
    classDef gateway fill:#4169E1,stroke:#333,stroke-width:2px
    classDef provider fill:#3CB371,stroke:#333,stroke-width:2px
    classDef observability fill:#FFA500,stroke:#333,stroke-width:2px
    classDef optimization fill:#DC143C,stroke:#333,stroke-width:2px
    classDef storage fill:#DEB887,stroke:#333,stroke-width:2px

    %% Legend
    subgraph Legend
        Client_L[Client Interface]:::client
        Gateway_L[Gateway Components]:::gateway
        Provider_L[LLM Providers]:::provider
        Obs_L[Observability]:::observability
        Opt_L[Optimization]:::optimization
        Storage_L[Storage]:::storage
    end
```
