# tensoropsai/llmstudio architecture

> Architecture diagram of the GitHub repository tensoropsai/llmstudio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tensoropsai/llmstudio
- Repository: https://github.com/tensoropsai/llmstudio

## Overview

##### Main Components
- Core Engine (LLMstudio Core): Located under libs/core, this module contains the essential logic for interacting with various LLM providers. It handles configurations, error management, and provider-specific integrations.
- Proxy Service (LLMstudio Proxy): Found under libs/proxy, this part manages the proxy access functionality, enabling seamless communications between external API calls and internal processing. It supports smart routing, fallback strategies, and potentially advanced task management.
- Tracker Service (LLMstudio Tracker): Under libs/tracker, this component is responsible for monitoring, logging, tracking usage, and storing data related to prompts and sessions, likely backed by a database as shown by the “database.py” and related files.
- CLI and Server Interfaces: Each module (from the proxy, tracker, and core) offers CLI commands and server capabilities (with endpoints reachable via Swagger as described in the README for both proxy and tracker). These help in starting the services and managing the system.
- SDK and Integration Layer: The project includes a Python SDK (documented in docs/sdk) that allows consumers to integrate LLMstudio with their own workflows. This also involves LangChain integration for extended functionality.

##### Relationships and Interactions
- The User Interface (Prompt Playground UI) and Python SDK are the entry points for users and developers. They interact directly with the Core Engine to process prompts.
- The Core Engine dispatches requests to external LLM providers (OpenAI, Anthropic, custom models like Ollama, etc.).
- The Proxy Service sits between the client/UI and the LLM providers, handling request routing, batching, and smart routing/fallback mechanisms.
- The Tracker Service monitors all the LLM requests/responses by intercepting the calls at various stages (or by being triggered as part of the workflow) to log metrics and session details.
- There is clear communication among these components: the CLI commands and REST API endpoints provided by each service allow for configuration, initiation, and runtime monitoring.

##### Architectural Patterns and Technologies
- Modular Microservices (or modular monolith) architecture: Although all components reside in one codebase, the clear separation into “core”, “proxy”, and “tracker” (each with their own submodules and tests) suggests loose coupling.
- API-first design: The usage of Swagger endpoints (e.g., for proxy and tracker) indicates a RESTful API design.
- Extensibility: With support for custom providers, LangChain integration, and batch calling, the design is extensible and plug-in oriented.
- CI/CD and Deployment: The presence of GitHub workflows, docker-compose, and Dockerfiles in the deploy/ directory shows that automated testing, linting, and deployment are integral to the architecture.

## Diagram (Mermaid)

```mermaid
graph TD
    %% User & Integration Layer
    subgraph "User & Integration"
        UI["Prompt Playground UI"]:::ui
        CLI1["General CLI"]:::cli
        CLI2["Proxy CLI"]:::cli
        CLI3["Tracker CLI"]:::cli
        SDK["Python SDK / LangChain"]:::sdk
    end

    %% Backend Services
    subgraph "Backend Services"
        CORE["LLMstudio Core Service"]:::core
        PROXY["LLMstudio Proxy Service"]:::proxy
        TRACKER["LLMstudio Tracker Service"]:::tracker
        DB["Database (Tracker)"]:::db
    end

    %% External Providers
    subgraph "External Providers"
        EXT["External LLM Providers"]:::external
    end

    %% Deployment & CI/CD
    subgraph "Deployment & CI/CD"
        DEPLOY["Deployment (docker-compose, Dockerfiles)"]:::deploy
        CICD["CI/CD (.github/workflows)"]:::deploy
    end

    %% Interactions from User & Integration to Backend
    UI -->|"submit"| CORE
    CLI1 -->|"start"| CORE
    CLI2 -->|"start"| PROXY
    CLI3 -->|"start"| TRACKER
    SDK -->|"integration"| CORE

    %% Internal Service Communication
    CORE -->|"route"| PROXY
    CORE -->|"log"| TRACKER
    PROXY -->|"log"| TRACKER
    PROXY -->|"LLM_request"| EXT
    EXT -->|"LLM_response"| PROXY
    TRACKER -->|"store_logs"| DB

    %% Deployment & CI/CD Interactions
    CICD -->|"trigger"| DEPLOY
    CICD -->|"deploy"| CORE
    CICD -->|"deploy"| PROXY
    CICD -->|"deploy"| TRACKER
    DEPLOY -->|"containerize"| CORE
    DEPLOY -->|"containerize"| PROXY
    DEPLOY -->|"containerize"| TRACKER

    %% Click Events for Component Mapping
    click CORE "https://github.com/tensoropsai/llmstudio/blob/main/libs/core/llmstudio_core/__init__.py"
    click PROXY "https://github.com/tensoropsai/llmstudio/blob/main/libs/proxy/llmstudio_proxy/__init__.py"
    click TRACKER "https://github.com/tensoropsai/llmstudio/blob/main/libs/tracker/llmstudio_tracker/__init__.py"
    click CLI1 "https://github.com/tensoropsai/llmstudio/blob/main/libs/llmstudio/llmstudio/cli.py"
    click CLI2 "https://github.com/tensoropsai/llmstudio/blob/main/libs/proxy/llmstudio_proxy/cli.py"
    click CLI3 "https://github.com/tensoropsai/llmstudio/blob/main/libs/tracker/llmstudio_tracker/cli.py"
    click SDK "https://github.com/tensoropsai/llmstudio/tree/main/docs/sdk/llm"
    click DEPLOY "https://github.com/tensoropsai/llmstudio/blob/main/deploy/proxy.Dockerfile"
    click CICD "https://github.com/tensoropsai/llmstudio/blob/main/.github/workflows/lint.yml"

    %% Styles
    classDef ui fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef cli fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef sdk fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef core fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef proxy fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef tracker fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef db fill:#ffffff,stroke:#6c757d,stroke-width:2px;
    classDef external fill:#d6d8d9,stroke:#343a40,stroke-width:2px;
    classDef deploy fill:#eaf2ff,stroke:#007bff,stroke-width:2px;
```
