# onlythompson/llm-powered-microservice-template architecture

> Architecture diagram of the GitHub repository onlythompson/llm-powered-microservice-template, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/onlythompson/llm-powered-microservice-template
- Repository: https://github.com/onlythompson/llm-powered-microservice-template

## Overview

##### Project type and purpose
- This is a backend‐only microservice template built on FastAPI, following Clean Architecture principles, to provide LLM-powered endpoints (e.g., generate, summarize). It orchestrates calls to multiple LLM providers, manages prompt templates, caches responses, and exposes HTTP APIs for consumption.

##### File structure
- Top‐level directories:
- src/presentation – FastAPI routes & schemas (API layer)
- src/application – Business logic: chains, models, prompt management, orchestrator & usage tracking
- src/domain – Core entities (LLMRequest, LLMResponse)
- src/infrastructure – Adapters: cache (Redis), LLM providers (OpenAI, Anthropic)
- src/core – Configuration, dependency injection, cross-cutting concerns (logging, tracing)
- Supporting directories: tests, docs (API, architecture, deployment, security), k8s (manifests), scripts, config files

##### README for additional insights
- Confirms Clean Architecture layering, async processing, support for multiple LLM providers, caching, DI, distributed tracing, Docker/K8s deployment.

a. Main components to include:
- Client (e.g., HTTP client or UI)
- API Gateway / FastAPI Server (presentation layer)
- Router & Schema layer (llm_routes.py + Pydantic schemas)
- Application Layer
– LLMOrchestrator service
– Chains (BaseChain + specific_chains)
– ModelFactory & BaseModel
– PromptRepository & PromptTemplate
– UsageTracker
- Domain Entities (LLMRequest, LLMResponse)
- Infrastructure Layer
– RedisCache adapter
– LLMProviders (OpenAI, Anthropic, BaseLLMProvider interface)
- Core / Cross-cutting
– Config (env vars, token limits)
– DI Container (dependencies.py)
– Logging & Tracing (OpenTelemetry / Jaeger)
- External Services
– Redis
– LLM Provider Endpoints (OpenAI API, Anthropic API)
– Jaeger/Tracing Collector
- Deployment Artifacts
– Docker Container
– Kubernetes Pod (k8s/llm_deployment.yaml)

b. Relationships and interactions:
1. Client → FastAPI HTTP endpoint (/api/llm/generate)
2. FastAPI Router resolves schema → calls LLMOrchestrator (injected with ModelFactory, PromptRepository, RedisCache)
3. LLMOrchestrator orchestrates:
– Check RedisCache for a cached response
– If cache miss, retrieve prompt template from PromptRepository
– Format prompt via PromptTemplate
– Select provider via ModelFactory → calls provider’s API adapter (OpenAIClient or AnthropicClient)
– Receive LLMResponse
– Store response in RedisCache
– Track usage via UsageTracker
4. Propagate response back up to presentation layer → HTTP response to client
5. Throughout the request, cross-cutting logging/tracing intercepts calls

c. Architectural patterns / design principles:
- Clean Architecture (separate layers: presentation, application, domain, infrastructure, core)
- Dependency Injection via core/dependencies.py
- Factory Pattern for models/providers
- Strategy Pattern for selecting LLM providers
- Adapter Pattern for cache & external APIs
- Asynchronous processing for I/O bound calls

d. Key technologies / frameworks / libraries:
- FastAPI (HTTP framework)
- Pydantic (request/response schemas)
- Redis (caching layer)
- OpenAI / Anthropic client libraries
- Python AsyncIO
- OpenTelemetry / Jaeger for tracing
- Docker & Kubernetes for containerization & orchestration

##### Essential elements
- Five Clean Architecture layers and the external dependencies.
- How DI injects configurations and adapters into the orchestrator.
- Caching decision points and provider selection.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client
    Client["Client"]:::external

    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        LLMRoutes["llm_routes.py"]:::presentation
        LLMSchemaReq["llm_request.py"]:::presentation
        LLMSchemaRes["llm_response.py"]:::presentation
    end

    %% Application Layer
    subgraph "Application Layer"
        direction TB
        LLMOrch["LLMOrchestrator"]:::application
        UsageTracker["UsageTracker"]:::application
        subgraph "Chains"
            direction TB
            BaseChain["base_chain.py"]:::application
            ExampleChain["example_chain.py"]:::application
        end
        subgraph "Models"
            direction TB
            BaseModel["base_model.py"]:::application
            ModelFactory["model_factory.py"]:::application
        end
        subgraph "Prompt Management"
            direction TB
            PromptRepo["prompt_repository.py"]:::application
            PromptTemplate["prompt_template.py"]:::application
        end
    end

    %% Domain Layer
    subgraph "Domain Layer"
        direction TB
        EntityReq["LLMRequest"]:::domain
        EntityRes["LLMResponse"]:::domain
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        direction TB
        RedisCache["redis_cache.py"]:::infrastructure
        BaseProvider["base.py"]:::infrastructure
        OpenAIProv["openai.py"]:::infrastructure
        AnthropicProv["anthropic.py"]:::infrastructure
    end

    %% Core / Cross-cutting
    subgraph "Core / Cross-cutting"
        direction TB
        Config["config.py"]:::core
        Dependencies["dependencies.py"]:::core
        CrossCutting["cross_cutting.py"]:::core
    end

    %% External Services
    subgraph "External Services"
        direction TB
        RedisExt["Redis"]:::external
        OpenAIExt["OpenAI API"]:::external
        AnthropicExt["Anthropic API"]:::external
        JaegerExt["Jaeger/Tracing"]:::external
    end

    %% Interactions
    Client -->|"HTTP Request"| LLMRoutes
    LLMSchemaReq --> LLMRoutes
    LLMSchemaRes --> LLMRoutes
    LLMRoutes -->|"calls"| LLMOrch
    Dependencies --> LLMRoutes
    Config --> LLMRoutes

    LLMOrch -->|"check cache"| RedisCache
    RedisCache --> RedisExt
    LLMOrch -->|"cache miss: get prompt"| PromptRepo
    PromptRepo --> PromptTemplate
    LLMOrch -->|"select provider"| ModelFactory
    ModelFactory --> BaseProvider
    BaseProvider --> OpenAIProv
    BaseProvider --> AnthropicProv
    OpenAIProv --> OpenAIExt
    AnthropicProv --> AnthropicExt
    LLMOrch -->|"store cache"| RedisCache
    LLMOrch -->|"track usage"| UsageTracker
    UsageTracker --> CrossCutting
    CrossCutting --> JaegerExt

    LLMOrch --> EntityReq
    LLMOrch --> EntityRes
    EntityRes --> LLMRoutes

    Dependencies --> LLMOrch
    Config --> LLMOrch
    CrossCutting --> LLMOrch

    %% Click Events
    click LLMRoutes "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/presentation/api/routes/llm_routes.py"
    click LLMSchemaReq "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/presentation/api/schemas/llm_request.py"
    click LLMSchemaRes "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/presentation/api/schemas/llm_response.py"
    click LLMOrch "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/application/services/llm_orchestrator.py"
    click UsageTracker "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/application/services/usage_tracker.py"
    click BaseChain "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/application/chains/base_chain.py"
    click ExampleChain "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/application/chains/specific_chains/example_chain.py"
    click BaseModel "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/application/models/base_model.py"
    click ModelFactory "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/application/models/model_factory.py"
    click PromptRepo "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/application/prompt_management/prompt_repository.py"
    click PromptTemplate "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/application/prompt_management/prompt_template.py"
    click EntityReq "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/domain/llm_request.py"
    click EntityRes "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/domain/llm_response.py"
    click RedisCache "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/infrastructure/cache/redis_cache.py"
    click BaseProvider "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/infrastructure/llm_providers/base.py"
    click OpenAIProv "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/infrastructure/llm_providers/openai.py"
    click AnthropicProv "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/infrastructure/llm_providers/anthropic.py"
    click Config "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/core/config.py"
    click Dependencies "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/core/dependencies.py"
    click CrossCutting "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/src/core/cross_cutting.py"
    click JaegerExt "https://github.com/onlythompson/llm-powered-microservice-template/blob/main/k8s/llm_deployment.yaml"

    %% Styles
    classDef presentation fill:#CCE5FF,stroke:#333,stroke-width:1px
    classDef application fill:#CFFFCF,stroke:#333,stroke-width:1px
    classDef domain fill:#FFFACD,stroke:#333,stroke-width:1px
    classDef infrastructure fill:#FFD8B1,stroke:#333,stroke-width:1px
    classDef core fill:#E0E0E0,stroke:#333,stroke-width:1px
    classDef external fill:none,stroke:#FF3333,stroke-dasharray: 5 5,stroke-width:1px
    class Client,RedisExt,OpenAIExt,AnthropicExt,JaegerExt external
```
