How onlythompson/llm-powered-microservice-template works
Project type and purpose
• This is a backend‐only microservice template built on FastAPI, following Clean Architecture principles, to provide LLM-powered endpoints (e.g., generate, summarize). It orchestrates calls to multiple LLM providers, manages prompt templates, caches responses, and exposes HTTP APIs for consumption.
File structure
• Top‐level directories:
• src/presentation – FastAPI routes & schemas (API layer)
• src/application – Business logic: chains, models, prompt management, orchestrator & usage tracking
• src/domain – Core entities (LLMRequest, LLMResponse)
• src/infrastructure – Adapters: cache (Redis), LLM providers (OpenAI, Anthropic)
• src/core – Configuration, dependency injection, cross-cutting concerns (logging, tracing)
• Supporting directories: tests, docs (API, architecture, deployment, security), k8s (manifests), scripts, config files
README for additional insights
• Confirms Clean Architecture layering, async processing, support for multiple LLM providers, caching, DI, distributed tracing, Docker/K8s deployment.
a. Main components to include:
• Client (e.g., HTTP client or UI)
• API Gateway / FastAPI Server (presentation layer)
• Router & Schema layer (llm_routes.py + Pydantic schemas)
• Application Layer
– LLMOrchestrator service
– Chains (BaseChain + specific_chains)
– ModelFactory & BaseModel
– PromptRepository & PromptTemplate
– UsageTracker
• Domain Entities (LLMRequest, LLMResponse)
• Infrastructure Layer
– RedisCache adapter
– LLMProviders (OpenAI, Anthropic, BaseLLMProvider interface)
• Core / Cross-cutting
– Config (env vars, token limits)
– DI Container (dependencies.py)
– Logging & Tracing (OpenTelemetry / Jaeger)
• External Services
– Redis
– LLM Provider Endpoints (OpenAI API, Anthropic API)
– Jaeger/Tracing Collector
• Deployment Artifacts
– Docker Container
– Kubernetes Pod (k8s/llm_deployment.yaml)
b. Relationships and interactions:
1. Client → FastAPI HTTP endpoint (/api/llm/generate)
2. FastAPI Router resolves schema → calls LLMOrchestrator (injected with ModelFactory, PromptRepository, RedisCache)
3. LLMOrchestrator orchestrates:
– Check RedisCache for a cached response
– If cache miss, retrieve prompt template from PromptRepository
– Format prompt via PromptTemplate
– Select provider via ModelFactory → calls provider’s API adapter (OpenAIClient or AnthropicClient)
– Receive LLMResponse
– Store response in RedisCache
– Track usage via UsageTracker
4. Propagate response back up to presentation layer → HTTP response to client
5. Throughout the request, cross-cutting logging/tracing intercepts calls
c. Architectural patterns / design principles:
• Clean Architecture (separate layers: presentation, application, domain, infrastructure, core)
• Dependency Injection via core/dependencies.py
• Factory Pattern for models/providers
• Strategy Pattern for selecting LLM providers
• Adapter Pattern for cache & external APIs
• Asynchronous processing for I/O bound calls
d. Key technologies / frameworks / libraries:
• FastAPI (HTTP framework)
• Pydantic (request/response schemas)
• Redis (caching layer)
• OpenAI / Anthropic client libraries
• Python AsyncIO
• OpenTelemetry / Jaeger for tracing
• Docker & Kubernetes for containerization & orchestration
Essential elements
• Five Clean Architecture layers and the external dependencies.
• How DI injects configurations and adapters into the orchestrator.
• Caching decision points and provider selection.