How agentjido/req_llm works
Project Type and Purpose
• ReqLLM is an Elixir library (not a standalone front-end/back-end app) that provides a unified, plugin-based interface for interacting with large language model (LLM) providers via HTTP.
• It builds on the Req HTTP client, adds typed data structures, streaming support, and a provider plugin architecture to simplify calls to OpenAI, Anthropic, Google, Groq, xAI, OpenRouter, and more.
High-Level Architectural Overview
• Core Library Modules (in lib/req_llm/)
• Context & Message builders (compose system/user prompts)
• Model metadata & schema definitions
• Provider plugin framework (registry, DSL, option translators)
• Capability modules (generation, embeddings, streaming, tool-calling)
• Response processing & usage extraction
• Utility modules (key management, codecs, JSON encoders)
• Mix Tasks (in lib/mix/tasks/):
• model_sync: auto-sync metadata from models.dev into JSON files under priv/models_dev/
• stream_text: a CLI helper for streaming text from the command line
• External Services:
• LLM Providers (via HTTP REST endpoints)
• models.dev API (for model metadata sync)
Main Components and Their Relationships
• Client API Layer (public functions like generate_text/3, stream_text/3, embed_many/2)
→ Builds a ReqLLM.Context struct and selects a ReqLLM.Model.
• Provider Plugin Layer
→ ReqLLM.Provider.Registry resolves the correct provider module (e.g., ReqLLM.Providers.OpenAI).
→ Uses a Provider DSL and ContextCodec to translate internal structs into provider-specific HTTP payloads.
• Req HTTP Pipeline
→ The provider module attaches middleware and executes Req.request/1.
→ Streaming calls produce a Stream of ReqLLM.StreamChunk entries.
• Response Handling
→ Raw JSON from HTTP response is decoded into Elixir structs (Response, Usage, content parts, etc.).
→ Cost and token usage are extracted into a ReqLLM.Usage struct.
→ Result is returned to the caller (either {:ok, struct} or raised on ! variants).
• Model Metadata Management
→ mix req_llm.models.sync fetches metadata from models.dev, writes to priv/models_dev/*.json.
→ On startup (ReqLLM.Application), metadata JSON is loaded into memory for auto-sync and option validation.
Architectural Patterns and Design Principles
• Plugin Architecture: providers implement a common behavior (ReqLLM.Plugin) and register themselves, enabling easy extension.
• Composability: builds on Req’s middleware pipeline for tracing, instrumentation, and custom middleware injection.
• Typed Data Structures: every call, response, and intermediate piece is a well-defined struct that can be serialized.
• Separation of Concerns: split into context construction, provider translation, HTTP transport, and response decoding.
• Mix Tasks for operational concerns (metadata sync, CLI tooling).