How marketagents-ai/multiinference works
Main Components
• Entities & Versioning: At the core, all persistent objects (including ChatThread, ChatMessage, LLMConfig, Tools, etc.) subclass an Entity. They are maintained via the EntityRegistry, which handles immutable snapshots and lineage (forking/versioning).
• Registries:
– EntityRegistry: Manages Entity snapshots and their history.
– CallableRegistry: Keeps track of registered Python callables (“tools”) and auto-generates JSON schemas from type hints or Pydantic models.
• Chat Thread & LLM Configuration: The ChatThread represents multi-turn conversations with embedded configurations (LLMConfig that holds provider details like OpenAI, Anthropic, etc.) plus a system prompt, history, and attached tools.
• Tools:
– CallableTool: Represents functions that the LLM can invoke.
– StructuredTool: Enforces JSON schema output.
• Concurrency Engine: Encapsulated principally in the InferenceOrchestrator (in minference/threads/inference.py) and the oai_parallel module. This component schedules and executes parallel LLM API calls, manages rate limits, retries, and post-processing of results.
• LLM Clients: Under minference/clients, these modules provide request formatting and schema (with Pydantic models) for providers such as OpenAI, Anthropic, VLLM, LiteLLM, and OpenRouter.
Establish the Relationships & Data Flow
• A “client” (often shown at the entry point from the examples or tests) creates a ChatThread with its configuration and desired system prompt.
• That ChatThread, as an immutable entity, is registered with the EntityRegistry. When changes occur (like modifying the LLMConfig), the entity is “forked” to retain version history.
• The ChatThread may include one or more Tools (Callable or Structured) that are registered via the CallableRegistry.
• The InferenceOrchestrator (concurrent execution component) receives one or multiple ChatThread objects.
• It then transforms these ChatThreads into a batch of asynchronous requests (using utilities from clients/requests.py and clients_models.py) for the targeted LLM provider.
• The concurrency engine (with logic from oai_parallel.py) schedules these calls honoring rate limits and collects processed responses.
Architectural Patterns & Design Concepts
• Immutable Data Pattern & Versioning: Clearly emphasize the fork-and-registry mechanism to maintain a history of entity states.
• Registry Pattern: Display both registries (entity and callable) which are key to this architecture.
Specify Relevant Technologies, Frameworks, and Libraries
• Python (core language) with type hints and Pydantic for schema validation.
• Asynchronous execution (via asyncio) as seen in examples.
• Integration with external LLM providers like OpenAI, Anthropic, vLLM, LiteLLM, and OpenRouter.
• Test components that provide examples of usage and ensure the system integrity (located in the tests folder).
• Each major component should be clearly labeled. For instance, “EntityRegistry”, “CallableRegistry”, “ChatThread”, “LLMConfig”, “InferenceOrchestrator”, “CallableTool”, “StructuredTool”, and “Providers (OpenAI, etc.)”.
– From creation of ChatThread (input) → InferenceOrchestrator → External LLM API call.
– Back from LLM response → processing/updating entity → new version stored in EntityRegistry.
– Core domain entities (e.g., rounded rectangles).
– Concurrent processing components (use hexagons or parallelograms).
• Annotations such as “forking mechanism”, “rate limiting”, or “automatic JSON schema generation” to clarify design decisions.