# marketagents-ai/multiinference architecture

> Architecture diagram of the GitHub repository marketagents-ai/multiinference, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marketagents-ai/multiinference
- Repository: https://github.com/marketagents-ai/multiinference

## Overview

##### Main Components
- Entities & Versioning: At the core, all persistent objects (including ChatThread, ChatMessage, LLMConfig, Tools, etc.) subclass an Entity. They are maintained via the EntityRegistry, which handles immutable snapshots and lineage (forking/versioning).
- Registries:
– EntityRegistry: Manages Entity snapshots and their history.
– CallableRegistry: Keeps track of registered Python callables (“tools”) and auto-generates JSON schemas from type hints or Pydantic models.
- Chat Thread & LLM Configuration: The ChatThread represents multi-turn conversations with embedded configurations (LLMConfig that holds provider details like OpenAI, Anthropic, etc.) plus a system prompt, history, and attached tools.
- Tools:
– CallableTool: Represents functions that the LLM can invoke.
– StructuredTool: Enforces JSON schema output.
- Concurrency Engine: Encapsulated principally in the InferenceOrchestrator (in minference/threads/inference.py) and the oai_parallel module. This component schedules and executes parallel LLM API calls, manages rate limits, retries, and post-processing of results.
- LLM Clients: Under minference/clients, these modules provide request formatting and schema (with Pydantic models) for providers such as OpenAI, Anthropic, VLLM, LiteLLM, and OpenRouter.

##### Establish the Relationships & Data Flow
- A “client” (often shown at the entry point from the examples or tests) creates a ChatThread with its configuration and desired system prompt.
- That ChatThread, as an immutable entity, is registered with the EntityRegistry. When changes occur (like modifying the LLMConfig), the entity is “forked” to retain version history.
- The ChatThread may include one or more Tools (Callable or Structured) that are registered via the CallableRegistry.
- The InferenceOrchestrator (concurrent execution component) receives one or multiple ChatThread objects.
- It then transforms these ChatThreads into a batch of asynchronous requests (using utilities from clients/requests.py and clients_models.py) for the targeted LLM provider.
- The concurrency engine (with logic from oai_parallel.py) schedules these calls honoring rate limits and collects processed responses.

##### Architectural Patterns & Design Concepts
- Immutable Data Pattern & Versioning: Clearly emphasize the fork-and-registry mechanism to maintain a history of entity states.
- Registry Pattern: Display both registries (entity and callable) which are key to this architecture.

##### Specify Relevant Technologies, Frameworks, and Libraries
- Python (core language) with type hints and Pydantic for schema validation.
- Asynchronous execution (via asyncio) as seen in examples.
- Integration with external LLM providers like OpenAI, Anthropic, vLLM, LiteLLM, and OpenRouter.
- Test components that provide examples of usage and ensure the system integrity (located in the tests folder).

- Each major component should be clearly labeled. For instance, “EntityRegistry”, “CallableRegistry”, “ChatThread”, “LLMConfig”, “InferenceOrchestrator”, “CallableTool”, “StructuredTool”, and “Providers (OpenAI, etc.)”.
– From creation of ChatThread (input) → InferenceOrchestrator → External LLM API call.
– Back from LLM response → processing/updating entity → new version stored in EntityRegistry.
– Core domain entities (e.g., rounded rectangles).
– Concurrent processing components (use hexagons or parallelograms).
- Annotations such as “forking mechanism”, “rate limiting”, or “automatic JSON schema generation” to clarify design decisions.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Client Request
    Client["Client Request"]:::external

    %% Domain Entities
    subgraph "Domain Entities"
        E1["Entity Base"]:::domain
        CT["ChatThread & LLMConfig"]:::domain
    end

    %% Tool & Registry Layer
    subgraph "Tool & Registry Layer"
        ER["EntityRegistry"]:::registry
        CR["CallableRegistry"]:::registry
        T["Tools"]:::registry
    end

    %% Concurrency Engine
    subgraph "Concurrency Engine"
        IO["InferenceOrchestrator"]:::concurrency
        OP["OAI Parallel"]:::concurrency
    end

    %% External LLM Services
    subgraph "External LLM Services"
        LC["LLM Clients"]:::external
        LP["LLM Providers"]:::external
    end

    %% Data Flow Connections
    Client -->|"creates"| CT
    CT -->|"registered"| ER
    CT -->|"attaches"| T
    CT -->|"submitted to"| IO
    T -->|"registered via"| CR
    IO -->|"schedules"| OP
    IO -->|"formats requests"| LC
    LC -->|"calls"| LP
    LP -->|"returns"| LC
    LC -->|"sends response to"| IO
    IO -->|"updates"| ER
    ER -->|"forks to update"| CT

    %% Click Events
    click E1 "https://github.com/marketagents-ai/multiinference/blob/main/minference/ecs/entity.py"
    click ER "https://github.com/marketagents-ai/multiinference/blob/main/minference/ecs/base_registry.py"
    click ER "https://github.com/marketagents-ai/multiinference/blob/main/minference/ecs/caregistry.py"
    click CT "https://github.com/marketagents-ai/multiinference/blob/main/minference/threads/models.py"
    click CT "https://github.com/marketagents-ai/multiinference/blob/main/examples/test_chat_thread.py"
    click T "https://github.com/marketagents-ai/multiinference/blob/main/examples/tools_setup_example.py"
    click T "https://github.com/marketagents-ai/multiinference/blob/main/examples/test_callable_tool_serialization.py"
    click IO "https://github.com/marketagents-ai/multiinference/blob/main/minference/threads/inference.py"
    click OP "https://github.com/marketagents-ai/multiinference/blob/main/minference/oai_parallel.py"
    click LC "https://github.com/marketagents-ai/multiinference/blob/main/minference/clients/clients_models.py"
    click LC "https://github.com/marketagents-ai/multiinference/blob/main/minference/clients/requests.py"
    click LC "https://github.com/marketagents-ai/multiinference/blob/main/minference/clients/utils.py"

    %% Styles
    classDef domain fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef registry fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef concurrency fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef external fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
```
