# scisharp/llamasharp architecture

> Architecture diagram of the GitHub repository scisharp/llamasharp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scisharp/llamasharp
- Repository: https://github.com/scisharp/llamasharp

## Overview

##### Main Components

- Core Inference Library: Represent the “LLama” project (along with subdirectories like Abstractions, Batched, ChatSession, Common, Native, Sampling, Transformers, etc.). This is the heart of the library and handles model loading, context management, text generation, prompt transformation, and integration with native backends.

- Native Backend Integration: Include the native library wrappers (the “Native” folder in LLama along with the subdirectories of “Load” and native types) that allow interop with llama.cpp. Indicate that there are multiple backend implementations (for CPU, CUDA, Vulkan, etc.) installed as separate NuGet packages.

- Executor & Inference Modules: Identify the executors (InteractiveExecutor, LLamaInstructExecutor, LLamaInteractExecutor, StatelessExecutor, etc.) as important components that control the inference flow. Indicate additional sampling pipelines and transforms.

- Integration Modules: Show integration projects:
- LLama.SemanticKernel for enabling integration with Microsoft’s Semantic Kernel.
- LLama.KernelMemory for Retrieval Augmented Generation (RAG) and memory support.
- Others (extensions for BotSharp, LangChain, etc.) as external integrations.

- Demos & Examples: Represent the examples/demos layer – these include LLama.Examples (for console demos and sample usage), LLama.Web (an ASP.NET site that shows integration and UI), and LLama.WebAPI (providing an API layer over the core inference engine).

##### Relationships and Interactions

- That the high-level application layers (Console Demo, Web/REST API demos) call into the Core Inference Library.
- The Core Library itself interacts with the Native Backend layer for performing the actual model computations.
- Integration components like Semantic Kernel and Kernel Memory extend or wrap the core functionality to provide additional features (e.g., semantic chat sessions or RAG support).
- That unit tests and benchmarks validate the functionality and performance of the Core Library.

##### Architectural Patterns or Design Principles

- Layered Architecture: Distinguish between the native integration layer, core library layer, and application (client) layer.
- Modular Design: Each major component is implemented as its own project (DLL) making use of clear abstraction interfaces (e.g., in the “Abstractions” folder).
- Extensibility: The project demonstrates extension points (e.g., ILLamaExecutor, ITextTransform, and various extension methods) that allow custom executors or integrations.
- Separation of Concerns: The responsibilities are clearly separated between model loading/inference, demonstration application logic, and integration with external services.

##### Relevant Technologies and Frameworks

- .NET/C# is used as the primary programming language.
- Native libraries and platform-specific backend packages (CPU, CUDA, Vulkan) are wrapped and called from .NET.
- For web/demo projects, ASP.NET Core (with SignalR for real-time communication in LLama.Web) is used.
- Testing frameworks and benchmarking tools used in LLama.Unittest and LLama.Benchmark.
- CI/CD pipelines and configuration management via GitHub Actions.

##### Detail but Keep It Manageable

- Essential architectural elements rather than trying to include every small file. Outline the high-level modules and how they relate.
- Primary execution flow:
1. Application/UI sends a user command.
2. The corresponding service/executor calls into the core library.
3. The core library invokes native calls for inference.
4. Results are returned upward to be processed/displayed.
- Optionally, add side notes for build and integration points (like GitHub Actions for CI/CD, documentation generation, etc.).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        Demo1["LLama.Examples – Console Demo"]:::demo
        Demo2["LLama.Web – ASP.NET Web App"]:::demo
        Demo3["LLama.WebAPI – API Interface"]:::demo
    end

    %% Integration Modules
    subgraph "Integration Modules"
        Int1["LLama.SemanticKernel"]:::integration
        Int2["LLama.KernelMemory"]:::integration
    end

    %% Service Layer with Executors
    subgraph "Service Layer"
        Core["Core Inference Engine"]:::core
        subgraph "Executors"
            Ex1["LLamaInstructExecutor"]:::executor
            Ex2["LLamaInteractExecutor"]:::executor
            Ex3["LLamaStatelessExecutor"]:::executor
            Ex4["Batched Executor"]:::executor
        end
    end

    %% Native Backend Layer
    subgraph "Native Backend"
        Native["Native Backend Integration"]:::native
    end

    %% Quality Assurance Layer
    subgraph "Quality Assurance"
        Test1["LLama.Unittest"]:::test
        Test2["LLama.Benchmark"]:::test
    end

    %% Infrastructure Layer
    subgraph "Infrastructure"
        Infra1["Documentation"]:::infra
        Infra2["CI/CD Pipeline"]:::infra
    end

    %% Relationships
    Demo1 -->|"calls"| Core
    Demo2 -->|"calls"| Core
    Demo3 -->|"calls"| Core

    Core -->|"executes"| Ex1
    Core -->|"executes"| Ex2
    Core -->|"executes"| Ex3
    Core -->|"executes"| Ex4

    Core -->|"delegatesTo"| Native

    Int1 -->|"enhances"| Core
    Int2 -->|"enhances"| Core

    Test1 -->|"tests"| Core
    Test2 -->|"benchmarks"| Core

    Infra1 -->|"supports"| Core
    Infra1 -->|"documents"| Test1
    Infra1 -->|"documents"| Test2
    Infra2 -->|"builds"| Core
    Infra2 -->|"runs"| Test1
    Infra2 -->|"runs"| Test2

    %% Click Events
    click Core "https://github.com/scisharp/llamasharp/tree/master/LLama"
    click Native "https://github.com/scisharp/llamasharp/tree/master/LLama/Native"
    click Ex1 "https://github.com/scisharp/llamasharp/blob/master/LLama/LLamaInstructExecutor.cs"
    click Ex2 "https://github.com/scisharp/llamasharp/blob/master/LLama/LLamaInteractExecutor.cs"
    click Ex3 "https://github.com/scisharp/llamasharp/blob/master/LLama/LLamaStatelessExecutor.cs"
    click Ex4 "https://github.com/scisharp/llamasharp/blob/master/LLama/Batched/BatchedExecutor.cs"
    click Int1 "https://github.com/scisharp/llamasharp/blob/master/LLama.SemanticKernel"
    click Int2 "https://github.com/scisharp/llamasharp/blob/master/LLama.KernelMemory"
    click Demo1 "https://github.com/scisharp/llamasharp/blob/master/LLama.Examples"
    click Demo2 "https://github.com/scisharp/llamasharp/blob/master/LLama.Web"
    click Demo3 "https://github.com/scisharp/llamasharp/blob/master/LLama.WebAPI"
    click Test1 "https://github.com/scisharp/llamasharp/blob/master/LLama.Unittest"
    click Test2 "https://github.com/scisharp/llamasharp/blob/master/LLama.Benchmark"
    click Infra1 "https://github.com/scisharp/llamasharp/tree/master/docs"
    click Infra2 "https://github.com/scisharp/llamasharp/blob/master/.github"

    %% Styles
    classDef demo fill:#f9e79f,stroke:#d35400,stroke-width:2px;
    classDef integration fill:#d4efdf,stroke:#196f3d,stroke-width:2px;
    classDef core fill:#aed6f1,stroke:#1f618d,stroke-width:2px;
    classDef executor fill:#fadbd8,stroke:#c0392b,stroke-width:2px;
    classDef native fill:#f5b7b1,stroke:#943126,stroke-width:2px;
    classDef test fill:#d2b4de,stroke:#6c3483,stroke-width:2px;
    classDef infra fill:#fcf3cf,stroke:#b7950b,stroke-width:2px;
```
