# breitburg/inference-for-dart architecture

> Architecture diagram of the GitHub repository breitburg/inference-for-dart, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/breitburg/inference-for-dart
- Repository: https://github.com/breitburg/inference-for-dart

## Overview

##### Project Type & Purpose
- This is a Dart/Flutter library (“inference-for-dart”) that exposes a human-friendly API for running large-language-model inference locally by binding to the llama.cpp C++ backend.
- It’s not a standalone application but a package consumed by Dart or Flutter apps (see the `example/` folder for sample usage).

##### File Structure Analysis
- Top-level:
- lib/ – the public API and implementation
- example/ – sample Dart/Flutter applications showing usage
- pubspec.yaml – Dart package configuration
- lib/src:
- inference.dart – export façade
- engine.dart – high-level InferenceEngine class
- model.dart – InferenceModel wrapper for GGUF files
- chat_messages.dart – ChatMessage data models
- sampler.dart – sampling logic (top-k, temperature)
- low_level.dart – FFI bindings to llama.cpp C API
- Configuration & metadata files for linting, license, and docs.

##### Key Components & Their Roles
a. Dart/Flutter Application (Example or End-User App)
– Calls into the inference library (e.g., `engine.chat()`, `engine.embed()`).
b. High-Level Dart API Layer
– `InferenceModel` loads metadata, manages file paths.
– `InferenceEngine` initializes the model in memory, runs inference, and disposes resources.
– `ChatMessage` encapsulates roles & content.
– `Sampler` applies temperature, top-k sampling.
c. Low-Level FFI Layer
– Loads `libllama.so`/`.dylib`/`.dll` via Dart FFI.
– Exposes C functions: `llama_load_model()`, `llama_eval()`, `llama_tokenize()`, etc.
d. Native C++ Library (llama.cpp)
– Performs the heavy lifting: reading `.gguf` weights into RAM, computing forward passes, sampling.
e. Model Files & External Services
– `.gguf` files on disk (or downloaded at runtime).
– Optionally fetch models from HuggingFace.

##### Interactions & Data Flow
1. App calls `InferenceModel(path).fetchMetadata()` → Dart I/O reads header only.
2. App constructs `InferenceEngine(model)` → Engine invokes `lowLevelInference.dynamicLibrary = …` and `C.llama_allocate_context()`.
3. `engine.initialize()` → loads full weights via FFI → RAM.
4. `engine.chat(messages)` →
- Tokenize prompt via `C.llama_tokenize()`
- Evaluate tokens via `C.llama_eval()`
- Sample next token via `C.llama_sample()`
- Detokenize via `C.llama_token_to_str()` → Dart string.
5. `engine.dispose()` → FFI calls to free context and buffers.

##### Architectural Patterns & Principles
- Layered architecture (High-Level API → FFI → Native).
- Single Responsibility (each class handles one concern).
- Clear FFI boundary for maintainability.
- Future plan: isolate-based async execution to prevent main-thread blocking.

- Clear, consistent labels on every component.
- Arrowheads showing call direction and data return paths.
- Optional: annotate RAM usage estimate near the Native layer.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Layer 1: App/UI
    subgraph "Layer 1: App/UI"
        ExampleApp["Example App"]:::app
        RAGDemo["RAG Demo App"]:::app
    end

    %% Layer 2: Dart Library
    subgraph "Layer 2: Dart Library"
        Facade["inference.dart"]:::api
        Model["InferenceModel"]:::dart
        Engine["InferenceEngine"]:::dart
        ChatMsg["ChatMessage"]:::dart
        Sampler["Sampler"]:::dart
    end

    %% Layer 3: FFI
    subgraph "Layer 3: FFI"
        LowLevel["Low-Level FFI"]:::ffi
    end

    %% Layer 4: Native
    subgraph "Layer 4: Native"
        LlamaLib["llama.cpp (Dynamic Library)"]:::native
        ModelFile[(.gguf Model File)]:::storage
    end

    %% Layer 5: External
    subgraph "Layer 5: External"
        HuggingFace["HuggingFace"]:::external
    end

    %% Connections
    ExampleApp -->|"engine.chat() / embed()"| Facade
    RAGDemo -->|"engine.chat() / embed()"| Facade

    Facade -->|"new InferenceEngine(model)"| Engine
    Facade -->|"InferenceModel.fetchMetadata()"| Model

    Engine -->|"initialize() / dispose()"| LowLevel
    Engine -->|"uses"| Model
    Engine -->|"chat(messages)"| ChatMsg
    Engine -->|"sampling"| Sampler

    LowLevel -->|"llama_load_model() / llama_eval() / llama_tokenize() / llama_sample() / llama_token_to_str()"| LlamaLib

    LlamaLib -->|"reads weights into RAM"| ModelFile
    HuggingFace -.->|"download .gguf"| ModelFile

    %% Click Events
    click ExampleApp "https://github.com/breitburg/inference-for-dart/blob/main/example/bin/example.dart"
    click RAGDemo "https://github.com/breitburg/inference-for-dart/blob/main/example/bin/rag.dart"
    click Facade "https://github.com/breitburg/inference-for-dart/blob/main/lib/inference.dart"
    click Engine "https://github.com/breitburg/inference-for-dart/blob/main/lib/src/engine.dart"
    click Model "https://github.com/breitburg/inference-for-dart/blob/main/lib/src/model.dart"
    click ChatMsg "https://github.com/breitburg/inference-for-dart/blob/main/lib/src/chat_messages.dart"
    click Sampler "https://github.com/breitburg/inference-for-dart/blob/main/lib/src/sampler.dart"
    click LowLevel "https://github.com/breitburg/inference-for-dart/blob/main/lib/src/low_level.dart"
    click HuggingFace "https://github.com/breitburg/inference-for-dart/blob/main/example/pubspec.yaml"
    click ExampleApp "https://github.com/breitburg/inference-for-dart/blob/main/example/analysis_options.yaml"
    click ExampleApp "https://github.com/breitburg/inference-for-dart/blob/main/pubspec.yaml"

    %% Styles
    classDef app fill:#D0E8FF,stroke:#0366d6,stroke-width:1px;
    classDef api fill:#ADE8FF,stroke:#1B95E0,stroke-width:1px;
    classDef dart fill:#CFE2FF,stroke:#2F6DB5,stroke-width:1px;
    classDef ffi fill:#FFF3BF,stroke:#F2C94C,stroke-width:1px;
    classDef native fill:#D0F2D0,stroke:#17A661,stroke-width:1px,stroke-dasharray: 5 5;
    classDef storage fill:#FFE8A1,stroke:#D97706,stroke-width:1px,stroke-dasharray: 2 2;
    classDef external fill:#E2E8F0,stroke:#64748B,stroke-width:1px,stroke-dasharray: 2 2;
```
