How breitburg/inference-for-dart works
Project Type & Purpose
• This is a Dart/Flutter library (“inference-for-dart”) that exposes a human-friendly API for running large-language-model inference locally by binding to the llama.cpp C++ backend.
• It’s not a standalone application but a package consumed by Dart or Flutter apps (see the example/ folder for sample usage).
File Structure Analysis
• Top-level:
• lib/ – the public API and implementation
• example/ – sample Dart/Flutter applications showing usage
• pubspec.yaml – Dart package configuration
• lib/src:
• inference.dart – export façade
• engine.dart – high-level InferenceEngine class
• model.dart – InferenceModel wrapper for GGUF files
• chat_messages.dart – ChatMessage data models
• sampler.dart – sampling logic (top-k, temperature)
• low_level.dart – FFI bindings to llama.cpp C API
• Configuration & metadata files for linting, license, and docs.
Key Components & Their Roles
a. Dart/Flutter Application (Example or End-User App)
– Calls into the inference library (e.g., engine.chat(), engine.embed()).
b. High-Level Dart API Layer
– InferenceModel loads metadata, manages file paths.
– InferenceEngine initializes the model in memory, runs inference, and disposes resources.
– ChatMessage encapsulates roles & content.
– Sampler applies temperature, top-k sampling.
c. Low-Level FFI Layer
– Loads libllama.so/.dylib/.dll via Dart FFI.
– Exposes C functions: llama_load_model(), llama_eval(), llama_tokenize(), etc.
d. Native C++ Library (llama.cpp)
– Performs the heavy lifting: reading .gguf weights into RAM, computing forward passes, sampling.
e. Model Files & External Services
– .gguf files on disk (or downloaded at runtime).
– Optionally fetch models from HuggingFace.
Interactions & Data Flow
1. App calls InferenceModel(path).fetchMetadata() → Dart I/O reads header only.
2. App constructs InferenceEngine(model) → Engine invokes lowLevelInference.dynamicLibrary = … and C.llama_allocate_context().
3. engine.initialize() → loads full weights via FFI → RAM.
4. engine.chat(messages) →
• Tokenize prompt via C.llama_tokenize()
• Evaluate tokens via C.llama_eval()
• Sample next token via C.llama_sample()
• Detokenize via C.llama_token_to_str() → Dart string.
5. engine.dispose() → FFI calls to free context and buffers.
Architectural Patterns & Principles
• Layered architecture (High-Level API → FFI → Native).
• Single Responsibility (each class handles one concern).
• Clear FFI boundary for maintainability.
• Future plan: isolate-based async execution to prevent main-thread blocking.
• Clear, consistent labels on every component.
• Arrowheads showing call direction and data return paths.
• Optional: annotate RAM usage estimate near the Native layer.