How omarhimada/local-llm-onnx works
1. Overall shape
- This is a local Windows desktop chat application built with WPF (App.xaml, MainWindow.xaml, LoadingWindow.xaml).
- The app is intentionally offline-only: no HTTP APIs, no websocket middle layer, no external inference service. The UI talks directly to local ONNX model runtimes and local assets.
2. Startup and app shell
- App.xaml.cs is the entry point, with a dedicated loading screen (LoadingWindow) used while models and runtime state are prepared.
- Initialization/EnsureModelsArePresent.cs suggests the startup path validates that required model files exist before the main UI becomes usable.
3. UI layer
- MainWindow is the primary chat interface.
- State/Thinking/FloatingAdorner.cs implies a visual “thinking” indicator overlay during inference.
- The UI is likely driven by state objects rather than direct model calls, keeping the window code thin and event-oriented.
4. Conversation / interaction layer
- Interact/LinearCommunication.cs is the core orchestration path for turn-by-turn chat.
- This layer likely assembles user input, context, model prompt, and any retrieved memory before sending a request to the local LLM.
- The name “LinearCommunication” suggests a sequential prompt-response flow rather than an agent graph or multi-worker architecture.
5. Model runtime boundary
- The repository ships model artifacts under ONNX/, including:
- gpt-oss-20b for the main chat model,
- all-MiniLM-L6-v2-onnx for embeddings,
- med-gemma-27b and phi4-related support in utility code,
- Stable Diffusion assets under ONNX/SD.
- This is the main compute boundary: the app does not call a server; it loads local model directories directly from disk.
6. Chat model abstraction
- State/ModelState.cs and Utility/ModelSpecific/* indicate the app has a layer for model-specific behavior.
- That abstraction is important because different models need different prompt formatting, token handling, or generation settings.
- The presence of gpt-oss-20b, MedGemma, and Phi4 helpers implies the app is designed to support multiple local LLM families, even if one is the default.
7. Embedding / retrieval memory
- Memory/Discussion.cs, Memory/Remember.cs, and Memory/MiniEmbedder.cs form the memory subsystem.
- State/EmbedderState.cs shows embeddings are treated as a first-class runtime resource, separate from the main chat model.
- The README describes a retrieval-augmented memory goal: store discussions locally, then search similar past items before responding. This is a key architectural flow: user input → embed/search memory → augment prompt → generate answer.
8. Parsing and incremental state
- State/IncrementalParserState.cs suggests the app can consume partial model output and maintain parser progress over time.
- This is often used for streaming-like UI updates, structured output detection, or progressive rendering of generated text.
9. Utility and formatting layer
- Utility/ contains reusable support code for text and output shaping:
- Md.cs, MdFd.cs, Markdown-related syntax utilities,
- Utility/Syntax/* for highlighting and language registry,
- Utility/StringCleaner.cs, Base64.cs, and Fd.cs for general helpers.
- These utilities sit between raw model output and the rendered UI, especially where markdown or code snippets need formatting.
10. Template and code-generation tooling
- Utility/J2CS/* is a notable subsystem: it parses Jinja-like templates and generates C#-side representations.
- The presence of chat_template.jinja inside ONNX/gpt-oss-20b makes this especially relevant: prompt/chat templates appear to be converted or interpreted locally rather than delegated to another toolchain.
11. Image generation sidecar
- SD/Diffusion.cs, DiffusionOptions.cs, and LmsScheduler.cs indicate a separate stable diffusion inference path exists in the same app.
- This is a parallel local-model capability, likely optional and isolated from the chat flow, but sharing the same no-network architecture and ONNX asset pattern.
12. State and lifecycle boundaries
- State/* holds runtime state objects for loaded models, embeddings, and parser progress.
- That separation implies the app is organized around UI → state → inference/memory subsystems, rather than letting view code manage model lifecycles directly.
- For an architecture diagram, the most important boundary is that the main window does not own the model files or memory logic directly; those responsibilities are split into initialization, state, interaction, and memory layers.
13. External dependencies and deployment
- The repo is packaged as a desktop app (OLLM.csproj, .slnx, publish profile), with the model payload stored locally under ONNX/.
- The main operational dependency is disk availability and local runtime support for ONNX Generative AI, not any cloud service.
- The README’s setup note confirms the app expects large model artifacts to be present locally, making model files part of the deployment boundary.
14. Practical architecture summary
- A good diagram would show:
- WPF shell on the left,
- interaction/orchestration in the middle,
- local ONNX model runtimes plus memory/embedding services as separate back-end subsystems,
- and model artifacts on disk as the persistent runtime dependency.
- The dominant data flow is: user prompt → optional memory retrieval/embedding → prompt assembly → local LLM inference → formatted response in UI.
Main entry points: LinearCommunication.cs Turn flow · LoadingWindow.xaml.cs Loading screen · MainWindow.xaml.cs Main chat · ModelState.cs Model state
Components 21
WPF shell
Desktop windows and app entry points.
- App shell · Application startup and top-level lifecycle.
App.xaml.cs - Loading screen · Startup window shown while models and runtime state initialize.
LoadingWindow.xaml.cs - Main chat · Primary chat interface for user interaction.
MainWindow.xaml.cs - Thinking overlay · Visual indicator for ongoing inference or processing.
State/Thinking/FloatingAdorner.cs
Conversation runtime
Turn-by-turn chat orchestration and prompt assembly.
- Model check · Ensures required model files are present before use.
Initialization/EnsureModelsArePresent.cs - Turn flow · Sequential chat orchestration for prompts, context, and responses.
Interact/LinearCommunication.cs
Memory search
Local discussion storage and embedding retrieval.
- Discussion store · Persistent local conversation and retrieval corpus.
Memory/Discussion.cs - Recall logic · Searches and selects relevant past discussions.
Memory/Remember.cs - Mini embedder · Embeds text using the local embedding model.
Memory/MiniEmbedder.cs
Runtime state
Loaded model and parser state shared across the app.
- Model state · Tracks loaded chat model and related runtime configuration.
State/ModelState.cs - Embed state · Tracks embedding runtime availability and state.
State/EmbedderState.cs - Parser state · Maintains progress while consuming partial model output.
State/IncrementalParserState.cs
Local models
On-disk ONNX and model artifacts.
- GPT-OSS · Main local chat model artifacts and chat template.
ONNX/gpt-oss-20b - Embed model · Local sentence embedding model used for memory retrieval.
ONNX/Embed/all-MiniLM-L6-v2-onnx - MedGemma · Alternate model family supported by model-specific helpers.
ONNX/med-gemma-27b - SD assets · Stable Diffusion model bundle and tokenizer/runtime assets.
ONNX/SD/0nnX00Aammnpdxebr
Image generation
Optional Stable Diffusion inference path.
- Diffusion · Local Stable Diffusion inference path.
SD/Diffusion.cs - LMS scheduler · Scheduler used by the diffusion pipeline.
SD/LmsScheduler.cs
Formatting and codegen
Shared text utilities and template tooling.
- Markdown fmt · Markdown and response shaping utilities.
Utility/Md.cs - Syntax tools · Code highlighting and language registry utilities.
Utility/Syntax/Highlighter.cs - Jinja parser · Parses Jinja-like templates and generates C# representations.
Utility/J2CS/Converter.cs
Connections 23
- App shell → Loading screen: shows startup · Entry point opens the loading window during initialization.
- App shell → Model check: validates models · Startup checks that local model assets are available.
- Loading screen → Model state: prepares · Loading screen waits while runtime state is initialized.
- Loading screen → Embed state: prepares · Embedding runtime is brought up during startup.
- Loading screen → Main chat: opens · Main window becomes available after initialization completes.
- Main chat → Turn flow: drives chat · UI hands user turns to the conversation orchestrator.
- Main chat → Thinking overlay: shows status · Window displays processing feedback during inference.
- Turn flow → Recall logic: retrieves memory · Conversation flow consults recalled discussions before responding.
- Turn flow → Model state: uses chat model · Orchestration relies on the loaded LLM state.
- Turn flow → Parser state: tracks output · Incremental parsing state supports partial or structured responses.
- Recall logic → Discussion store: queries · Retrieval logic searches stored discussions.
- Recall logic → Mini embedder: embeds query · Relevant search requires text embeddings.
- Mini embedder → Embed model: loads from disk · Embedding wrapper uses the local ONNX embedding model.
- Model state → GPT-OSS: binds to · Main chat state points at the GPT-OSS model bundle.
- Embed state → Embed model: binds to · Embedding state points at the local embedding model bundle.
- Model state → MedGemma: supports · Model-specific behavior can target alternate LLM families.
- Turn flow → GPT-OSS: prompts · Final prompt assembly is sent to the local chat model.
- Turn flow → Markdown fmt: formats response · Generated text is shaped for display in the UI.
- Markdown fmt → Syntax tools: highlights code · Markdown rendering may include syntax-aware code formatting.
- GPT-OSS → Jinja parser: uses template · Chat template tooling supports model prompt formatting.
- Diffusion → SD assets: loads assets · Image generation reads local Stable Diffusion files from disk.
- Diffusion → LMS scheduler: uses scheduler · Diffusion inference depends on LMS scheduling.
- Main chat → Diffusion: optional image flow · Desktop app can invoke the separate local image pipeline.