# fvliang/dart architecture

> Architecture diagram of the GitHub repository fvliang/dart, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fvliang/dart
- Repository: https://github.com/fvliang/dart

## Overview

DART is a speculative-decoding library and Gradio application for LLM inference. A user supplies a prompt and model weights; the application or Python API tokenizes it, then DART combines a draft-layer forward pass with n-gram-guided tree search before generating output. The model package also includes model-family implementations, configuration, templates, and KV-cache support. The README documents the main capabilities and API; some runtime internals were not sampled, so uncertain links are omitted.

Main entry points: [`dart/model/dart_model.py`](https://github.com/fvliang/dart/blob/main/dart/model/dart_model.py) (DART Generation), [`dart/tree_search/tree_search.py`](https://github.com/fvliang/dart/blob/main/dart/tree_search/tree_search.py) (Tree Search), [`dart/app/app.py`](https://github.com/fvliang/dart/blob/main/dart/app/app.py) (Gradio UI), [`main.py`](https://github.com/fvliang/dart/blob/main/main.py) (Python Demo)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User Entry"]
  node_ui["Gradio UI<br/>[app.py]"]
  node_api["Python Demo<br/>[main.py]"]
end

subgraph group_inference["DART Inference"]
  node_templates["Prompt Templates<br/>[template.py]"]
  node_dartmodel["DART Generation<br/>[dart_model.py]"]
  node_utils["Generation Utilities<br/>[dart_utils.py]"]
end

subgraph group_search["Tree Search"]
  node_tree["Tree Search<br/>[tree_search.py]"]
  node_searchcpp["C++ Search<br/>[searcher.cpp]"]
  node_ngram[("N-gram Trie<br/>[trie_ngram.cpp]")]
  node_ngramdata["N-gram Builder<br/>[ngram_build.py]"]
end

subgraph group_models["Model Runtime"]
  node_base["Base Model<br/>[base.py]"]
  node_configs["DART Configurations<br/>[dart_configs.py]"]
  node_cache["KV Cache<br/>[kv_cache.py]"]
  node_llama["Llama Runtime"]
  node_mixtral["Mixtral Runtime"]
  node_qwen["Qwen3 Runtime"]
  node_qwenmoe["Qwen3 MoE Runtime"]
  node_llama3["Llama3 DART Layer<br/>[llama3_dart.py]"]
end

node_user(("User"))

node_user -->|"submits prompt"| node_ui
node_user -->|"calls API"| node_api
node_ui -->|"formats prompt"| node_templates
node_ui -->|"streams generation"| node_dartmodel
node_api -->|"loads and generates"| node_dartmodel
node_api -->|"formats prompt"| node_templates
node_dartmodel -->|"uses helpers"| node_utils
node_utils -->|"invokes search"| node_tree
node_tree -->|"searches candidates"| node_searchcpp
node_tree -->|"queries trie"| node_ngram
node_ngramdata -.->|"builds trie"| node_ngram
node_dartmodel -->|"wraps base model"| node_base
node_dartmodel -->|"loads configuration"| node_configs
node_dartmodel -->|"uses cache"| node_cache
node_dartmodel -->|"selects runtime"| node_llama
node_dartmodel -->|"selects runtime"| node_mixtral
node_dartmodel -->|"selects runtime"| node_qwen
node_dartmodel -->|"selects runtime"| node_qwenmoe
node_dartmodel -->|"uses draft layer"| node_llama3
node_dartmodel -.->|"constructs draft tree"| node_tree

click node_ui "https://github.com/fvliang/dart/blob/main/dart/app/app.py"
click node_api "https://github.com/fvliang/dart/blob/main/main.py"
click node_templates "https://github.com/fvliang/dart/blob/main/dart/model/template.py"
click node_dartmodel "https://github.com/fvliang/dart/blob/main/dart/model/dart_model.py"
click node_utils "https://github.com/fvliang/dart/blob/main/dart/model/dart_utils.py"
click node_tree "https://github.com/fvliang/dart/blob/main/dart/tree_search/tree_search.py"
click node_searchcpp "https://github.com/fvliang/dart/blob/main/dart/tree_search/cpplib/cpp_search/searcher.cpp"
click node_ngram "https://github.com/fvliang/dart/blob/main/dart/tree_search/cpplib/cpp_ngram/trie_ngram.cpp"
click node_ngramdata "https://github.com/fvliang/dart/blob/main/dart/tree_search/ngram_build/ngram_build.py"
click node_base "https://github.com/fvliang/dart/blob/main/dart/model/base.py"
click node_configs "https://github.com/fvliang/dart/blob/main/dart/model/dart_configs.py"
click node_cache "https://github.com/fvliang/dart/blob/main/dart/model/kv_cache.py"
click node_llama "https://github.com/fvliang/dart/blob/main/dart/model/modeling_llama_kv.py"
click node_mixtral "https://github.com/fvliang/dart/blob/main/dart/model/modeling_mixtral_kv.py"
click node_qwen "https://github.com/fvliang/dart/blob/main/dart/model/modeling_qwen3_kv.py"
click node_qwenmoe "https://github.com/fvliang/dart/blob/main/dart/model/modeling_qwen3_moe_kv.py"
click node_llama3 "https://github.com/fvliang/dart/blob/main/dart/model/llama3_dart.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_api,node_user toneBlue
class node_templates,node_dartmodel,node_utils toneAmber
class node_tree,node_searchcpp,node_ngram,node_ngramdata toneMint
class node_base,node_configs,node_cache,node_llama,node_mixtral,node_qwen,node_qwenmoe,node_llama3 toneRose
```

## Components

### User Entry

- [Gradio UI](https://github.com/fvliang/dart/blob/main/dart/app/app.py): `dart/app/app.py`
- [Python Demo](https://github.com/fvliang/dart/blob/main/main.py): `main.py`

### DART Inference

- [Prompt Templates](https://github.com/fvliang/dart/blob/main/dart/model/template.py): `dart/model/template.py`
- [DART Generation](https://github.com/fvliang/dart/blob/main/dart/model/dart_model.py): `dart/model/dart_model.py`
- [Generation Utilities](https://github.com/fvliang/dart/blob/main/dart/model/dart_utils.py): `dart/model/dart_utils.py`

### Tree Search

- [Tree Search](https://github.com/fvliang/dart/blob/main/dart/tree_search/tree_search.py): `dart/tree_search/tree_search.py`
- [C++ Search](https://github.com/fvliang/dart/blob/main/dart/tree_search/cpplib/cpp_search/searcher.cpp): `dart/tree_search/cpplib/cpp_search/searcher.cpp`
- [N-gram Trie](https://github.com/fvliang/dart/blob/main/dart/tree_search/cpplib/cpp_ngram/trie_ngram.cpp): `dart/tree_search/cpplib/cpp_ngram/trie_ngram.cpp`
- [N-gram Builder](https://github.com/fvliang/dart/blob/main/dart/tree_search/ngram_build/ngram_build.py): `dart/tree_search/ngram_build/ngram_build.py`

### Model Runtime

- [Base Model](https://github.com/fvliang/dart/blob/main/dart/model/base.py): `dart/model/base.py`
- [DART Configurations](https://github.com/fvliang/dart/blob/main/dart/model/dart_configs.py): `dart/model/dart_configs.py`
- [KV Cache](https://github.com/fvliang/dart/blob/main/dart/model/kv_cache.py): `dart/model/kv_cache.py`
- [Llama Runtime](https://github.com/fvliang/dart/blob/main/dart/model/modeling_llama_kv.py): `dart/model/modeling_llama_kv.py`
- [Mixtral Runtime](https://github.com/fvliang/dart/blob/main/dart/model/modeling_mixtral_kv.py): `dart/model/modeling_mixtral_kv.py`
- [Qwen3 Runtime](https://github.com/fvliang/dart/blob/main/dart/model/modeling_qwen3_kv.py): `dart/model/modeling_qwen3_kv.py`
- [Qwen3 MoE Runtime](https://github.com/fvliang/dart/blob/main/dart/model/modeling_qwen3_moe_kv.py): `dart/model/modeling_qwen3_moe_kv.py`
- [Llama3 DART Layer](https://github.com/fvliang/dart/blob/main/dart/model/llama3_dart.py): `dart/model/llama3_dart.py`

### Other components

- **User**

## Connections

- User → Gradio UI: submits prompt (evidence: `README.md`)
- User → Python Demo: calls API (evidence: `README.md`)
- Gradio UI → Prompt Templates: formats prompt (evidence: `dart/app/app.py`)
- Gradio UI → DART Generation: streams generation (evidence: `dart/app/app.py`)
- Python Demo → DART Generation: loads and generates (evidence: `main.py`)
- Python Demo → Prompt Templates: formats prompt (evidence: `main.py`)
- DART Generation → Generation Utilities: uses helpers (evidence: `dart/model/dart_model.py`)
- Generation Utilities → Tree Search: invokes search (evidence: `dart/model/dart_utils.py`)
- Tree Search → C++ Search: searches candidates (evidence: `dart/tree_search/tree_search.py`)
- Tree Search → N-gram Trie: queries trie (evidence: `dart/tree_search/tree_search.py`)
- N-gram Builder → N-gram Trie: builds trie
- DART Generation → Base Model: wraps base model (evidence: `dart/model/dart_model.py`)
- DART Generation → DART Configurations: loads configuration (evidence: `dart/model/dart_model.py`)
- DART Generation → KV Cache: uses cache (evidence: `dart/model/dart_model.py`)
- DART Generation → Llama Runtime: selects runtime (evidence: `dart/model/dart_model.py`)
- DART Generation → Mixtral Runtime: selects runtime (evidence: `dart/model/dart_model.py`)
- DART Generation → Qwen3 Runtime: selects runtime (evidence: `dart/model/dart_model.py`)
- DART Generation → Qwen3 MoE Runtime: selects runtime (evidence: `dart/model/dart_model.py`)
- DART Generation → Llama3 DART Layer: uses draft layer (evidence: `dart/model/dart_model.py`)
- DART Generation → Tree Search: constructs draft tree
