# huyquoctrinh/firebolt-vl architecture

> Architecture diagram of the GitHub repository huyquoctrinh/firebolt-vl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huyquoctrinh/firebolt-vl
- Repository: https://github.com/huyquoctrinh/firebolt-vl

## Overview

Firebolt-VL is a vision-language model for image-grounded prompt answering, with inference and training entry points. The graph follows the documented image-to-visual-feature, multimodal-fusion, and language-generation stages, plus training data preparation and optimization. Some implementation excerpts are partial; subsystem links not established by the sampled source or README are omitted.

Main entry points: [`modeling/model.py`](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/model.py) (Firebolt Language Model), [`infer.py`](https://github.com/huyquoctrinh/firebolt-vl/blob/main/infer.py) (Inference), [`train.py`](https://github.com/huyquoctrinh/firebolt-vl/blob/main/train.py) (Training Loop), [`modeling/cossm.py`](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/cossm.py) (Token-Grid Modulator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User Workflows"]
  node_infer["Inference<br/>[infer.py]"]
end

subgraph group_training["Training Pipeline"]
  node_dataset["Dataset Loader<br/>[dataset.py]"]
  node_train["Training Loop<br/>[train.py]"]
  node_ddp["Distributed Training<br/>[train_ddp.py]"]
  node_metric["Perplexity Metric<br/>[metric.py]"]
end

subgraph group_model["Multimodal Model"]
  node_model["Firebolt Language Model<br/>[model.py]"]
end

subgraph group_components["Model Components"]
  node_vision["SigLIP Vision Encoder<br/>[siglip.py]"]
  node_gridvision["Grid Vision Encoder<br/>[siglip_grid.py]"]
  node_projector["Vision Projector<br/>[projector.py]"]
  node_fuser["Multimodal Fuser<br/>[fuser.py]"]
  node_cossm["Token-Grid Modulator<br/>[cossm.py]"]
  node_s4["S4 Sequence Model<br/>[s4.py]"]
  node_s4d["S4D Sequence Model<br/>[s4d.py]"]
  node_factory["Component Registry<br/>[factory.py]"]
end

node_user(("User"))
node_image["Input Image"]
node_tokenizer["Tokenizer"]
node_processor["Image Processor"]
node_answer["Generated Answer"]
node_corpus["Training Data"]
node_language["LFM Language Decoder"]

node_user -->|"submits prompt"| node_infer
node_image -->|"provides image"| node_infer
node_infer -->|"formats prompt"| node_tokenizer
node_infer -->|"processes image"| node_processor
node_infer -->|"generates with"| node_model
node_model -->|"returns tokens"| node_answer
node_model -->|"decodes with"| node_language
node_model -->|"encodes image"| node_vision
node_model -->|"projects features"| node_projector
node_model -->|"fuses modalities"| node_fuser
node_vision -.->|"supports grid features"| node_gridvision
node_fuser -->|"selects modulator"| node_cossm
node_cossm -->|"uses sequence model"| node_s4
node_cossm -.->|"supports alternative"| node_s4d
node_factory -->|"builds registered fuser"| node_fuser
node_factory -->|"builds registered projector"| node_projector
node_corpus -->|"supplies examples"| node_dataset
node_dataset -->|"batches data"| node_train
node_train -->|"optimizes"| node_model
node_train -->|"evaluates perplexity"| node_metric
node_ddp -->|"loads batches"| node_dataset
node_ddp -->|"optimizes"| node_model

click node_infer "https://github.com/huyquoctrinh/firebolt-vl/blob/main/infer.py"
click node_dataset "https://github.com/huyquoctrinh/firebolt-vl/blob/main/dataset.py"
click node_train "https://github.com/huyquoctrinh/firebolt-vl/blob/main/train.py"
click node_ddp "https://github.com/huyquoctrinh/firebolt-vl/blob/main/train_ddp.py"
click node_metric "https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/metric.py"
click node_model "https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/model.py"
click node_vision "https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/vision_encoder/siglip.py"
click node_gridvision "https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/vision_encoder/siglip_grid.py"
click node_projector "https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/projector.py"
click node_fuser "https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/fuser.py"
click node_cossm "https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/cossm.py"
click node_s4 "https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/ssm/s4.py"
click node_s4d "https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/ssm/s4d.py"
click node_factory "https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/factory.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_infer,node_user toneBlue
class node_dataset,node_train,node_ddp,node_metric toneAmber
class node_model toneMint
class node_vision,node_gridvision,node_projector,node_fuser,node_cossm,node_s4,node_s4d,node_factory toneRose
class node_image,node_tokenizer,node_processor,node_answer,node_corpus,node_language toneIndigo
```

## Components

### User Workflows

- [Inference](https://github.com/huyquoctrinh/firebolt-vl/blob/main/infer.py): `infer.py`

### Training Pipeline

- [Dataset Loader](https://github.com/huyquoctrinh/firebolt-vl/blob/main/dataset.py): `dataset.py`
- [Training Loop](https://github.com/huyquoctrinh/firebolt-vl/blob/main/train.py): `train.py`
- [Distributed Training](https://github.com/huyquoctrinh/firebolt-vl/blob/main/train_ddp.py): `train_ddp.py`
- [Perplexity Metric](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/metric.py): `modeling/metric.py`

### Multimodal Model

- [Firebolt Language Model](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/model.py): `modeling/model.py`

### Model Components

- [SigLIP Vision Encoder](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/vision_encoder/siglip.py): `modeling/vision_encoder/siglip.py`
- [Grid Vision Encoder](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/vision_encoder/siglip_grid.py): `modeling/vision_encoder/siglip_grid.py`
- [Vision Projector](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/projector.py): `modeling/projector.py`
- [Multimodal Fuser](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/fuser.py): `modeling/fuser.py`
- [Token-Grid Modulator](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/cossm.py): `modeling/cossm.py`
- [S4 Sequence Model](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/ssm/s4.py): `modeling/ssm/s4.py`
- [S4D Sequence Model](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/ssm/s4d.py): `modeling/ssm/s4d.py`
- [Component Registry](https://github.com/huyquoctrinh/firebolt-vl/blob/main/modeling/factory.py): `modeling/factory.py`

### Other components

- **User**
- **Input Image**
- **Tokenizer**
- **Image Processor**
- **Generated Answer**
- **Training Data**
- **LFM Language Decoder**

## Connections

- User → Inference: submits prompt
- Input Image → Inference: provides image
- Inference → Tokenizer: formats prompt
- Inference → Image Processor: processes image
- Inference → Firebolt Language Model: generates with
- Firebolt Language Model → Generated Answer: returns tokens
- Firebolt Language Model → LFM Language Decoder: decodes with
- Firebolt Language Model → SigLIP Vision Encoder: encodes image
- Firebolt Language Model → Vision Projector: projects features
- Firebolt Language Model → Multimodal Fuser: fuses modalities
- SigLIP Vision Encoder → Grid Vision Encoder: supports grid features
- Multimodal Fuser → Token-Grid Modulator: selects modulator
- Token-Grid Modulator → S4 Sequence Model: uses sequence model
- Token-Grid Modulator → S4D Sequence Model: supports alternative
- Component Registry → Multimodal Fuser: builds registered fuser
- Component Registry → Vision Projector: builds registered projector
- Training Data → Dataset Loader: supplies examples
- Dataset Loader → Training Loop: batches data
- Training Loop → Firebolt Language Model: optimizes
- Training Loop → Perplexity Metric: evaluates perplexity
- Distributed Training → Dataset Loader: loads batches
- Distributed Training → Firebolt Language Model: optimizes