How huyquoctrinh/firebolt-vl works
Firebolt-VL is a vision-language model for image-grounded prompt answering, with inference and training entry points. The graph follows the documented image-to-visual-feature, multimodal-fusion, and language-generation stages, plus training data preparation and optimization. Some implementation excerpts are partial; subsystem links not established by the sampled source or README are omitted.
Main entry points: model.py Firebolt Language Model · infer.py Inference · train.py Training Loop · cossm.py Token-Grid Modulator
Components 21
User Workflows
- Inference
infer.py
Training Pipeline
- Dataset Loader
dataset.py - Training Loop
train.py - Distributed Training
train_ddp.py - Perplexity Metric
modeling/metric.py
Multimodal Model
- Firebolt Language Model
modeling/model.py
Model Components
- SigLIP Vision Encoder
modeling/vision_encoder/siglip.py - Grid Vision Encoder
modeling/vision_encoder/siglip_grid.py - Vision Projector
modeling/projector.py - Multimodal Fuser
modeling/fuser.py - Token-Grid Modulator
modeling/cossm.py - S4 Sequence Model
modeling/ssm/s4.py - S4D Sequence Model
modeling/ssm/s4d.py - Component Registry
modeling/factory.py
Other components
- User
- Input Image
- Tokenizer
- Image Processor
- Generated Answer
- Training Data
- LFM Language Decoder
Connections 22
- User → Inference: submits prompt
- Input Image → Inference: provides image
- Inference → Tokenizer: formats prompt
- Inference → Image Processor: processes image
- Inference → Firebolt Language Model: generates with
- Firebolt Language Model → Generated Answer: returns tokens
- Firebolt Language Model → LFM Language Decoder: decodes with
- Firebolt Language Model → SigLIP Vision Encoder: encodes image
- Firebolt Language Model → Vision Projector: projects features
- Firebolt Language Model → Multimodal Fuser: fuses modalities
- SigLIP Vision Encoder → Grid Vision Encoder: supports grid features
- Multimodal Fuser → Token-Grid Modulator: selects modulator
- Token-Grid Modulator → S4 Sequence Model: uses sequence model
- Token-Grid Modulator → S4D Sequence Model: supports alternative
- Component Registry → Multimodal Fuser: builds registered fuser
- Component Registry → Vision Projector: builds registered projector
- Training Data → Dataset Loader: supplies examples
- Dataset Loader → Training Loop: batches data
- Training Loop → Firebolt Language Model: optimizes
- Training Loop → Perplexity Metric: evaluates perplexity
- Distributed Training → Dataset Loader: loads batches
- Distributed Training → Firebolt Language Model: optimizes