# dbiswas55/vlmflow architecture

> Architecture diagram of the GitHub repository dbiswas55/vlmflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dbiswas55/vlmflow
- Repository: https://github.com/dbiswas55/vlmflow

## Overview

vlmflow runs config-defined, multi-step vision-language experiments: the task entry point resolves settings, prepares ScienceQA samples, sends each prompt step to a vlmhub model, parses intermediate outputs for later steps, then scores and saves each result while recording resumable progress. The graph separates task orchestration, dataset handling, workflow execution, and persistence. The README documents the model-provider options and configurable workflows; unsampled configuration and submodule internals are not expanded beyond those documented capabilities.

Main entry points: [`src/scienceqa_task.py`](https://github.com/dbiswas55/vlmflow/blob/master/src/scienceqa_task.py) (ScienceQA task), [`src/engine/runner.py`](https://github.com/dbiswas55/vlmflow/blob/master/src/engine/runner.py) (Workflow runner), [`src/utils/config.py`](https://github.com/dbiswas55/vlmflow/blob/master/src/utils/config.py) (Experiment config), [`src/dataset/scienceqa.py`](https://github.com/dbiswas55/vlmflow/blob/master/src/dataset/scienceqa.py) (Sample preparation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Task orchestration"]
  node_task["ScienceQA task<br/>[scienceqa_task.py]"]
  node_config["Experiment config<br/>[config.py]"]
  node_workflow_config["Workflow definitions<br/>[experiment.json]"]
end

subgraph group_data["ScienceQA data"]
  node_dataset["Sample preparation<br/>[scienceqa.py]"]
  node_sample_store[("Local sample files<br/>[scienceqa.py]")]
  node_tag_builder["Prompt tag builder<br/>[scienceqa.py]"]
end

subgraph group_workflow["Workflow engine"]
  node_prompts["Prompt templates"]
  node_runner["Workflow runner<br/>[runner.py]"]
  node_model["vlmhub model"]
  node_parser["Response parser<br/>[parser.py]"]
end

subgraph group_persistence["Results and state"]
  node_result_writer["Result scoring<br/>[scienceqa_task.py]"]
  node_io["JSON writer<br/>[io_utils.py]"]
  node_run_state[("Results and progress<br/>[scienceqa_task.py]")]
  node_debug["Input debug dumps<br/>[runner.py]"]
end

node_user(("Experiment operator"))
node_hf["Hugging Face dataset"]
node_providers["Model providers"]

node_user -->|"starts run"| node_task
node_task -->|"loads settings"| node_config
node_config -->|"reads config"| node_workflow_config
node_task -->|"prepares samples"| node_dataset
node_dataset -->|"reads and writes"| node_sample_store
node_dataset -->|"downloads samples"| node_hf
node_task -->|"runs workflow"| node_runner
node_runner -->|"loads steps"| node_config
node_config -->|"resolves prompt paths"| node_prompts
node_runner -->|"builds tag values"| node_tag_builder
node_runner -->|"generates response"| node_model
node_model -.->|"uses configured backend"| node_providers
node_runner -->|"parses responses"| node_parser
node_runner -->|"dumps model input"| node_debug
node_task -->|"scores sample"| node_result_writer
node_result_writer -->|"writes JSON"| node_io
node_task -->|"resumes and records"| node_run_state
node_result_writer -->|"saves result"| node_run_state
node_task -->|"writes progress"| node_io

click node_task "https://github.com/dbiswas55/vlmflow/blob/master/src/scienceqa_task.py"
click node_config "https://github.com/dbiswas55/vlmflow/blob/master/src/utils/config.py"
click node_workflow_config "https://github.com/dbiswas55/vlmflow/blob/master/configs/experiment.json"
click node_dataset "https://github.com/dbiswas55/vlmflow/blob/master/src/dataset/scienceqa.py"
click node_sample_store "https://github.com/dbiswas55/vlmflow/blob/master/src/dataset/scienceqa.py"
click node_tag_builder "https://github.com/dbiswas55/vlmflow/blob/master/src/dataset/scienceqa.py"
click node_prompts "https://github.com/dbiswas55/vlmflow/tree/master/src/prompts/scienceqa"
click node_runner "https://github.com/dbiswas55/vlmflow/blob/master/src/engine/runner.py"
click node_model "https://github.com/dbiswas55/vlmflow/tree/master/src/_libs/vlmhub"
click node_parser "https://github.com/dbiswas55/vlmflow/blob/master/src/engine/parser.py"
click node_result_writer "https://github.com/dbiswas55/vlmflow/blob/master/src/scienceqa_task.py"
click node_io "https://github.com/dbiswas55/vlmflow/blob/master/src/engine/io_utils.py"
click node_run_state "https://github.com/dbiswas55/vlmflow/blob/master/src/scienceqa_task.py"
click node_debug "https://github.com/dbiswas55/vlmflow/blob/master/src/engine/runner.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_task,node_config,node_workflow_config toneBlue
class node_dataset,node_sample_store,node_tag_builder toneAmber
class node_prompts,node_runner,node_model,node_parser toneMint
class node_result_writer,node_io,node_run_state,node_debug toneRose
class node_user,node_hf,node_providers toneIndigo
```

## Components

### Task orchestration

- [ScienceQA task](https://github.com/dbiswas55/vlmflow/blob/master/src/scienceqa_task.py): `src/scienceqa_task.py`
- [Experiment config](https://github.com/dbiswas55/vlmflow/blob/master/src/utils/config.py): `src/utils/config.py`
- [Workflow definitions](https://github.com/dbiswas55/vlmflow/blob/master/configs/experiment.json): `configs/experiment.json`

### ScienceQA data

- [Sample preparation](https://github.com/dbiswas55/vlmflow/blob/master/src/dataset/scienceqa.py): `src/dataset/scienceqa.py`
- [Local sample files](https://github.com/dbiswas55/vlmflow/blob/master/src/dataset/scienceqa.py): `src/dataset/scienceqa.py`
- [Prompt tag builder](https://github.com/dbiswas55/vlmflow/blob/master/src/dataset/scienceqa.py): `src/dataset/scienceqa.py`

### Workflow engine

- [Prompt templates](https://github.com/dbiswas55/vlmflow/tree/master/src/prompts/scienceqa): `src/prompts/scienceqa`
- [Workflow runner](https://github.com/dbiswas55/vlmflow/blob/master/src/engine/runner.py): `src/engine/runner.py`
- [vlmhub model](https://github.com/dbiswas55/vlmflow/tree/master/src/_libs/vlmhub): `src/_libs/vlmhub`
- [Response parser](https://github.com/dbiswas55/vlmflow/blob/master/src/engine/parser.py): `src/engine/parser.py`

### Results and state

- [Result scoring](https://github.com/dbiswas55/vlmflow/blob/master/src/scienceqa_task.py): `src/scienceqa_task.py`
- [JSON writer](https://github.com/dbiswas55/vlmflow/blob/master/src/engine/io_utils.py): `src/engine/io_utils.py`
- [Results and progress](https://github.com/dbiswas55/vlmflow/blob/master/src/scienceqa_task.py): `src/scienceqa_task.py`
- [Input debug dumps](https://github.com/dbiswas55/vlmflow/blob/master/src/engine/runner.py): `src/engine/runner.py`

### Other components

- **Experiment operator**
- **Hugging Face dataset**
- **Model providers**

## Connections

- Experiment operator → ScienceQA task: starts run (evidence: `README.md`)
- ScienceQA task → Experiment config: loads settings (evidence: `src/scienceqa_task.py`)
- Experiment config → Workflow definitions: reads config (evidence: `src/utils/config.py`)
- ScienceQA task → Sample preparation: prepares samples (evidence: `src/scienceqa_task.py`)
- Sample preparation → Local sample files: reads and writes (evidence: `src/dataset/scienceqa.py`)
- Sample preparation → Hugging Face dataset: downloads samples (evidence: `src/dataset/scienceqa.py`)
- ScienceQA task → Workflow runner: runs workflow (evidence: `src/scienceqa_task.py`)
- Workflow runner → Experiment config: loads steps (evidence: `src/engine/runner.py`)
- Experiment config → Prompt templates: resolves prompt paths (evidence: `README.md`)
- Workflow runner → Prompt tag builder: builds tag values (evidence: `src/engine/runner.py`)
- Workflow runner → vlmhub model: generates response (evidence: `src/engine/runner.py`)
- vlmhub model → Model providers: uses configured backend
- Workflow runner → Response parser: parses responses (evidence: `src/engine/runner.py`)
- Workflow runner → Input debug dumps: dumps model input (evidence: `src/engine/runner.py`)
- ScienceQA task → Result scoring: scores sample (evidence: `src/scienceqa_task.py`)
- Result scoring → JSON writer: writes JSON (evidence: `src/scienceqa_task.py`)
- ScienceQA task → Results and progress: resumes and records (evidence: `src/scienceqa_task.py`)
- Result scoring → Results and progress: saves result (evidence: `src/scienceqa_task.py`)
- ScienceQA task → JSON writer: writes progress (evidence: `src/scienceqa_task.py`)