# fanwu66/vlmidentification architecture

> Architecture diagram of the GitHub repository fanwu66/vlmidentification, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fanwu66/vlmidentification
- Repository: https://github.com/fanwu66/vlmidentification

## Overview

VLMIdentification is a research pipeline for matching a passenger description to people in robotaxi image sequences. The principal workflow reads dataset sequence metadata, summarizes the requested passenger, adaptively detects and crops people, describes each crop with a vision-language model, compares descriptions, then annotates frames and evaluates detections against ground truth. The README also documents dataset and model preparation; their detailed implementations are not shown in the sampled sources, so those branches are represented at the documented directory level. The graph reflects calls visible in main.py and evaluation code; it does not imply unsampled integrations.

Main entry points: [`main.py`](https://github.com/fanwu66/vlmidentification/blob/main/main.py) (Sequence runner), [`module/detection.py`](https://github.com/fanwu66/vlmidentification/blob/main/module/detection.py) (Adaptive person detection), [`module/summary.py`](https://github.com/fanwu66/vlmidentification/blob/main/module/summary.py) (Passenger summary), [`module/vlm.py`](https://github.com/fanwu66/vlmidentification/blob/main/module/vlm.py) (Crop description)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Identification workflow"]
  node_runner["Sequence runner<br/>[main.py]"]
  node_sequence_data["Sequence metadata<br/>[main.py]"]
  node_passenger_summary["Passenger summary<br/>[summary.py]"]
end

subgraph group_models["Model inference"]
  node_adaptive_detection["Adaptive person detection<br/>[detection.py]"]
  node_crop_adjustment["Box expansion helper<br/>[tool.py]"]
  node_crop_description["Crop description<br/>[vlm.py]"]
  node_text_similarity["Description similarity<br/>[similarity.py]"]
end

subgraph group_assessment["Output and assessment"]
  node_annotation_output["Frame annotation<br/>[img.py]"]
  node_ground_truth_eval["Ground-truth evaluation<br/>[eval.py]"]
  node_evaluation_report["Metrics and report<br/>[main.py]"]
end

subgraph group_preparation["Dataset and model setup"]
  node_dataset_preparation["Dataset preparation<br/>[prepare_dataset.md]"]
  node_model_preparation["Model preparation<br/>[prepare_model.md]"]
end

node_operator(("Research operator"))
node_dataset[("nuScenes images and labels")]
node_qwen["Qwen-VL model"]
node_yolo["YOLOv10 detector"]

node_operator -->|"runs"| node_runner
node_runner -->|"reads"| node_sequence_data
node_sequence_data -->|"loads"| node_dataset
node_runner -->|"summarizes"| node_passenger_summary
node_passenger_summary -->|"queries"| node_qwen
node_runner -->|"detects people"| node_adaptive_detection
node_adaptive_detection -->|"sets threshold"| node_qwen
node_adaptive_detection -->|"runs detection"| node_yolo
node_runner -->|"describes crops"| node_crop_description
node_crop_description -->|"queries"| node_qwen
node_runner -->|"compares text"| node_text_similarity
node_text_similarity -->|"encodes text"| node_qwen
node_runner -->|"draws boxes"| node_annotation_output
node_runner -->|"checks overlap"| node_ground_truth_eval
node_ground_truth_eval -->|"reads labels"| node_dataset
node_runner -->|"aggregates metrics"| node_evaluation_report
node_dataset_preparation -->|"prepares"| node_dataset
node_model_preparation -->|"prepares"| node_qwen
node_adaptive_detection -.->|"uses helper"| node_crop_adjustment

click node_runner "https://github.com/fanwu66/vlmidentification/blob/main/main.py"
click node_sequence_data "https://github.com/fanwu66/vlmidentification/blob/main/main.py"
click node_passenger_summary "https://github.com/fanwu66/vlmidentification/blob/main/module/summary.py"
click node_adaptive_detection "https://github.com/fanwu66/vlmidentification/blob/main/module/detection.py"
click node_crop_adjustment "https://github.com/fanwu66/vlmidentification/blob/main/tool/tool.py"
click node_crop_description "https://github.com/fanwu66/vlmidentification/blob/main/module/vlm.py"
click node_text_similarity "https://github.com/fanwu66/vlmidentification/blob/main/module/similarity.py"
click node_annotation_output "https://github.com/fanwu66/vlmidentification/blob/main/module/img.py"
click node_ground_truth_eval "https://github.com/fanwu66/vlmidentification/blob/main/eval/eval.py"
click node_evaluation_report "https://github.com/fanwu66/vlmidentification/blob/main/main.py"
click node_dataset_preparation "https://github.com/fanwu66/vlmidentification/blob/main/docs/prepare_dataset.md"
click node_model_preparation "https://github.com/fanwu66/vlmidentification/blob/main/docs/prepare_model.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_sequence_data,node_passenger_summary toneBlue
class node_adaptive_detection,node_crop_adjustment,node_crop_description,node_text_similarity,node_dataset toneAmber
class node_annotation_output,node_ground_truth_eval,node_evaluation_report toneMint
class node_dataset_preparation,node_model_preparation toneRose
class node_operator,node_qwen,node_yolo toneIndigo
```

## Components

### Identification workflow

- [Sequence runner](https://github.com/fanwu66/vlmidentification/blob/main/main.py): `main.py`
- [Sequence metadata](https://github.com/fanwu66/vlmidentification/blob/main/main.py): `main.py`
- [Passenger summary](https://github.com/fanwu66/vlmidentification/blob/main/module/summary.py): `module/summary.py`

### Model inference

- [Adaptive person detection](https://github.com/fanwu66/vlmidentification/blob/main/module/detection.py): `module/detection.py`
- [Box expansion helper](https://github.com/fanwu66/vlmidentification/blob/main/tool/tool.py): `tool/tool.py`
- [Crop description](https://github.com/fanwu66/vlmidentification/blob/main/module/vlm.py): `module/vlm.py`
- [Description similarity](https://github.com/fanwu66/vlmidentification/blob/main/module/similarity.py): `module/similarity.py`

### Output and assessment

- [Frame annotation](https://github.com/fanwu66/vlmidentification/blob/main/module/img.py): `module/img.py`
- [Ground-truth evaluation](https://github.com/fanwu66/vlmidentification/blob/main/eval/eval.py): `eval/eval.py`
- [Metrics and report](https://github.com/fanwu66/vlmidentification/blob/main/main.py): `main.py`

### Dataset and model setup

- [Dataset preparation](https://github.com/fanwu66/vlmidentification/blob/main/docs/prepare_dataset.md): `docs/prepare_dataset.md`
- [Model preparation](https://github.com/fanwu66/vlmidentification/blob/main/docs/prepare_model.md): `docs/prepare_model.md`

### Other components

- **Research operator**
- **nuScenes images and labels**
- **Qwen-VL model**
- **YOLOv10 detector**

## Connections

- Research operator → Sequence runner: runs (evidence: `README.md`)
- Sequence runner → Sequence metadata: reads (evidence: `main.py`)
- Sequence metadata → nuScenes images and labels: loads (evidence: `main.py`)
- Sequence runner → Passenger summary: summarizes (evidence: `main.py`)
- Passenger summary → Qwen-VL model: queries (evidence: `module/summary.py`)
- Sequence runner → Adaptive person detection: detects people (evidence: `main.py`)
- Adaptive person detection → Qwen-VL model: sets threshold (evidence: `module/detection.py`)
- Adaptive person detection → YOLOv10 detector: runs detection (evidence: `module/detection.py`)
- Sequence runner → Crop description: describes crops (evidence: `main.py`)
- Crop description → Qwen-VL model: queries (evidence: `module/vlm.py`)
- Sequence runner → Description similarity: compares text (evidence: `main.py`)
- Description similarity → Qwen-VL model: encodes text (evidence: `module/similarity.py`)
- Sequence runner → Frame annotation: draws boxes (evidence: `main.py`)
- Sequence runner → Ground-truth evaluation: checks overlap (evidence: `main.py`)
- Ground-truth evaluation → nuScenes images and labels: reads labels (evidence: `eval/eval.py`)
- Sequence runner → Metrics and report: aggregates metrics (evidence: `main.py`)
- Dataset preparation → nuScenes images and labels: prepares (evidence: `README.md`)
- Model preparation → Qwen-VL model: prepares (evidence: `README.md`)
- Adaptive person detection → Box expansion helper: uses helper