How fanwu66/vlmidentification works
VLMIdentification is a research pipeline for matching a passenger description to people in robotaxi image sequences. The principal workflow reads dataset sequence metadata, summarizes the requested passenger, adaptively detects and crops people, describes each crop with a vision-language model, compares descriptions, then annotates frames and evaluates detections against ground truth. The README also documents dataset and model preparation; their detailed implementations are not shown in the sampled sources, so those branches are represented at the documented directory level. The graph reflects calls visible in main.py and evaluation code; it does not imply unsampled integrations.
Main entry points: main.py Sequence runner · detection.py Adaptive person detection · summary.py Passenger summary · vlm.py Crop description
Components 16
Identification workflow
- Sequence runner
main.py - Sequence metadata
main.py - Passenger summary
module/summary.py
Model inference
- Adaptive person detection
module/detection.py - Box expansion helper
tool/tool.py - Crop description
module/vlm.py - Description similarity
module/similarity.py
Output and assessment
- Frame annotation
module/img.py - Ground-truth evaluation
eval/eval.py - Metrics and report
main.py
Dataset and model setup
- Dataset preparation
docs/prepare_dataset.md - Model preparation
docs/prepare_model.md
Other components
- Research operator
- nuScenes images and labels
- Qwen-VL model
- YOLOv10 detector
Connections 19
- Research operator → Sequence runner: runs
README.md - Sequence runner → Sequence metadata: reads
main.py - Sequence metadata → nuScenes images and labels: loads
main.py - Sequence runner → Passenger summary: summarizes
main.py - Passenger summary → Qwen-VL model: queries
module/summary.py - Sequence runner → Adaptive person detection: detects people
main.py - Adaptive person detection → Qwen-VL model: sets threshold
module/detection.py - Adaptive person detection → YOLOv10 detector: runs detection
module/detection.py - Sequence runner → Crop description: describes crops
main.py - Crop description → Qwen-VL model: queries
module/vlm.py - Sequence runner → Description similarity: compares text
main.py - Description similarity → Qwen-VL model: encodes text
module/similarity.py - Sequence runner → Frame annotation: draws boxes
main.py - Sequence runner → Ground-truth evaluation: checks overlap
main.py - Ground-truth evaluation → nuScenes images and labels: reads labels
eval/eval.py - Sequence runner → Metrics and report: aggregates metrics
main.py - Dataset preparation → nuScenes images and labels: prepares
README.md - Model preparation → Qwen-VL model: prepares
README.md - Adaptive person detection → Box expansion helper: uses helper