How khushi-070906/v2t-graph works
V-to-T Graph turns RGB frames into a navigation-aware semantic graph, ranks objects by affordance, distance, and heading, then produces spatial cues and spoken instructions for blind or low-vision users. The principal workflow is coordinated by pipeline.py; the live-camera variants repeat it continuously, with the tracked variant adding temporal confirmation and approach estimates. Depth calibration and evaluation are supporting product capabilities. The excerpts and README establish the main stages; some live-loop and evaluation internals are unsampled, so their uncertain connections are omitted.
Main entry points: pipeline.py Single-frame pipeline · live_camera_loop_tracked.py Tracked camera loop · detect_depth.py Detection and depth · graph_builder.py Spatial graph
Components 19
Vision and geometry
- Detection and depth
detect_depth.py - Spatial graph
graph_builder.py
Attention selection
- Affordance pruning
pruning.py
Guidance outputs
- Haptic and audio encoding
encoders.py - Instruction planner
navigation_planner.py - Offline speech
speech_output.py
Runtime entry points
- Single-frame pipeline
pipeline.py - Stateless camera loop
live_camera_loop.py - Tracked camera loop
live_camera_loop_tracked.py - Object tracking
tracker.py
Calibration and evaluation
- Camera calibration
calibrate_depth.py - Image calibration
calibrate_depth_from_images.py - Batch evaluation
eval/batch_eval.py - Simulated walk evaluation
eval/simulate_walk.py - Compression metrics
compression_ratio.py - Pruning visualization
visualize_pruning.py - Calibration parameters
calibration.json
Other components
- Low-vision user
- RGB frame
Connections 31
- RGB frame → Single-frame pipeline: provides frame
README.md - Single-frame pipeline → Detection and depth: runs detection
pipeline.py - Detection and depth → Spatial graph: returns detections
graph_builder.py - Single-frame pipeline → Spatial graph: builds graph
pipeline.py - Single-frame pipeline → Affordance pruning: prunes graph
README.md - Single-frame pipeline → Haptic and audio encoding: encodes graph
pipeline.py - Single-frame pipeline → Instruction planner: plans instructions
pipeline.py - Single-frame pipeline → Compression metrics: computes metrics
pipeline.py - Single-frame pipeline → Offline speech: speaks optionally
README.md - RGB frame → Stateless camera loop: camera feed
live_camera_loop.py - Stateless camera loop → Detection and depth: runs detection
live_camera_loop.py - Stateless camera loop → Offline speech: speaks instructions
live_camera_loop.py - RGB frame → Tracked camera loop: camera feed
live_camera_loop_tracked.py - Tracked camera loop → Detection and depth: runs detection
live_camera_loop_tracked.py - Tracked camera loop → Object tracking: updates tracks
live_camera_loop_tracked.py - Tracked camera loop → Spatial graph: builds graph
live_camera_loop_tracked.py - Tracked camera loop → Affordance pruning: prunes graph
live_camera_loop_tracked.py - Tracked camera loop → Haptic and audio encoding: encodes graph
live_camera_loop_tracked.py - Tracked camera loop → Instruction planner: plans instructions
live_camera_loop_tracked.py - Tracked camera loop → Offline speech: speaks instructions
live_camera_loop_tracked.py - Camera calibration → Calibration parameters: writes fit
calibrate_depth.py - Image calibration → Calibration parameters: writes fit
- Calibration parameters → Detection and depth: corrects depth
README.md - Batch evaluation → Single-frame pipeline: runs per frame
eval/batch_eval.py - Batch evaluation → Compression metrics: aggregates metrics
eval/batch_eval.py - Simulated walk evaluation → Spatial graph: builds test graph
eval/simulate_walk.py - Simulated walk evaluation → Affordance pruning: compares pruning
eval/simulate_walk.py - Pruning visualization → Detection and depth: loads detections
visualize_pruning.py - Pruning visualization → Spatial graph: visualizes graph
visualize_pruning.py - Pruning visualization → Affordance pruning: visualizes pruning
visualize_pruning.py - Low-vision user → Offline speech: receives cues