How shwplus/oa-vat works
OA-VAT is an object-following system that combines target reference features, detection/tracking, and vehicle control; it also supports UnrealCV evaluation and optional diffusion-policy planning and person-identity enhancement. The primary workflow prepares a target feature file, initializes the runtime from main.py, processes video frames to track or reacquire the target, then sends motion commands. The README documents the feature-preparation and optional branches; sampled source confirms selected detector, worker, controller, and policy behavior, but not every runtime call between them.
Main entry points: main.py Following runtime · extract_target_feature_dinov3.py Feature preparation · dp_inference.py Policy inference · drone_control.py Vehicle control
Components 19
Target features
- Feature preparation
extract_target_feature_dinov3.py - DINOv3 encoder
dinov3_feature_extractor.py - Person generation
Pose2ID/IPG/inference.py - Reference features
Tracking runtime
- Following runtime
main.py - YOLO detection
ORTrack/inference_module.py - Object tracker
detectors/ortrack_tracker.py - Enhance and reacquire
tracking/workers.py
Control and planning
- Policy inference
dp/dp_inference.py - Diffusion policy
gym-unrealcv/diffusion_policy-main/diffusion_policy/policy - Vehicle control
control/drone_control.py - Simulation control
DRONE/unreal_sim_controller.py
Optional integrations
- Unreal evaluation
eval_in_unrealcv.py - UnrealCV environment
gym-unrealcv/gym_unrealcv/envs/tracking/__init__.py
Other components
- Operator
- Reference images
- Video input
- Vehicle or simulator
- Saved video
Connections 21
- Operator → Feature preparation: prepares target
- Reference images → Feature preparation: supplies images
- Feature preparation → DINOv3 encoder: extracts features
- Feature preparation → Person generation: optionally enhances
- Feature preparation → Reference features: writes features
- Operator → Following runtime: starts following
- Video input → Following runtime: provides frames
- Reference features → Following runtime: loads target features
- Following runtime → YOLO detection: initializes detection
- Following runtime → Object tracker: initializes tracking
- Following runtime → Policy inference: optionally preloads
- Following runtime → Enhance and reacquire: runs enhancement
- Enhance and reacquire → YOLO detection: detects candidates
- Following runtime → Vehicle control: sends motion commands
- Following runtime → Simulation control: controls simulation
- Policy inference → Diffusion policy: loads policy
- Policy inference → Vehicle control: converts actions
- Vehicle control → Vehicle or simulator: commands vehicle
- Simulation control → UnrealCV environment: steps environment
- Unreal evaluation → UnrealCV environment: runs episodes
- Following runtime → Saved video: writes recording