How wrainw/vrscout_agent works
1. Repository role in the larger system
- This repo is the Python + model side of VRScout’s VR agent stack.
- It does not contain the data-collection Unity project or the OpenVR virtual driver itself; it sits between those external components by doing policy inference, dataset preprocessing, and training.
2. Runtime inference path
- The main production entry point is unity_vr_inference.py.
- At runtime, it loads a trained ACT-style checkpoint from --ckpt_dir and serves actions to Unity over TCP on 127.0.0.1:5556 by default.
- Unity sends observations such as poses, images, and optional audio; Python returns action outputs for the VR application.
3. Unity-to-Python boundary
- The Unity-side client scripts under unity/ are the observation/action bridge: InferencePoseClient.cs is the primary runtime client, with related helpers for calibration, recording, reward tracking, and OpenVR device control.
- These scripts are the boundary where real VR state is captured from Unity and translated into the action format expected by the Python policy.
- The README notes these scripts are largely legacy/deprecated, with current Unity integration living in the separate orchestration repo.
4. Policy and model subsystem
- policy.py, distributions.py, and detr/ contain the learned controller implementation.
- detr/models/ provides the ACT/DETR-inspired transformer backbone, transformer blocks, and output heads used for sequence-based action prediction.
- The architecture is centered on chunked action prediction rather than single-step reactive control.
5. Action-chunking and temporal smoothing
- The repo’s key behavioral feature is temporal aggregation of predicted action chunks.
- unity_vr_inference.py supports --temporal_agg and optional --use_dynamic_window, which change how overlapping action chunks are blended online.
- This is an important architectural boundary: the policy predicts chunks, while the inference server decides how aggressively to smooth or adapt them before sending to Unity.
6. Adaptive normalization and runtime state
- The system persists normalization/runtime adaptation state, including dynamic-window statistics in normalization_stats.pkl under the checkpoint directory.
- That means the inference service is not stateless; it carries learned or tuned normalization behavior across runs.
- Runtime toggle synchronization is bidirectional: Unity can change windowing/smoothing settings live, and Python reflects current state back in responses.
7. Optional audio feature branch
- If enabled, the inference server can ingest audio and produce embeddings via YAMNet through TensorFlow Hub.
- This is an optional multimodal path alongside pose/image inputs.
- It affects the observation pipeline but is isolated from the core action-chunking logic.
8. Training and evaluation pipeline
- imitate_episodes.py is the main ACT-style training script, and evaluate_episodes.py handles evaluation.
- These scripts use the same policy/model stack as inference, but operate on recorded episodes rather than live Unity traffic.
- The training loop is specialized for VR settings and likely consumes the converted VR datasets produced by the preprocessing scripts.
9. Dataset conversion and preprocessing
- convert_vr_dataset.py, convert_new_vr_dataset.py, and the batch wrappers prepare recorded VR sessions for training.
- batch_convert_new_vr_datasets.py and batch_convert_vr_datasets.py indicate the repository supports converting many recorded sessions in bulk.
- Utilities like custom_collate.py, depth_zfpy_preprocessor.py, and my_utils.py suggest the data pipeline handles nontrivial batching and modality-specific preprocessing.
10. Environment and simulation helpers
- sim_env.py and ee_sim_env.py define simulation-side abstractions used for training or evaluation outside the live Unity loop.
- These form a separate environment boundary from the real-time VR runtime, letting the policy be exercised in non-live settings.
11. Support scripts and data handling
- action_io.py and action_utils.py centralize action formatting and I/O concerns.
- helper_scripts/ contains offline utilities such as image copying and dataset counting, which support dataset hygiene rather than runtime control.
- constants.py likely holds shared configuration values used across training, preprocessing, and inference.
12. External dependencies and architectural implications
- The repo depends on PyTorch for the core model stack, with DETR-derived transformer code embedded locally under detr/.
- It also relies on the external Unity/orchestration repo and the OpenVR virtual driver repo for the actual VR device emulation path.
- The separation is important: this repository decides what actions to take, while other repos handle how those actions are injected into the VR runtime.
13. Overall data flow
- Recorded VR episodes are converted into training-ready datasets.
- The ACT/DETR-style policy is trained and checkpointed.
- At runtime, Unity streams observations to the Python inference server.
- Python optionally enriches observations with audio features, runs the policy, applies temporal aggregation/dynamic window logic, and returns actions for Unity/OpenVR execution.
Main entry points: imitate_episodes.py Train episodes · unity_vr_inference.py Inference server · policy.py Policy · sim_env.py Sim env
Components 20
Runtime inference
Live Unity-to-Python action serving path.
- Inference server · Main live entry point; loads checkpoints, receives Unity observations, and returns actions over TCP.
unity_vr_inference.py
Policy model
ACT/DETR-style controller and distribution logic.
- Policy · Top-level policy wrapper used for inference and training.
policy.py - Distributions · Action distribution and output parameterization used by the policy.
distributions.py - DETR models · ACT/DETR-inspired backbone, transformer, and heads for sequence action prediction.
detr/models
Data pipeline
Dataset conversion, batching, and preprocessing.
- Convert VR · Converts recorded VR sessions into training-ready format.
convert_vr_dataset.py - Convert new VR · Updated VR dataset conversion path for newer recordings.
convert_new_vr_dataset.py - Batch convert · Wrappers for converting many VR datasets in bulk.
batch_convert_vr_datasets.py - Batch new convert · Bulk conversion wrapper for the newer VR dataset format.
batch_convert_new_vr_datasets.py - Collate · Custom batching logic for multimodal training samples.
custom_collate.py - Depth prep · Depth-specific preprocessing for compressed or specialized inputs.
depth_zfpy_preprocessor.py
Train & eval
Offline episode training and evaluation.
- Train episodes · Main ACT-style imitation training script over recorded episodes.
imitate_episodes.py - Evaluate episodes · Runs offline evaluation on recorded or prepared episodes.
evaluate_episodes.py
Simulation
Non-live environment abstractions.
- Sim env · Training/evaluation environment abstraction outside the live Unity loop.
sim_env.py - EE sim env · End-effector simulation variant used alongside the main simulated environment.
ee_sim_env.py
Unity bridge
Legacy Unity-side clients and VR helpers.
- Pose client · Primary Unity runtime client for streaming observations and receiving actions.
unity/InferencePoseClient.cs - Policy client · Legacy Unity client for policy interaction and action dispatch.
unity/ACTPolicyClient.cs - VR helpers · Calibration, recording, reward tracking, and OpenVR device control helpers.
unity/VRAvatarCalibrator.cs
Shared utilities
Cross-cutting action and config helpers.
- Action I/O · Encodes and decodes action messages exchanged between policy and runtime clients.
action_io.py - Action utils · Shared action formatting and transformation helpers.
action_utils.py - Shared utils · Repository-wide helper functions used across preprocessing and runtime paths.
my_utils.py
Connections 21
- Inference server → Action I/O: formats actions · Uses the action protocol helpers to serialize and parse messages.
- Inference server → Action utils: adapts actions · Applies shared action transformations before returning outputs.
- Inference server → Policy: runs policy · Loads the trained controller and produces action chunks.
- Policy → Distributions: parameterizes · Uses distribution heads to form action outputs.
- Policy → DETR models: backed by · Delegates sequence modeling to the DETR/ACT transformer stack.
- Train episodes → Policy: trains · Optimizes the same policy stack used in inference.
- Evaluate episodes → Policy: evaluates · Runs policy checkpoints on offline episodes.
- Train episodes → Collate: batches data · Uses custom collation for training samples.
- Train episodes → Depth prep: preprocesses depth · Applies modality-specific preprocessing during training.
- Convert VR → Collate: prepares samples · Produces training-ready records aligned with the batching path.
- Convert new VR → Collate: prepares samples · Produces training-ready records aligned with the batching path.
- Batch convert → Convert VR: orchestrates · Wraps conversion for many datasets.
- Batch new convert → Convert new VR: orchestrates · Wraps conversion for many datasets.
- Sim env → EE sim env: extends · Provides a related simulation boundary for evaluation and training.
- Train episodes → Sim env: uses · Can train or validate through the simulation boundary.
- Evaluate episodes → Sim env: uses · Exercises policies in simulated or offline environments.
- Pose client → VR helpers: relies on · Uses calibration, recording, and device control support.
- Policy client → VR helpers: relies on · Legacy client path shares the same Unity-side support utilities.
- Pose client → Inference server: streams to · Sends observations and receives actions from the Python server.
- Inference server → Shared utils: shared helpers · Uses common helper functions for runtime processing.
- Train episodes → Shared utils: shared helpers · Reuses repository-wide utilities during training.