# wrainw/vrscout_agent architecture

> Architecture diagram of the GitHub repository wrainw/vrscout_agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wrainw/vrscout_agent
- Repository: https://github.com/wrainw/vrscout_agent

## Overview

1. **Repository role in the larger system**
   - This repo is the **Python + model side** of VRScout’s VR agent stack.
   - It does not contain the data-collection Unity project or the OpenVR virtual driver itself; it sits between those external components by doing **policy inference, dataset preprocessing, and training**.

2. **Runtime inference path**
   - The main production entry point is `unity_vr_inference.py`.
   - At runtime, it loads a trained ACT-style checkpoint from `--ckpt_dir` and serves actions to Unity over **TCP** on `127.0.0.1:5556` by default.
   - Unity sends observations such as **poses, images, and optional audio**; Python returns action outputs for the VR application.

3. **Unity-to-Python boundary**
   - The Unity-side client scripts under `unity/` are the observation/action bridge: `InferencePoseClient.cs` is the primary runtime client, with related helpers for calibration, recording, reward tracking, and OpenVR device control.
   - These scripts are the boundary where real VR state is captured from Unity and translated into the action format expected by the Python policy.
   - The README notes these scripts are largely **legacy/deprecated**, with current Unity integration living in the separate orchestration repo.

4. **Policy and model subsystem**
   - `policy.py`, `distributions.py`, and `detr/` contain the learned controller implementation.
   - `detr/models/` provides the ACT/DETR-inspired transformer backbone, transformer blocks, and output heads used for sequence-based action prediction.
   - The architecture is centered on **chunked action prediction** rather than single-step reactive control.

5. **Action-chunking and temporal smoothing**
   - The repo’s key behavioral feature is **temporal aggregation** of predicted action chunks.
   - `unity_vr_inference.py` supports `--temporal_agg` and optional `--use_dynamic_window`, which change how overlapping action chunks are blended online.
   - This is an important architectural boundary: the policy predicts chunks, while the inference server decides how aggressively to smooth or adapt them before sending to Unity.

6. **Adaptive normalization and runtime state**
   - The system persists normalization/runtime adaptation state, including **dynamic-window statistics** in `normalization_stats.pkl` under the checkpoint directory.
   - That means the inference service is not stateless; it carries learned or tuned normalization behavior across runs.
   - Runtime toggle synchronization is bidirectional: Unity can change windowing/smoothing settings live, and Python reflects current state back in responses.

7. **Optional audio feature branch**
   - If enabled, the inference server can ingest audio and produce embeddings via **YAMNet through TensorFlow Hub**.
   - This is an optional multimodal path alongside pose/image inputs.
   - It affects the observation pipeline but is isolated from the core action-chunking logic.

8. **Training and evaluation pipeline**
   - `imitate_episodes.py` is the main ACT-style training script, and `evaluate_episodes.py` handles evaluation.
   - These scripts use the same policy/model stack as inference, but operate on recorded episodes rather than live Unity traffic.
   - The training loop is specialized for VR settings and likely consumes the converted VR datasets produced by the preprocessing scripts.

9. **Dataset conversion and preprocessing**
   - `convert_vr_dataset.py`, `convert_new_vr_dataset.py`, and the batch wrappers prepare recorded VR sessions for training.
   - `batch_convert_new_vr_datasets.py` and `batch_convert_vr_datasets.py` indicate the repository supports converting many recorded sessions in bulk.
   - Utilities like `custom_collate.py`, `depth_zfpy_preprocessor.py`, and `my_utils.py` suggest the data pipeline handles nontrivial batching and modality-specific preprocessing.

10. **Environment and simulation helpers**
   - `sim_env.py` and `ee_sim_env.py` define simulation-side abstractions used for training or evaluation outside the live Unity loop.
   - These form a separate environment boundary from the real-time VR runtime, letting the policy be exercised in non-live settings.

11. **Support scripts and data handling**
   - `action_io.py` and `action_utils.py` centralize action formatting and I/O concerns.
   - `helper_scripts/` contains offline utilities such as image copying and dataset counting, which support dataset hygiene rather than runtime control.
   - `constants.py` likely holds shared configuration values used across training, preprocessing, and inference.

12. **External dependencies and architectural implications**
   - The repo depends on **PyTorch** for the core model stack, with DETR-derived transformer code embedded locally under `detr/`.
   - It also relies on the external Unity/orchestration repo and the OpenVR virtual driver repo for the actual VR device emulation path.
   - The separation is important: this repository decides what actions to take, while other repos handle how those actions are injected into the VR runtime.

13. **Overall data flow**
   - Recorded VR episodes are converted into training-ready datasets.
   - The ACT/DETR-style policy is trained and checkpointed.
   - At runtime, Unity streams observations to the Python inference server.
   - Python optionally enriches observations with audio features, runs the policy, applies temporal aggregation/dynamic window logic, and returns actions for Unity/OpenVR execution.

Main entry points: [`imitate_episodes.py`](https://github.com/wrainw/vrscout_agent/blob/Main/imitate_episodes.py) (Train episodes), [`unity_vr_inference.py`](https://github.com/wrainw/vrscout_agent/blob/Main/unity_vr_inference.py) (Inference server), [`policy.py`](https://github.com/wrainw/vrscout_agent/blob/Main/policy.py) (Policy), [`sim_env.py`](https://github.com/wrainw/vrscout_agent/blob/Main/sim_env.py) (Sim env)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime inference"]
  node_unity_vr_inference["Inference server<br/>python runtime"]
end

subgraph group_model["Policy model"]
  node_policy["Policy<br/>controller<br/>[policy.py]"]
  node_distributions["Distributions<br/>sampling heads<br/>[distributions.py]"]
  node_detr_models["DETR models<br/>transformer backbone"]
end

subgraph group_data["Data pipeline"]
  node_convert_vr_dataset["Convert VR<br/>dataset converter"]
  node_convert_new_vr_dataset["Convert new VR<br/>dataset converter"]
  node_batch_convert["Batch convert<br/>bulk preprocessing"]
  node_batch_convert_new["Batch new convert<br/>bulk preprocessing"]
  node_custom_collate["Collate<br/>batching<br/>[custom_collate.py]"]
  node_depth_zfpy_preprocessor["Depth prep<br/>preprocessor"]
end

subgraph group_train["Train & eval"]
  node_imitate_episodes["Train episodes<br/>training loop"]
  node_evaluate_episodes["Evaluate episodes<br/>evaluation"]
end

subgraph group_sim["Simulation"]
  node_sim_env["Sim env<br/>simulation env<br/>[sim_env.py]"]
  node_ee_sim_env["EE sim env<br/>simulation env<br/>[ee_sim_env.py]"]
end

subgraph group_unity["Unity bridge"]
  node_inference_pose_client["Pose client<br/>Unity bridge"]
  node_act_policy_client["Policy client<br/>Unity bridge<br/>[ACTPolicyClient.cs]"]
  node_vr_helpers["VR helpers<br/>Unity support"]
end

subgraph group_utils["Shared utilities"]
  node_action_io["Action I/O<br/>protocol utils<br/>[action_io.py]"]
  node_action_utils["Action utils<br/>action helpers<br/>[action_utils.py]"]
  node_my_utils["Shared utils<br/>misc helpers<br/>[my_utils.py]"]
end

node_unity_vr_inference -->|"formats actions"| node_action_io
node_unity_vr_inference -->|"adapts actions"| node_action_utils
node_unity_vr_inference -->|"runs policy"| node_policy
node_policy -->|"parameterizes"| node_distributions
node_policy -->|"backed by"| node_detr_models
node_imitate_episodes -->|"trains"| node_policy
node_evaluate_episodes -->|"evaluates"| node_policy
node_imitate_episodes -->|"batches data"| node_custom_collate
node_imitate_episodes -->|"preprocesses depth"| node_depth_zfpy_preprocessor
node_convert_vr_dataset -->|"prepares samples"| node_custom_collate
node_convert_new_vr_dataset -->|"prepares samples"| node_custom_collate
node_batch_convert -->|"orchestrates"| node_convert_vr_dataset
node_batch_convert_new -->|"orchestrates"| node_convert_new_vr_dataset
node_sim_env -->|"extends"| node_ee_sim_env
node_imitate_episodes -->|"uses"| node_sim_env
node_evaluate_episodes -->|"uses"| node_sim_env
node_inference_pose_client -->|"relies on"| node_vr_helpers
node_act_policy_client -->|"relies on"| node_vr_helpers
node_inference_pose_client -->|"streams to"| node_unity_vr_inference
node_unity_vr_inference -->|"shared helpers"| node_my_utils
node_imitate_episodes -->|"shared helpers"| node_my_utils

click node_unity_vr_inference "https://github.com/wrainw/vrscout_agent/blob/Main/unity_vr_inference.py"
click node_action_io "https://github.com/wrainw/vrscout_agent/blob/Main/action_io.py"
click node_action_utils "https://github.com/wrainw/vrscout_agent/blob/Main/action_utils.py"
click node_policy "https://github.com/wrainw/vrscout_agent/blob/Main/policy.py"
click node_distributions "https://github.com/wrainw/vrscout_agent/blob/Main/distributions.py"
click node_detr_models "https://github.com/wrainw/vrscout_agent/tree/Main/detr/models"
click node_convert_vr_dataset "https://github.com/wrainw/vrscout_agent/blob/Main/convert_vr_dataset.py"
click node_convert_new_vr_dataset "https://github.com/wrainw/vrscout_agent/blob/Main/convert_new_vr_dataset.py"
click node_batch_convert "https://github.com/wrainw/vrscout_agent/blob/Main/batch_convert_vr_datasets.py"
click node_batch_convert_new "https://github.com/wrainw/vrscout_agent/blob/Main/batch_convert_new_vr_datasets.py"
click node_custom_collate "https://github.com/wrainw/vrscout_agent/blob/Main/custom_collate.py"
click node_depth_zfpy_preprocessor "https://github.com/wrainw/vrscout_agent/blob/Main/depth_zfpy_preprocessor.py"
click node_imitate_episodes "https://github.com/wrainw/vrscout_agent/blob/Main/imitate_episodes.py"
click node_evaluate_episodes "https://github.com/wrainw/vrscout_agent/blob/Main/evaluate_episodes.py"
click node_sim_env "https://github.com/wrainw/vrscout_agent/blob/Main/sim_env.py"
click node_ee_sim_env "https://github.com/wrainw/vrscout_agent/blob/Main/ee_sim_env.py"
click node_inference_pose_client "https://github.com/wrainw/vrscout_agent/blob/Main/unity/InferencePoseClient.cs"
click node_act_policy_client "https://github.com/wrainw/vrscout_agent/blob/Main/unity/ACTPolicyClient.cs"
click node_vr_helpers "https://github.com/wrainw/vrscout_agent/blob/Main/unity/VRAvatarCalibrator.cs"
click node_my_utils "https://github.com/wrainw/vrscout_agent/blob/Main/my_utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_unity_vr_inference,node_action_io,node_action_utils,node_my_utils toneBlue
class node_policy,node_distributions,node_detr_models toneAmber
class node_convert_vr_dataset,node_convert_new_vr_dataset,node_batch_convert,node_batch_convert_new,node_custom_collate,node_depth_zfpy_preprocessor toneMint
class node_imitate_episodes,node_evaluate_episodes toneRose
class node_sim_env,node_ee_sim_env toneIndigo
class node_inference_pose_client,node_act_policy_client,node_vr_helpers toneTeal
```

## Components

### Runtime inference

Live Unity-to-Python action serving path.

- [Inference server](https://github.com/wrainw/vrscout_agent/blob/Main/unity_vr_inference.py): python runtime · Main live entry point; loads checkpoints, receives Unity observations, and returns actions over TCP. · `unity_vr_inference.py`

### Policy model

ACT/DETR-style controller and distribution logic.

- [Policy](https://github.com/wrainw/vrscout_agent/blob/Main/policy.py): controller · Top-level policy wrapper used for inference and training. · `policy.py`
- [Distributions](https://github.com/wrainw/vrscout_agent/blob/Main/distributions.py): sampling heads · Action distribution and output parameterization used by the policy. · `distributions.py`
- [DETR models](https://github.com/wrainw/vrscout_agent/tree/Main/detr/models): transformer backbone · ACT/DETR-inspired backbone, transformer, and heads for sequence action prediction. · `detr/models`

### Data pipeline

Dataset conversion, batching, and preprocessing.

- [Convert VR](https://github.com/wrainw/vrscout_agent/blob/Main/convert_vr_dataset.py): dataset converter · Converts recorded VR sessions into training-ready format. · `convert_vr_dataset.py`
- [Convert new VR](https://github.com/wrainw/vrscout_agent/blob/Main/convert_new_vr_dataset.py): dataset converter · Updated VR dataset conversion path for newer recordings. · `convert_new_vr_dataset.py`
- [Batch convert](https://github.com/wrainw/vrscout_agent/blob/Main/batch_convert_vr_datasets.py): bulk preprocessing · Wrappers for converting many VR datasets in bulk. · `batch_convert_vr_datasets.py`
- [Batch new convert](https://github.com/wrainw/vrscout_agent/blob/Main/batch_convert_new_vr_datasets.py): bulk preprocessing · Bulk conversion wrapper for the newer VR dataset format. · `batch_convert_new_vr_datasets.py`
- [Collate](https://github.com/wrainw/vrscout_agent/blob/Main/custom_collate.py): batching · Custom batching logic for multimodal training samples. · `custom_collate.py`
- [Depth prep](https://github.com/wrainw/vrscout_agent/blob/Main/depth_zfpy_preprocessor.py): preprocessor · Depth-specific preprocessing for compressed or specialized inputs. · `depth_zfpy_preprocessor.py`

### Train & eval

Offline episode training and evaluation.

- [Train episodes](https://github.com/wrainw/vrscout_agent/blob/Main/imitate_episodes.py): training loop · Main ACT-style imitation training script over recorded episodes. · `imitate_episodes.py`
- [Evaluate episodes](https://github.com/wrainw/vrscout_agent/blob/Main/evaluate_episodes.py): evaluation · Runs offline evaluation on recorded or prepared episodes. · `evaluate_episodes.py`

### Simulation

Non-live environment abstractions.

- [Sim env](https://github.com/wrainw/vrscout_agent/blob/Main/sim_env.py): simulation env · Training/evaluation environment abstraction outside the live Unity loop. · `sim_env.py`
- [EE sim env](https://github.com/wrainw/vrscout_agent/blob/Main/ee_sim_env.py): simulation env · End-effector simulation variant used alongside the main simulated environment. · `ee_sim_env.py`

### Unity bridge

Legacy Unity-side clients and VR helpers.

- [Pose client](https://github.com/wrainw/vrscout_agent/blob/Main/unity/InferencePoseClient.cs): Unity bridge · Primary Unity runtime client for streaming observations and receiving actions. · `unity/InferencePoseClient.cs`
- [Policy client](https://github.com/wrainw/vrscout_agent/blob/Main/unity/ACTPolicyClient.cs): Unity bridge · Legacy Unity client for policy interaction and action dispatch. · `unity/ACTPolicyClient.cs`
- [VR helpers](https://github.com/wrainw/vrscout_agent/blob/Main/unity/VRAvatarCalibrator.cs): Unity support · Calibration, recording, reward tracking, and OpenVR device control helpers. · `unity/VRAvatarCalibrator.cs`

### Shared utilities

Cross-cutting action and config helpers.

- [Action I/O](https://github.com/wrainw/vrscout_agent/blob/Main/action_io.py): protocol utils · Encodes and decodes action messages exchanged between policy and runtime clients. · `action_io.py`
- [Action utils](https://github.com/wrainw/vrscout_agent/blob/Main/action_utils.py): action helpers · Shared action formatting and transformation helpers. · `action_utils.py`
- [Shared utils](https://github.com/wrainw/vrscout_agent/blob/Main/my_utils.py): misc helpers · Repository-wide helper functions used across preprocessing and runtime paths. · `my_utils.py`

## Connections

- Inference server → Action I/O: formats actions · Uses the action protocol helpers to serialize and parse messages.
- Inference server → Action utils: adapts actions · Applies shared action transformations before returning outputs.
- Inference server → Policy: runs policy · Loads the trained controller and produces action chunks.
- Policy → Distributions: parameterizes · Uses distribution heads to form action outputs.
- Policy → DETR models: backed by · Delegates sequence modeling to the DETR/ACT transformer stack.
- Train episodes → Policy: trains · Optimizes the same policy stack used in inference.
- Evaluate episodes → Policy: evaluates · Runs policy checkpoints on offline episodes.
- Train episodes → Collate: batches data · Uses custom collation for training samples.
- Train episodes → Depth prep: preprocesses depth · Applies modality-specific preprocessing during training.
- Convert VR → Collate: prepares samples · Produces training-ready records aligned with the batching path.
- Convert new VR → Collate: prepares samples · Produces training-ready records aligned with the batching path.
- Batch convert → Convert VR: orchestrates · Wraps conversion for many datasets.
- Batch new convert → Convert new VR: orchestrates · Wraps conversion for many datasets.
- Sim env → EE sim env: extends · Provides a related simulation boundary for evaluation and training.
- Train episodes → Sim env: uses · Can train or validate through the simulation boundary.
- Evaluate episodes → Sim env: uses · Exercises policies in simulated or offline environments.
- Pose client → VR helpers: relies on · Uses calibration, recording, and device control support.
- Policy client → VR helpers: relies on · Legacy client path shares the same Unity-side support utilities.
- Pose client → Inference server: streams to · Sends observations and receives actions from the Python server.
- Inference server → Shared utils: shared helpers · Uses common helper functions for runtime processing.
- Train episodes → Shared utils: shared helpers · Reuses repository-wide utilities during training.
