# ut-austin-robin/datamil architecture

> Architecture diagram of the GitHub repository ut-austin-robin/datamil, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ut-austin-robin/datamil
- Repository: https://github.com/ut-austin-robin/datamil

## Overview

DataMIL computes trajectory-level datamodel scores for robot imitation datasets, selects examples, then trains and evaluates policies on the selected data. The principal workflow covers dataset conversion/loading, benchmark-specific selection, and policy training/evaluation; internal metagradient/VJP machinery computes the selection signal. The README documents the end-to-end MetaWorld and LIBERO workflows. Some implementation links are based on SOURCE INDEX references because those files were not excerpted.

Main entry points: [`datamil/metagradients/do_mw_data_selection_parallel.py`](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/do_mw_data_selection_parallel.py) (MetaWorld selector), [`datamil/metagradients/do_libero_data_selection_parallel.py`](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/do_libero_data_selection_parallel.py) (LIBERO selector), [`scripts`](https://github.com/ut-austin-robin/datamil/tree/main/scripts) (Dataset conversion), [`datamil/metagradients/core/vjp.py`](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/vjp.py) (VJP estimation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_rlds["LIBERO RLDS builders"]
  node_conversion["Dataset conversion"]
  node_trajectory["Trajectory transforms<br/>[dataset.py]"]
  node_mwloader["MetaWorld MDS loader"]
  node_liberoloader["LIBERO MDS loader"]
end

subgraph group_selection["Datamodel selection"]
  node_mwselect["MetaWorld selector"]
  node_liberoselect["LIBERO selector"]
  node_scores["Selected data scores"]
end

subgraph group_training["Policy training"]
  node_mwmodel["MetaWorld policy<br/>[mw_model_jax.py]"]
  node_octomodel["Octo policy<br/>[make_octo_model.py]"]
  node_actionhead["Octo action head"]
  node_cotrain["LIBERO cotraining<br/>[cotraining.py]"]
  node_mwtrain["MetaWorld selected training"]
  node_evaluation["Benchmark evaluation"]
end

subgraph group_runtime["Core runtime"]
  node_streaming["Replay data loading<br/>[dataloading.py]"]
  node_vjp["VJP estimation<br/>[vjp.py]"]
  node_vjpblocks["VJP blocks<br/>[vjp_blocks.py]"]
  node_robodm["RoboDM VJP<br/>[vjp_robodm.py]"]
  node_adam["Adam optimizer<br/>[adam.py]"]
  node_interpolation["Optimizer schedules<br/>[interpolation.py]"]
  node_adam_schedule["Learning-rate schedule<br/>[schedules.py]"]
  node_runtimeutils["JAX and tree utilities<br/>[utils.py]"]
  node_dlpack["Device transfer<br/>[dlpack.py]"]
  node_config["Selection configuration<br/>[flags_config.py]"]
end

node_researcher(("Researcher"))
node_robotdata["Robot datasets"]

node_researcher -->|"provides"| node_robotdata
node_robotdata -->|"converts to RLDS"| node_rlds
node_robotdata -->|"converts to MDS"| node_conversion
node_rlds -->|"converts to MDS"| node_conversion
node_conversion -->|"prepares trajectories"| node_trajectory
node_trajectory -->|"prepares batches"| node_cotrain
node_conversion -->|"produces MDS input"| node_mwloader
node_conversion -->|"produces MDS input"| node_liberoloader
node_mwloader -->|"loads batches"| node_mwselect
node_liberoloader -->|"loads batches"| node_liberoselect
node_mwselect -->|"selects trajectories"| node_scores
node_liberoselect -->|"selects trajectories"| node_scores
node_mwselect -->|"uses policy"| node_mwmodel
node_liberoselect -->|"uses policy"| node_octomodel
node_mwselect -->|"computes metagradients"| node_robodm
node_liberoselect -->|"computes metagradients"| node_robodm
node_scores -->|"selects training data"| node_mwtrain
node_scores -->|"selects training data"| node_cotrain
node_mwtrain -->|"trains policy"| node_mwmodel
node_cotrain -->|"trains policy"| node_octomodel
node_mwtrain -->|"evaluates policy"| node_evaluation
node_cotrain -->|"evaluates policy"| node_evaluation
node_evaluation -->|"reports results"| node_researcher
node_mwselect -->|"requests replay batches"| node_streaming
node_liberoselect -->|"requests replay batches"| node_streaming
node_robodm -->|"calls VJP"| node_vjp
node_vjpblocks -->|"uses VJP"| node_vjp
node_vjp -->|"consumes batchers"| node_streaming
node_vjp -->|"transfers arrays"| node_dlpack
node_vjp -->|"uses tree utilities"| node_runtimeutils
node_robodm -->|"uses optimizer"| node_adam
node_adam -->|"builds schedule"| node_adam_schedule
node_mwselect -->|"uses schedules"| node_interpolation
node_liberoselect -->|"uses schedules"| node_interpolation
node_mwselect -->|"reads configuration"| node_config
node_liberoselect -->|"reads configuration"| node_config
node_octomodel -->|"configures action head"| node_actionhead

click node_rlds "https://github.com/ut-austin-robin/datamil/tree/main/rlds_utils/rlds_dataset_builders"
click node_conversion "https://github.com/ut-austin-robin/datamil/tree/main/scripts"
click node_trajectory "https://github.com/ut-austin-robin/datamil/blob/main/datamil/data/dataset.py"
click node_mwloader "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/make_mw_loader_mds.py"
click node_liberoloader "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/make_libero_loader_mds.py"
click node_mwselect "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/do_mw_data_selection_parallel.py"
click node_liberoselect "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/do_libero_data_selection_parallel.py"
click node_mwmodel "https://github.com/ut-austin-robin/datamil/blob/main/datamil/models/mw_model_jax.py"
click node_octomodel "https://github.com/ut-austin-robin/datamil/blob/main/datamil/models/make_octo_model.py"
click node_actionhead "https://github.com/ut-austin-robin/datamil/blob/main/datamil/models/octo_action_head.py"
click node_cotrain "https://github.com/ut-austin-robin/datamil/blob/main/scripts/cotraining.py"
click node_mwtrain "https://github.com/ut-austin-robin/datamil/blob/main/scripts/selected_training_mw.py"
click node_evaluation "https://github.com/ut-austin-robin/datamil/tree/main/datamil"
click node_streaming "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/dataloading.py"
click node_vjp "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/vjp.py"
click node_vjpblocks "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/vjp_blocks.py"
click node_robodm "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/vjp_robodm.py"
click node_adam "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/optimizers/adam.py"
click node_interpolation "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/optimizers/interpolation.py"
click node_adam_schedule "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/optimizers/schedules.py"
click node_runtimeutils "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/utils.py"
click node_dlpack "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/dlpack.py"
click node_config "https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/flags_config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_rlds,node_conversion,node_trajectory,node_mwloader,node_liberoloader toneBlue
class node_mwselect,node_liberoselect,node_scores toneAmber
class node_mwmodel,node_octomodel,node_actionhead,node_cotrain,node_mwtrain,node_evaluation toneMint
class node_streaming,node_vjp,node_vjpblocks,node_robodm,node_adam,node_interpolation,node_adam_schedule,node_runtimeutils,node_dlpack,node_config toneRose
class node_researcher,node_robotdata toneIndigo
```

## Components

### Data preparation

- [LIBERO RLDS builders](https://github.com/ut-austin-robin/datamil/tree/main/rlds_utils/rlds_dataset_builders): `rlds_utils/rlds_dataset_builders`
- [Dataset conversion](https://github.com/ut-austin-robin/datamil/tree/main/scripts): `scripts`
- [Trajectory transforms](https://github.com/ut-austin-robin/datamil/blob/main/datamil/data/dataset.py): `datamil/data/dataset.py`
- [MetaWorld MDS loader](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/make_mw_loader_mds.py): `datamil/metagradients/make_mw_loader_mds.py`
- [LIBERO MDS loader](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/make_libero_loader_mds.py): `datamil/metagradients/make_libero_loader_mds.py`

### Datamodel selection

- [MetaWorld selector](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/do_mw_data_selection_parallel.py): `datamil/metagradients/do_mw_data_selection_parallel.py`
- [LIBERO selector](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/do_libero_data_selection_parallel.py): `datamil/metagradients/do_libero_data_selection_parallel.py`
- **Selected data scores**

### Policy training

- [MetaWorld policy](https://github.com/ut-austin-robin/datamil/blob/main/datamil/models/mw_model_jax.py): `datamil/models/mw_model_jax.py`
- [Octo policy](https://github.com/ut-austin-robin/datamil/blob/main/datamil/models/make_octo_model.py): `datamil/models/make_octo_model.py`
- [Octo action head](https://github.com/ut-austin-robin/datamil/blob/main/datamil/models/octo_action_head.py): `datamil/models/octo_action_head.py`
- [LIBERO cotraining](https://github.com/ut-austin-robin/datamil/blob/main/scripts/cotraining.py): `scripts/cotraining.py`
- [MetaWorld selected training](https://github.com/ut-austin-robin/datamil/blob/main/scripts/selected_training_mw.py): `scripts/selected_training_mw.py`
- [Benchmark evaluation](https://github.com/ut-austin-robin/datamil/tree/main/datamil): `datamil`

### Core runtime

- [Replay data loading](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/dataloading.py): `datamil/metagradients/core/dataloading.py`
- [VJP estimation](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/vjp.py): `datamil/metagradients/core/vjp.py`
- [VJP blocks](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/vjp_blocks.py): `datamil/metagradients/core/vjp_blocks.py`
- [RoboDM VJP](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/vjp_robodm.py): `datamil/metagradients/core/vjp_robodm.py`
- [Adam optimizer](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/optimizers/adam.py): `datamil/metagradients/core/optimizers/adam.py`
- [Optimizer schedules](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/optimizers/interpolation.py): `datamil/metagradients/core/optimizers/interpolation.py`
- [Learning-rate schedule](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/optimizers/schedules.py): `datamil/metagradients/core/optimizers/schedules.py`
- [JAX and tree utilities](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/utils.py): `datamil/metagradients/core/utils.py`
- [Device transfer](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/core/dlpack.py): `datamil/metagradients/core/dlpack.py`
- [Selection configuration](https://github.com/ut-austin-robin/datamil/blob/main/datamil/metagradients/flags_config.py): `datamil/metagradients/flags_config.py`

### Other components

- **Researcher**
- **Robot datasets**

## Connections

- Researcher → Robot datasets: provides
- Robot datasets → LIBERO RLDS builders: converts to RLDS
- Robot datasets → Dataset conversion: converts to MDS
- LIBERO RLDS builders → Dataset conversion: converts to MDS
- Dataset conversion → Trajectory transforms: prepares trajectories (evidence: `scripts/libero_rlds_to_mds.py`)
- Trajectory transforms → LIBERO cotraining: prepares batches (evidence: `scripts/cotraining.py`)
- Dataset conversion → MetaWorld MDS loader: produces MDS input
- Dataset conversion → LIBERO MDS loader: produces MDS input
- MetaWorld MDS loader → MetaWorld selector: loads batches (evidence: `datamil/metagradients/do_mw_data_selection_parallel.py`)
- LIBERO MDS loader → LIBERO selector: loads batches (evidence: `datamil/metagradients/do_libero_data_selection_parallel.py`)
- MetaWorld selector → Selected data scores: selects trajectories
- LIBERO selector → Selected data scores: selects trajectories
- MetaWorld selector → MetaWorld policy: uses policy (evidence: `datamil/metagradients/do_mw_data_selection_parallel.py`)
- LIBERO selector → Octo policy: uses policy (evidence: `datamil/metagradients/do_libero_data_selection_parallel.py`)
- MetaWorld selector → RoboDM VJP: computes metagradients (evidence: `datamil/metagradients/do_mw_data_selection_parallel.py`)
- LIBERO selector → RoboDM VJP: computes metagradients (evidence: `datamil/metagradients/do_libero_data_selection_parallel.py`)
- Selected data scores → MetaWorld selected training: selects training data
- Selected data scores → LIBERO cotraining: selects training data
- MetaWorld selected training → MetaWorld policy: trains policy (evidence: `scripts/selected_training_mw.py`)
- LIBERO cotraining → Octo policy: trains policy
- MetaWorld selected training → Benchmark evaluation: evaluates policy (evidence: `scripts/selected_training_mw.py`)
- LIBERO cotraining → Benchmark evaluation: evaluates policy (evidence: `scripts/cotraining.py`)
- Benchmark evaluation → Researcher: reports results
- MetaWorld selector → Replay data loading: requests replay batches (evidence: `datamil/metagradients/do_mw_data_selection_parallel.py`)
- LIBERO selector → Replay data loading: requests replay batches (evidence: `datamil/metagradients/do_libero_data_selection_parallel.py`)
- RoboDM VJP → VJP estimation: calls VJP (evidence: `datamil/metagradients/core/vjp_robodm.py`)
- VJP blocks → VJP estimation: uses VJP (evidence: `datamil/metagradients/core/vjp_blocks.py`)
- VJP estimation → Replay data loading: consumes batchers (evidence: `datamil/metagradients/core/vjp.py`)
- VJP estimation → Device transfer: transfers arrays (evidence: `datamil/metagradients/core/vjp.py`)
- VJP estimation → JAX and tree utilities: uses tree utilities (evidence: `datamil/metagradients/core/vjp.py`)
- RoboDM VJP → Adam optimizer: uses optimizer (evidence: `datamil/metagradients/core/vjp_robodm.py`)
- Adam optimizer → Learning-rate schedule: builds schedule (evidence: `datamil/metagradients/core/optimizers/adam.py`)
- MetaWorld selector → Optimizer schedules: uses schedules (evidence: `datamil/metagradients/do_mw_data_selection_parallel.py`)
- LIBERO selector → Optimizer schedules: uses schedules (evidence: `datamil/metagradients/do_libero_data_selection_parallel.py`)
- MetaWorld selector → Selection configuration: reads configuration (evidence: `datamil/metagradients/do_mw_data_selection_parallel.py`)
- LIBERO selector → Selection configuration: reads configuration (evidence: `datamil/metagradients/do_libero_data_selection_parallel.py`)
- Octo policy → Octo action head: configures action head (evidence: `datamil/models/make_octo_model.py`)
