# yuxuanjiang123/lerobotdatalabel architecture

> Architecture diagram of the GitHub repository yuxuanjiang123/lerobotdatalabel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yuxuanjiang123/lerobotdatalabel
- Repository: https://github.com/yuxuanjiang123/lerobotdatalabel

## Overview

LerobotDataLabel combines automated LeRobot episode labeling, segmentation and quality filtering with a Flask-based manual annotation application. The automated workflow configures dataset paths, generates visual inputs, uses a selected VLM backend to label or classify episodes, then materializes segments or applies quality gates. Manual users browse and annotate episodes through the web app, which reads dataset/video data and persists annotations. README capabilities are included, but several referenced pipeline scripts and configuration directories are absent from the supplied tree; their internal wiring is therefore not expanded.

Main entry points: [`manual_labeling/web/app.py`](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/web/app.py) (Annotation web app), [`scripts/label_core.py`](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/label_core.py) (Labeling core), [`scripts/classify_microwave_phase.py`](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/classify_microwave_phase.py) (Phase classifier), [`microwave_split/boundary_pipeline.py`](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/microwave_split/boundary_pipeline.py) (Boundary detection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_labeling["VLM Labeling"]
  node_label_cli["Label entrypoints"]
  node_label_core["Labeling core<br/>[label_core.py]"]
  node_backend_api{{"DashScope API"}}
  node_backend_local["Local Qwen VL"]
  node_backend_vllm{{"vLLM service"}}
  node_contact_sheets["Contact sheets"]
  node_phase_classify["Phase classifier"]
end

subgraph group_split["Segmentation"]
  node_phase_results["Frame index export"]
  node_phase_slice["Phase slicing<br/>[slice_by_phase.py]"]
  node_boundary["Boundary detection"]
  node_materialize["Segment materialization<br/>[materialize.py]"]
  node_subtask_slice["Subtask video slicing"]
end

subgraph group_quality["Quality Screening"]
  node_clean["Dead-time cleanup"]
  node_movement["Action quality scoring<br/>[runner.py]"]
  node_filter["HQ dataset filter"]
  node_quantity["Semantic success gate"]
end

subgraph group_manual["Manual Annotation"]
  node_web["Annotation web app<br/>[app.py]"]
  node_browser["Annotator UI<br/>[annotator.js]"]
  node_loader["Dataset loader<br/>[data_loader.py]"]
  node_video["Video handler<br/>[video_handler.py]"]
  node_annotations[("Annotation store")]
  node_manager["Dataset manager<br/>[dataset_manager.py]"]
  node_v01["LeRobot v0.1 export"]
end

subgraph group_shared["Shared Data Services"]
  node_paths["Path resolver<br/>[paths.py]"]
end

node_user(("Dataset operator"))
node_dataset[("LeRobot dataset")]

node_user -->|"sets paths"| node_paths
node_paths -->|"configures inputs"| node_label_cli
node_user -->|"starts labeling"| node_contact_sheets
node_contact_sheets -->|"reads videos"| node_dataset
node_label_cli -->|"dispatches mode"| node_label_core
node_label_core -.->|"calls API"| node_backend_api
node_label_core -.->|"runs inference"| node_backend_local
node_label_core -.->|"calls service"| node_backend_vllm
node_label_core -->|"reads episode inputs"| node_dataset
node_label_core -->|"produces boundaries"| node_subtask_slice
node_user -->|"starts classification"| node_phase_classify
node_phase_classify -->|"reads episodes"| node_dataset
node_phase_classify -.->|"requests phase"| node_backend_api
node_phase_classify -->|"writes results"| node_phase_results
node_phase_results -->|"provides frame index"| node_phase_slice
node_phase_slice -->|"reads and slices"| node_dataset
node_boundary -->|"requests boundaries"| node_backend_local
node_boundary -->|"samples video frames"| node_dataset
node_boundary -->|"returns manifest"| node_materialize
node_user -->|"starts cleanup"| node_clean
node_clean -->|"rewrites segments"| node_dataset
node_user -->|"starts scoring"| node_movement
node_movement -->|"reads action data"| node_dataset
node_movement -->|"provides scores"| node_filter
node_filter -->|"writes HQ subset"| node_dataset
node_user -->|"starts judgement"| node_quantity
node_quantity -.->|"requests judgement"| node_backend_vllm
node_quantity -->|"reads episodes"| node_dataset
node_user -->|"opens annotator"| node_web
node_web -->|"serves UI"| node_browser
node_browser -->|"calls APIs"| node_web
node_web -->|"loads episode data"| node_loader
node_web -->|"serves video details"| node_video
node_web -->|"loads and saves"| node_annotations
node_web -->|"manages dataset"| node_manager
node_manager -->|"syncs task labels"| node_v01
node_loader -->|"reads dataset"| node_dataset
node_video -->|"reads videos"| node_dataset
node_annotations -->|"persists labels"| node_dataset

click node_paths "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/paths.py"
click node_label_cli "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/label_episode_fixed_subtasks_vllm.py"
click node_label_core "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/label_core.py"
click node_backend_local "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/microwave_split/qwen3vl_inference.py"
click node_contact_sheets "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/gen_contact_sheets.py"
click node_phase_classify "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/classify_microwave_phase.py"
click node_phase_results "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/results_to_frames_jsonl.py"
click node_phase_slice "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/slice_by_phase.py"
click node_boundary "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/microwave_split/boundary_pipeline.py"
click node_materialize "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/microwave_split/materialize.py"
click node_subtask_slice "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/segment_subtask_videos.py"
click node_clean "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/clean_dead_segments.py"
click node_movement "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/movement_detection_core/runner.py"
click node_filter "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/filter_hq_dataset.py"
click node_quantity "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/quantity_judge_core.py"
click node_web "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/web/app.py"
click node_browser "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/static/js/annotator.js"
click node_loader "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/core/data_loader.py"
click node_video "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/core/video_handler.py"
click node_annotations "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/core/annotation_store.py"
click node_manager "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/core/dataset_manager.py"
click node_v01 "https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/core/v01_format_handler.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_label_cli,node_label_core,node_backend_api,node_backend_local,node_backend_vllm,node_contact_sheets,node_phase_classify toneBlue
class node_phase_results,node_phase_slice,node_boundary,node_materialize,node_subtask_slice,node_dataset toneAmber
class node_clean,node_movement,node_filter,node_quantity toneMint
class node_web,node_browser,node_loader,node_video,node_annotations,node_manager,node_v01 toneRose
class node_paths,node_user toneIndigo
```

## Components

### VLM Labeling

- [Label entrypoints](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/label_episode_fixed_subtasks_vllm.py): `scripts/label_episode_fixed_subtasks_vllm.py`
- [Labeling core](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/label_core.py): `scripts/label_core.py`
- **DashScope API**
- [Local Qwen VL](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/microwave_split/qwen3vl_inference.py): `microwave_split/qwen3vl_inference.py`
- **vLLM service**
- [Contact sheets](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/gen_contact_sheets.py): `scripts/gen_contact_sheets.py`
- [Phase classifier](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/classify_microwave_phase.py): `scripts/classify_microwave_phase.py`

### Segmentation

- [Frame index export](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/results_to_frames_jsonl.py): `scripts/results_to_frames_jsonl.py`
- [Phase slicing](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/slice_by_phase.py): `scripts/slice_by_phase.py`
- [Boundary detection](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/microwave_split/boundary_pipeline.py): `microwave_split/boundary_pipeline.py`
- [Segment materialization](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/microwave_split/materialize.py): `microwave_split/materialize.py`
- [Subtask video slicing](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/segment_subtask_videos.py): `scripts/segment_subtask_videos.py`

### Quality Screening

- [Dead-time cleanup](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/clean_dead_segments.py): `scripts/clean_dead_segments.py`
- [Action quality scoring](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/movement_detection_core/runner.py): `movement_detection_core/runner.py`
- [HQ dataset filter](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/filter_hq_dataset.py): `scripts/filter_hq_dataset.py`
- [Semantic success gate](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/scripts/quantity_judge_core.py): `scripts/quantity_judge_core.py`

### Manual Annotation

- [Annotation web app](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/web/app.py): `manual_labeling/web/app.py`
- [Annotator UI](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/static/js/annotator.js): `manual_labeling/static/js/annotator.js`
- [Dataset loader](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/core/data_loader.py): `manual_labeling/core/data_loader.py`
- [Video handler](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/core/video_handler.py): `manual_labeling/core/video_handler.py`
- [Annotation store](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/core/annotation_store.py): `manual_labeling/core/annotation_store.py`
- [Dataset manager](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/core/dataset_manager.py): `manual_labeling/core/dataset_manager.py`
- [LeRobot v0.1 export](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/manual_labeling/core/v01_format_handler.py): `manual_labeling/core/v01_format_handler.py`

### Shared Data Services

- [Path resolver](https://github.com/yuxuanjiang123/lerobotdatalabel/blob/main/paths.py): `paths.py`

### Other components

- **Dataset operator**
- **LeRobot dataset**

## Connections

- Dataset operator → Path resolver: sets paths
- Path resolver → Label entrypoints: configures inputs
- Dataset operator → Contact sheets: starts labeling
- Contact sheets → LeRobot dataset: reads videos
- Label entrypoints → Labeling core: dispatches mode
- Labeling core → DashScope API: calls API
- Labeling core → Local Qwen VL: runs inference
- Labeling core → vLLM service: calls service
- Labeling core → LeRobot dataset: reads episode inputs
- Labeling core → Subtask video slicing: produces boundaries
- Dataset operator → Phase classifier: starts classification
- Phase classifier → LeRobot dataset: reads episodes
- Phase classifier → DashScope API: requests phase
- Phase classifier → Frame index export: writes results
- Frame index export → Phase slicing: provides frame index
- Phase slicing → LeRobot dataset: reads and slices
- Boundary detection → Local Qwen VL: requests boundaries
- Boundary detection → LeRobot dataset: samples video frames
- Boundary detection → Segment materialization: returns manifest
- Dataset operator → Dead-time cleanup: starts cleanup
- Dead-time cleanup → LeRobot dataset: rewrites segments
- Dataset operator → Action quality scoring: starts scoring
- Action quality scoring → LeRobot dataset: reads action data
- Action quality scoring → HQ dataset filter: provides scores
- HQ dataset filter → LeRobot dataset: writes HQ subset
- Dataset operator → Semantic success gate: starts judgement
- Semantic success gate → vLLM service: requests judgement
- Semantic success gate → LeRobot dataset: reads episodes
- Dataset operator → Annotation web app: opens annotator
- Annotation web app → Annotator UI: serves UI
- Annotator UI → Annotation web app: calls APIs
- Annotation web app → Dataset loader: loads episode data
- Annotation web app → Video handler: serves video details
- Annotation web app → Annotation store: loads and saves
- Annotation web app → Dataset manager: manages dataset
- Dataset manager → LeRobot v0.1 export: syncs task labels
- Dataset loader → LeRobot dataset: reads dataset
- Video handler → LeRobot dataset: reads videos
- Annotation store → LeRobot dataset: persists labels