# opentelevision/television architecture

> Architecture diagram of the GitHub repository opentelevision/television, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opentelevision/television
- Repository: https://github.com/opentelevision/television

## Overview

##### Project Type and Purpose
- This is a robotics teleoperation system with integrated imitation learning components. It supports immersive teleoperation using visual feedback (both local and network streaming via WebXR) and imitation learning for training robot policies.
- The codebase includes modules for training policies (deep learning models under act/detr), teleoperation control (teleop directory), simulation deployment (scripts and assets), and robot description assets (URDF and mesh files in assets).

##### Main Components
a. Imitation Learning & Policy Training Module (act/detr & related scripts):
- Components include training scripts (imitate_episodes.py), model definitions (files inside act/detr/models such as detr_vae.py, transformer.py, backbone.py, etc.), and utilities (act/detr/util).
- The training pipeline uses datasets processed via scripts (post_process.py, replay_demo.py) and eventually exports a JIT model for deployment.

b. Teleoperation Module (teleop):
- Contains core teleoperation logic, including visualization (TeleVision.py), data pre-processing (Preprocessor.py), and hardware interfacing (dynamixel subdirectory with drivers, robot interface classes).
- Also handles network aspects (webrtc folder for client-side JavaScript & browser interface) to allow immersive VR experiences on devices like the VisionPro and Meta Quest3.

c. Simulation and Deployment Components (scripts & assets):
- Assets, including URDF files, meshes, launch files, and CMake configuration for simulation environments (e.g., assets/h1_inspire) and robot hand models (assets/inspire_hand).
- These files suggest integration with simulation tools like Gazebo or Isaac Gym, along with ROS-like configurations (package.xml and launch files).

d. External Dependencies & Integration Points:
- External libraries and SDKs include the ZED SDK and its Python API (for stereoscopic vision), Isaac Gym for simulated environments, and ngrok/WebXR for establishing streaming tunnels.
- The conda environment (conda_env.yaml) and requirements.txt direct you on managing these dependencies.

##### Interactions and Data Flow
- Data Flow: The teleoperation front-end captures visual and control inputs (from VR devices or standard browsers) and passes them to the TeleVision module within teleop. This module communicates with underlying drivers (dynamixel) to control the robot hardware or its simulated counterpart.
- The imitation learning module (located in act/detr) is responsible for processing recorded episodes (using imitate_episodes.py) and training deep learning models. Once trained, the model is saved (optionally via JIT) and can be deployed in simulation (via scripts/deploy_sim.py) or even be used to autonomously adjust control policies.
- The assets (URDF and meshes) support both simulation and actual visualization of the robots. They’re referenced by Gazebo/ROS launch files and simulation scripts.

##### Architectural Patterns and Design Principles
- Modularity is a clear design principle here. Teleoperation, imitation learning, simulation deployment, and asset management are separated into distinct folders.
- The project embraces a layered architecture where:
- The lower level encapsulates the physical robot models and control interfaces (dynamixel drivers, URDF simulation assets).
- The mid-level includes policy training and inference modules.
- The top level handles interfacing with end-users via multi-platform visual feedback (using web-based components, WebXR, and VR).
- A clear separation of concerns provides the ease of updating individual modules (e.g., switching the teleop streaming mechanism between local and network streaming).

## Diagram (Mermaid)

```mermaid
graph TD
    %% Frontend / Teleoperation Interface
    subgraph "Frontend / Teleoperation Interface"
        USER["User/Sensor Input"]:::external
        TV["TeleVision (teleop)"]:::teleop
        WS["WebRTC (teleop/webrtc)"]:::teleop
        PP["Preprocessor (teleop)"]:::teleop
    end

    %% Middleware / Control & Processing
    subgraph "Middleware / Control & Processing"
        DYN["Dynamixel Controllers (teleop/dynamixel)"]:::teleop
    end

    %% Data & Training Pipeline
    subgraph "Data & Training Pipeline"
        IMT["Imitate Episodes (act/imitate_episodes.py)"]:::training
        MODELS["Model Definitions (act/detr/models)"]:::training
        UTILS["Utilities (act/detr/util)"]:::training
        PMC["Policy Module (act/policy.py)"]:::training
    end

    %% Simulation & Deployment Components
    subgraph "Simulation & Deployment"
        DS["Deploy Simulation (scripts/deploy_sim.py)"]:::simulation
        PA["Plot Action (scripts/plot_action.py)"]:::simulation
        PP2["Post Process (scripts/post_process.py)"]:::simulation
        RD["Replay Demo (scripts/replay_demo.py)"]:::simulation
        ASSET1["Robot Assets (assets/h1_inspire)"]:::simulation
        ASSET2["Robot Hand Models (assets/inspire_hand)"]:::simulation
    end

    %% External Dependencies & Environment Setup
    subgraph "External Dependencies & Environment Setup"
        ENV1["Conda Environment (act/conda_env.yaml)"]:::external
        ENV2["Requirements (requirements.txt)"]:::external
    end

    %% Connections
    USER -->|"sends_input"| TV
    USER -->|"initiates_stream"| WS

    TV -->|"sends_control"| DYN
    PP -->|"preprocesses_data"| DYN
    WS -->|"streams_video"| TV

    %% Teleoperation influencing simulation
    DYN -->|"executes_commands"| DS

    %% Simulation feedback to training pipeline
    DS -->|"logs_data"| PP2
    PP2 -->|"feeds_preprocessed"| IMT
    IMT -->|"trains_models"| MODELS
    MODELS -->|"updates_policy"| PMC
    PMC -->|"applies_policy"| DS

    %% Simulation assets integration
    DS -->|"utilizes"| ASSET1
    DS -->|"utilizes"| ASSET2

    %% External dependencies integration
    ENV1 ---|"configures"| TV
    ENV1 ---|"configures"| DS
    ENV2 ---|"provides_libs"| TV
    ENV2 ---|"provides_libs"| DS

    %% Additional simulation component
    DS -->|"visualizes_results"| PA
    DS -->|"replays_demo"| RD

    %% Click Events for Imitation Learning & Policy Training Module
    click MODELS "https://github.com/opentelevision/television/tree/main/act/detr/models"
    click UTILS "https://github.com/opentelevision/television/tree/main/act/detr/util"
    click IMT "https://github.com/opentelevision/television/blob/main/act/imitate_episodes.py"
    click PMC "https://github.com/opentelevision/television/blob/main/act/policy.py"

    %% Click Events for Teleoperation Module
    click TV "https://github.com/opentelevision/television/blob/main/teleop/TeleVision.py"
    click PP "https://github.com/opentelevision/television/blob/main/teleop/Preprocessor.py"
    click DYN "https://github.com/opentelevision/television/tree/main/teleop/dynamixel"
    click WS "https://github.com/opentelevision/television/tree/main/teleop/webrtc"

    %% Click Events for Simulation and Deployment Components
    click DS "https://github.com/opentelevision/television/blob/main/scripts/deploy_sim.py"
    click PA "https://github.com/opentelevision/television/blob/main/scripts/plot_action.py"
    click PP2 "https://github.com/opentelevision/television/blob/main/scripts/post_process.py"
    click RD "https://github.com/opentelevision/television/blob/main/scripts/replay_demo.py"
    click ASSET1 "https://github.com/opentelevision/television/tree/main/assets/h1_inspire"
    click ASSET2 "https://github.com/opentelevision/television/tree/main/assets/inspire_hand"

    %% Click Events for External Dependencies & Environment
    click ENV1 "https://github.com/opentelevision/television/blob/main/act/conda_env.yaml"
    click ENV2 "https://github.com/opentelevision/television/blob/main/requirements.txt"

    %% Styles
    classDef teleop fill:#FFDDCC,stroke:#BB5544,stroke-width:2px;
    classDef training fill:#CCE5FF,stroke:#3366CC,stroke-width:2px;
    classDef simulation fill:#CCFFCC,stroke:#228B22,stroke-width:2px;
    classDef external fill:#FFF2CC,stroke:#B8860B,stroke-width:2px;
```
