# jellyho/twinvla architecture

> Architecture diagram of the GitHub repository jellyho/twinvla, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jellyho/twinvla
- Repository: https://github.com/jellyho/twinvla

## Overview

##### Project Type and Purpose
- The project is a research/experimental machine learning framework for vision-language modeling. The presence of a “twinvla” directory containing datasets, various model architectures (eagle2, internvl2_5_MPO, mobilevlmv2, etc.), and extensive simulation and training scripts indicate that this repository supports developing and evaluating multi-modal (vision + language) models.
- The README highlights installation steps and instructions for integrating with a “Simpler” environment for realistic simulation and evaluation. This makes it clear that, besides model development, the project is designed for experimentation with simulation environments.

##### Analyzing the File Structure
- Top-Level Files & Directories:
– Files like README.md, requirements.txt, setup.py, and several notebooks (blob.ipynb, kosmostest.ipynb) indicate both documentation and interactive experimental testing.
– The “rlds_generator” directory contains multiple modules (e.g., “aloha_dish_drainer”, “libero_10_no_noops_abs”, etc.). These likely serve as data/dataset generation components that either simulate environments or generate data for training.
– Scripts are organized under the “scripts” directory (for training, simulation experiments, and utility functions) and “slurm” for batch job scripts, highlighting the execution-layer design.
– The “twinvla” directory is core to the project and is divided into:
▪ “datasets”: Handles data ingestion, processing, and transformations (including specialized RLDS dataset-related functionalities).
▪ “model”: Contains submodules for base models, heads (for predictions and tokenization), and distinct modeling architectures (found under “modeling”, “singlevlas”, “twinvlas”). These modules include configuration, tokenizer utilities, and specific implementations (e.g., eagle2, internvl2_5_MPO, mobilevlmv2).
- Configuration files and batch scripts indicate a clear separation between model definitions, dataset preparation, and execution (training/simulation).

##### README Insights
- Guidance on running evaluations (via slurm scripts) shows that the system is designed to handle automated or batch executions on clusters, highlighting a production/experiment pipeline.

a. Main Components:
– Data/Dataset Layer:
- TwinVLA/datasets: Responsible for loading and processing RLDS datasets, including sub-modules for observation transformations, trajectory processing, task augmentation, and goal relabeling.
- rlds_generator modules: Specialized data generation components (e.g., “aloha_dish_drainer” and various “libero_*” modules).
– Model/Computation Layer:
- TwinVLA/model: Houses the core modeling components including base models, specialized heads (DiT_policy, diffusion heads, cross attention modules), and modeling submodules (eagle2, internvl2_5_MPO, mobilevlmv2). Also include tokenizers and single/twin variants.
– Training and Experimentation:
- Scripts: Contains training pipelines (e.g., train.py, train_single_batch*) and evaluation/experiment scripts (simpler_experiments, tabletop_experiments, simulation utilities).
- Slurm: Batch execution scripts to schedule training and benchmarking jobs.
– External Dependencies and Integration:
- Simpler Environment: An external system (SimplerEnv) that is integrated for simulation and real-to-sim experiments. Indicate this as an external module.
– Utility and Configuration:
- Scripts/configs: Environment configuration files.
- Other utility scripts (e.g., dummy.py, singlevla_gen.py) that support helper functions and possibly quick tests.

b. Relationships and Interactions:
- The “rlds_generator” modules to the “datasets” layer, indicating that data generation feeds into the data pipeline.
- From the “datasets” component to the “model” component, showing that processed data is fed into model training and inference.
- Between training/experiment scripts and the model, reflecting that these scripts invoke model training and evaluation.
- From the “slurm” scripts to the training scripts, capturing the job scheduling and automation process.

c. Architectural Patterns and Design Principles:
– Layered Architecture: Clearly separate data ingestion/processing (datasets and rlds_generator), computation (model definition, processing, and inference), and execution (training, evaluation, and batch job management).
– Modularity: Each major functionality (dataset generation, model heads, experiment scripts) is organized as a distinct Python package or module.
– Integration with external systems (SimplerEnv) shows a design pattern where the core framework is extended with simulation-specific components.

d. Technologies, Frameworks, or Libraries:
– Python is the core language.
– Package management via pip and conda.
– Jupyter Notebooks for experimentation (blob.ipynb, kosmostest.ipynb).
– Use of Slurm for managing batch jobs in cluster mode.
– External tools: ffmpeg (for video logging) and SimplerEnv for simulation experiments.
– Likely deep learning libraries are used under the hood (though names aren’t explicit, the organization hints at integration with libraries for diffusion models and tokenization).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        A["Data Generation (rlds_generator)"]:::generation
        B["Dataset Processing (twinvla/datasets)"]:::dataset
    end

    subgraph "Model/Computation Layer"
        C["Model Architectures (twinvla/model)"]:::model
        C1["eagle2 (twinvla/model/modeling/eagle2)"]:::model
        C2["internvl2_5_MPO (twinvla/model/modeling/internvl2_5_MPO)"]:::model
        C3["mobilevlmv2 (twinvla/model/modeling/mobilevlmv2)"]:::model
        C4["Heads (twinvla/model/heads)"]:::model
        C5["Tokenizers (twinvla/model/tokenizers.py)"]:::model
        C6["SingleVLA models (twinvla/model/singlevlas)"]:::model
        C7["TwinVLA models (twinvla/model/twinvlas)"]:::model
    end

    subgraph "Execution Layer"
        D["Training Scripts (scripts)"]:::training
        E["Simulations & Evaluation (scripts/simulations)"]:::simulation
        F["Batch Job Scheduling (slurm)"]:::training
    end

    subgraph "Configuration & Utilities"
        G["Configuration Scripts (scripts/configs)"]:::utility
        H["dummy.py"]:::utility
        I["singlevla_gen.py"]:::utility
        J["Utils (scripts/utils.py)"]:::utility
    end

    K["SimplerEnv (External)"]:::external

    %% Data Flow Relationships
    A -->|"generates"| B
    B -->|"processes"| C

    %% Training/Execution Relationships
    D -->|"trains"| C
    E -->|"evaluates"| C
    F -->|"schedules"| D
    K -->|"integrates"| E

    %% Utility Support
    G -->|"configures"| D
    H -->|"supports"| D
    I -->|"supports"| D
    J -->|"supports"| D

    %% Model Submodule Connections
    C --> C1
    C --> C2
    C --> C3
    C --> C4
    C --> C5
    C --> C6
    C --> C7

    %% Styles
    classDef generation fill:#a1d99b,stroke:#2c7fb8,stroke-width:2px;
    classDef dataset fill:#fd8d3c,stroke:#bd0026,stroke-width:2px;
    classDef model fill:#9ecae1,stroke:#08519c,stroke-width:2px;
    classDef training fill:#a6cee3,stroke:#1f78b4,stroke-width:2px;
    classDef simulation fill:#fdae6b,stroke:#e6550d,stroke-width:2px;
    classDef external fill:#dadaeb,stroke:#6a51a3,stroke-width:2px;
    classDef utility fill:#fdd0a2,stroke:#f16913,stroke-width:2px;

    %% Click Events
    click A "https://github.com/jellyho/twinvla/tree/main/rlds_generator"
    click B "https://github.com/jellyho/twinvla/tree/main/twinvla/datasets"
    click C "https://github.com/jellyho/twinvla/tree/main/twinvla/model"
    click C1 "https://github.com/jellyho/twinvla/tree/main/twinvla/model/modeling/eagle2"
    click C2 "https://github.com/jellyho/twinvla/tree/main/twinvla/model/modeling/internvl2_5_MPO"
    click C3 "https://github.com/jellyho/twinvla/tree/main/twinvla/model/modeling/mobilevlmv2"
    click C4 "https://github.com/jellyho/twinvla/tree/main/twinvla/model/heads"
    click C5 "https://github.com/jellyho/twinvla/blob/main/twinvla/model/tokenizers.py"
    click C6 "https://github.com/jellyho/twinvla/tree/main/twinvla/model/singlevlas"
    click C7 "https://github.com/jellyho/twinvla/tree/main/twinvla/model/twinvlas"
    click D "https://github.com/jellyho/twinvla/tree/main/scripts"
    click E "https://github.com/jellyho/twinvla/tree/main/scripts/simulations"
    click F "https://github.com/jellyho/twinvla/tree/main/slurm"
    click G "https://github.com/jellyho/twinvla/tree/main/scripts/configs"
    click H "https://github.com/jellyho/twinvla/blob/main/dummy.py"
    click I "https://github.com/jellyho/twinvla/blob/main/singlevla_gen.py"
    click J "https://github.com/jellyho/twinvla/blob/main/scripts/utils.py"
```
