# bytedance/ui-tars architecture

> Architecture diagram of the GitHub repository bytedance/ui-tars, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance/ui-tars
- Repository: https://github.com/bytedance/ui-tars

## Overview

##### Project type and purpose
- UI-TARS is an open-source, multimodal GUI-automation tool built around a vision-language model.
- It is not a full-stack web app but a library/toolchain that:
- Sends screenshots or observations to a remote model endpoint (Hugging Face)
- Receives text-based action plans
- Parses and executes those actions on a target GUI (desktop/mobile/emulator)

##### File structure
- Top-level:
- README*.md, LICENSE, paper PDFs (documentation)
- codes/ – core scripts
- data/ – sample messages and training examples (testing, dev)
- codes/ breakdown:
- prompts.py – prompt templates (COMPUTER_USE, MOBILE_USE, GROUNDING)
- action_parser.py – converts model output into pyautogui calls
- utils.py – shared helper functions (e.g., coordinate transforms)
- inference_test.py – scripts for local inference tests
- Configuration & deployment:
- README_deploy.md – instructions for setting up a Hugging Face endpoint

##### README for additional insights
- Describes a two-stage runtime:
1) Deployment & inference on remote model
2) Post-processing to yield executable GUI actions
- Mentions coordinate processing via README_coordinates.md
- Lists benchmarks and model scales (72B, 7B, 1.5)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        Inference_Client["Inference Client"]:::computation
        Prompt_Builder["Prompt Builder"]:::computation
    end

    %% Model Serving Layer
    Model_Endpoint["Hugging Face Endpoint"]:::external

    %% Post-Processing Layer
    subgraph "Post-Processing Layer"
        direction TB
        Action_Parser["Action Parser"]:::computation
        Utils_File["Coordinate Utility (utils.py)"]:::computation
        Coords_Doc["Coordinates Guide"]:::computation
    end

    %% Execution Layer
    Executor["Executor (pyautogui/ADB)"]:::computation

    %% Data & Testing
    subgraph "Data & Testing"
        direction TB
        Sample_Data["Sample Messages (data/)"]:::resources
    end

    %% Documentation & Diagrams
    subgraph "Documentation & Diagrams"
        direction TB
        Readme["README.md"]:::resources
        Readme_v1["README_v1.md"]:::resources
        Paper["UI_TARS_paper.pdf"]:::resources
        Figures["figures/"]:::resources
    end

    %% Flow connections
    Sample_Data -->|provides inputs| Inference_Client
    Inference_Client -->|builds prompt| Prompt_Builder
    Prompt_Builder -->|sends request| Model_Endpoint
    Model_Endpoint -->|returns actions| Action_Parser
    Action_Parser -->|uses utility| Utils_File
    Action_Parser -->|references guide| Coords_Doc
    Action_Parser -->|generates commands| Executor
    Executor -->|executes GUI actions| Executor

    %% Click Events
    click Inference_Client "https://github.com/bytedance/ui-tars/blob/main/codes/inference_test.py"
    click Prompt_Builder "https://github.com/bytedance/ui-tars/blob/main/codes/prompts.py"
    click Model_Endpoint "https://github.com/bytedance/ui-tars/blob/main/README_deploy.md"
    click Action_Parser "https://github.com/bytedance/ui-tars/blob/main/codes/action_parser.py"
    click Utils_File "https://github.com/bytedance/ui-tars/blob/main/codes/utils.py"
    click Coords_Doc "https://github.com/bytedance/ui-tars/blob/main/README_coordinates.md"
    click Sample_Data "https://github.com/bytedance/ui-tars/tree/main/data/"
    click Readme "https://github.com/bytedance/ui-tars/blob/main/README.md"
    click Readme_v1 "https://github.com/bytedance/ui-tars/blob/main/README_v1.md"
    click Paper "https://github.com/bytedance/ui-tars/blob/main/UI_TARS_paper.pdf"
    click Figures "https://github.com/bytedance/ui-tars/tree/main/figures/"

    %% Styles
    classDef computation fill:#E0F7FA,stroke:#006064,color:#006064
    classDef external    fill:#E8F5E9,stroke:#1B5E20,color:#1B5E20
    classDef resources   fill:#FFF3E0,stroke:#E65100,color:#E65100
```
