# showlab/showui architecture

> Architecture diagram of the GitHub repository showlab/showui, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/showlab/showui
- Repository: https://github.com/showlab/showui

## Overview

##### Main Components

a) Core Model Architecture:
- Vision Processing Module
- Language Processing Module
- Action Generation Module
- UI-Guided Token Selection System

b) Data Processing Pipeline:
- Dataset Handlers (from data/ directory)
- Template Processing System
- Data Utils and Transformations

c) Model Implementation:
- Qwen2 VL Integration
- ShowUI Model Components
- Processing and Configuration Systems

##### Input Layer
- GUI Screenshot Input
- Natural Language Instructions
- UI Component Detection

##### Processing Layer
- Vision-Language Fusion
- UI-Guided Token Selection
- Action Generation

##### Output Layer
- Computer Control Actions
- Interactive Responses

##### Specific Implementation Details

- Rounded Rectangles: Data processing modules
- Hexagons: External interfaces
- Diamonds: Decision points in the workflow

##### Component Organization

Top Level:
**[GUI Input] → [Vision Processing] → [Language-Vision Fusion] → [Action Generation] → [Computer Control]**
- ↑                         ↑                         ↑
- [Data Pipeline]          [Model Core]              [UI Guidance]

##### Technical Stack Representation

Specific technology indicators:
- DeepSpeed Integration (shown in deployment section)
- Wandb Training Monitor
- Flash Attention 2
- Multiple Dataset Support
- LoRA/QLoRA Support

##### Additional Elements

Supporting components:
- Training Pipeline
- Evaluation Systems
- Dataset Management
- Configuration Management

This system design should emphasize ShowUI's unique position as a Vision-Language-Action model, highlighting how it processes GUI screenshots, interprets natural language commands, and generates appropriate computer control actions.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        GUI[GUI Screenshot]
        NLI[Natural Language Instructions]
        UIC[UI Component Detection]
    end

    subgraph Processing
        VP[Vision Processing]:::vision
        LVF[Language-Vision Fusion]:::fusion
        UIG[UI-Guided Token Selection]:::ui
        AG[Action Generation]:::action
    end

    subgraph Output
        CA[Computer Control Actions]
        IR[Interactive Responses]
    end

    subgraph "Data Layer"
        DP[Dataset Processing]:::data
        TP[Template Processing]:::data
        DU[Data Utils]:::data
        
        subgraph Datasets
            SS[ScreenSpot]:::dataset
            SG[Shared Grounding]:::dataset
            SN[Shared Navigation]:::dataset
        end
    end

    subgraph "Model Core"
        Q2[Qwen2 VL Model]:::model
        SUI[ShowUI Model]:::model
        MC[Model Configuration]:::config
    end

    subgraph "Training & Evaluation"
        TR[Training Pipeline]:::train
        EV[Evaluation System]:::eval
        DS[DeepSpeed Integration]:::deploy
        WB[Wandb Monitor]:::monitor
    end

    subgraph "Interface"
        WI[Web Interface]:::web
    end

    %% Connections
    GUI --> VP
    NLI --> LVF
    UIC --> UIG
    
    VP --> LVF
    LVF --> UIG
    UIG --> AG
    AG --> CA
    AG --> IR

    DP --> VP
    DP --> LVF
    TP --> DP
    DU --> DP

    SS --> DP
    SG --> DP
    SN --> DP

    Q2 --> LVF
    SUI --> AG
    MC --> SUI

    %% Click Events
    click VP "https://github.com/showlab/ShowUI/blob/main/model/qwen2_vl/image_processing_qwen2_vl.py"
    click LVF "https://github.com/showlab/ShowUI/blob/main/model/qwen2_vl/modeling_qwen2_vl.py"
    click AG "https://github.com/showlab/ShowUI/blob/main/model/showui/modeling_showui.py"
    click UIG "https://github.com/showlab/ShowUI/blob/main/model/showui/processing_showui.py"
    click DP "https://github.com/showlab/ShowUI/blob/main/data/dataset.py"
    click TP "https://github.com/showlab/ShowUI/tree/main/data/template/"
    click DU "https://github.com/showlab/ShowUI/blob/main/data/data_utils.py"
    click TR "https://github.com/showlab/ShowUI/blob/main/main/trainer.py"
    click EV "https://github.com/showlab/ShowUI/blob/main/main/evaluator.py"
    click MC "https://github.com/showlab/ShowUI/blob/main/model/showui/configuration_showui.py"
    click DS "https://github.com/showlab/ShowUI/tree/main/ds_configs/"
    click Q2 "https://github.com/showlab/ShowUI/tree/main/model/qwen2_vl/"
    click SUI "https://github.com/showlab/ShowUI/tree/main/model/showui/"
    click SS "https://github.com/showlab/ShowUI/blob/main/data/dset_screenspot.py"
    click SG "https://github.com/showlab/ShowUI/blob/main/data/dset_shared_grounding.py"
    click SN "https://github.com/showlab/ShowUI/blob/main/data/dset_shared_navigation.py"
    click WI "https://github.com/showlab/ShowUI/blob/main/app.py"

    %% Styles
    classDef vision fill:#3498db
    classDef fusion fill:#9b59b6
    classDef ui fill:#e74c3c
    classDef action fill:#f39c12
    classDef data fill:#2ecc71
    classDef model fill:#1abc9c
    classDef config fill:#95a5a6
    classDef train fill:#d35400
    classDef eval fill:#c0392b
    classDef deploy fill:#7f8c8d
    classDef monitor fill:#27ae60
    classDef web fill:#2980b9
    classDef dataset fill:#16a085

    %% Legend
    subgraph Legend
        L1[Vision Components]:::vision
        L2[Fusion Components]:::fusion
        L3[UI Components]:::ui
        L4[Action Components]:::action
        L5[Data Components]:::data
        L6[Model Components]:::model
        L7[Config Components]:::config
        L8[Training Components]:::train
        L9[Evaluation Components]:::eval
        L10[Deployment Components]:::deploy
        L11[Monitoring Components]:::monitor
        L12[Web Components]:::web
        L13[Dataset Components]:::dataset
    end
```
