How bytedance/ui-tars works
Project type and purpose
• UI-TARS is an open-source, multimodal GUI-automation tool built around a vision-language model.
• It is not a full-stack web app but a library/toolchain that:
• Sends screenshots or observations to a remote model endpoint (Hugging Face)
• Receives text-based action plans
• Parses and executes those actions on a target GUI (desktop/mobile/emulator)
File structure
• Top-level:
• README*.md, LICENSE, paper PDFs (documentation)
• codes/ – core scripts
• data/ – sample messages and training examples (testing, dev)
• codes/ breakdown:
• prompts.py – prompt templates (COMPUTER_USE, MOBILE_USE, GROUNDING)
• action_parser.py – converts model output into pyautogui calls
• utils.py – shared helper functions (e.g., coordinate transforms)
• inference_test.py – scripts for local inference tests
• Configuration & deployment:
• README_deploy.md – instructions for setting up a Hugging Face endpoint
README for additional insights
• Describes a two-stage runtime:
1) Deployment & inference on remote model
2) Post-processing to yield executable GUI actions
• Mentions coordinate processing via README_coordinates.md
• Lists benchmarks and model scales (72B, 7B, 1.5)