# microsoft/ufo architecture

> Architecture diagram of the GitHub repository microsoft/ufo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/ufo
- Repository: https://github.com/microsoft/ufo

## Overview

##### Project type and purpose
- UFO² is a Python-based “Desktop AgentOS” framework for Windows.
- It turns natural-language tasks into multi-application workflows via a multi-agent, LLM-backed automation pipeline.

##### File structure & README
- Top-level packages:
– “ufo/”: core multi-agent framework (agents, automator, llm, prompter, module, config, utils)
– “dataflow/”: generic workflow orchestration engine (instantiation, execution, prompter, data-flow controller)
– “record_processor/”, “learner/”, “model_worker/”, “vectordb/”: support tooling for RAG and offline knowledge processing
- Configuration: YAML templates under ufo/config and dataflow/config
- Entry points: CLI modules in ufo/__main__.py, dataflow/__main__.py, record_processor/__main__.py
- README’s Architecture Overview defines five key runtime components.

##### Main components & relationships
1. User CLI
- “python -m ufo” / dataflow / record_processor – accepts natural-language requests.
↓
2. HostAgent (ufo/agents/agent/host_agent.py)
- Parses the goal, launches apps, manages a global FSM.
- Coordinates workflows via the “dataflow” engine (data_flow_controller.py).
↓
3. AppAgents (ufo/agents/agent/app_agent.py + processors + prompter)
- One per target application.
- Runs a ReAct loop:
– Instantiation: builds prompt templates (dataflow/instantiation/… & ufo/prompter/…)
– Execution: calls LLM (ufo/llm/… → openai, qwen, etc.)
– Speculative Executor (batch action prediction & UIA state validation)
↓
4. Automator (ufo/automator/…)
- Puppeteer subsystem decides hybrid API vs. GUI actions.
- UI-control modules for Win32, UIA, vision.
↓
5. OS & Virtual Desktop (Picture-in-Picture)
- Isolated Windows session where actual clicks/keystrokes/APIs run.
↕
6. Knowledge Substrate (RAG)
- Offline docs (assets/, documents/docs/…) and external Bing search.
- Vector store (vectordb/) contains embeddings; served by retriever.
- record_processor/ and learner/ preprocess logs, experiences, help docs.

##### Architectural patterns & principles
- Multi-agent orchestration (HostAgent → AppAgents)
- Dataflow-driven FSM workflows
- ReAct loop in each agent (prompt → LLM → action → environment)
- Retrieval-augmented generation with a vector DB
- Plugin pattern for LLM backends & app-specific automations
- Clear separation of concerns: orchestration, prompting, LLM calls, UI execution, knowledge storage.

– User Interface (CLI)
– HostAgent / Dataflow Orchestrator
– AppAgents (with sub-components: Prompter, LLM Client, Speculative Executor)
– Automator (GUI/API executor)
– Windows OS sandbox
– Knowledge Substrate (vector DB, record_processor, learner, external search)
– CLI → HostAgent → Dataflow controller → instantiate workflows
– Workflows → AppAgents → Prompter → LLM → actions → Automator → OS
– OS state feedback → Speculative Executor → AppAgents
– AppAgents/HostAgent → RAG retriever → vectordb/record_processor/learner
– Rectangles for services/components (e.g., HostAgent, AppAgent, Automator)
– Clouds for external services (OpenAI / Bing)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface
    subgraph "User Interface"
        CLI_UFO["CLI: python -m ufo"]:::cli
        CLI_Dataflow["CLI: python -m dataflow"]:::cli
        CLI_Record["CLI: python -m record_processor"]:::cli
    end

    %% Orchestrator Layer
    subgraph "Orchestrator"
        HostAgent["HostAgent"]:::orchestrator
        HostAgentProcessor["HostAgentProcessor"]:::orchestrator
        HostAgentState["HostAgentState"]:::orchestrator
        DataflowController["DataflowController"]:::orchestrator
    end

    %% AppAgents Layer
    subgraph "AppAgents"
        AppAgent["AppAgent"]:::agent
        AppAgentProcessor["AppAgentProcessor"]:::agent
        AppAgentState["AppAgentState"]:::agent
        Prompter["Prompter"]:::agent
        LLMClient["LLM Client"]:::agent
        SpecExecutor["Speculative Executor"]:::agent
    end

    %% Execution Layer
    subgraph "Automator"
        Puppeteer["Puppeteer"]:::exec
        API_Clients["App API Clients"]:::exec
        UI_Control["UI Control Modules"]:::exec
    end

    %% OS Sandbox
    subgraph "Windows OS Sandbox"
        OS["Windows OS"]:::os
        VM["Virtual Desktop (PiP)"]:::os
    end

    %% Knowledge Substrate
    subgraph "Knowledge Substrate"
        VectorDB["Vector DB (FAISS)"]:::storage
        Retriever["RAG Retriever"]:::storage
        WebSearch["Web Search (Bing)"]:::external
        RecordProc["Record Processor"]:::storage
        Learner["Learner"]:::storage
    end

    %% Connections
    CLI_UFO -->|"invoke"| HostAgent
    CLI_Dataflow -->|"invoke"| DataflowController
    CLI_Record -->|"invoke"| RecordProc

    HostAgent -->|"orchestrates"| HostAgentProcessor
    HostAgent -->|"maintains FSM"| HostAgentState
    HostAgent -->|"executes workflows"| DataflowController

    DataflowController -->|"instantiate & execute"| AppAgent

    AppAgent -->|"process"| AppAgentProcessor
    AppAgent -->|"state"| AppAgentState
    AppAgent -->|"build prompt"| Prompter
    Prompter -->|"LLM call"| LLMClient
    LLMClient -->|"action"| SpecExecutor
    SpecExecutor -->|"execute UI/API"| Puppeteer
    Puppeteer -->|"API/UI actions"| API_Clients
    Puppeteer -->|"UI control"| UI_Control
    API_Clients -->|"interact"| OS
    UI_Control -->|"interact"| OS
    SpecExecutor -->|"state feedback"| AppAgent

    %% Knowledge flows
    AppAgent -->|"RAG query"| Retriever
    Retriever -->|"fetch embeddings"| VectorDB
    Retriever -->|"external search"| WebSearch
    RecordProc -->|"ingest docs"| VectorDB
    Learner -->|"update indexes"| VectorDB

    %% Virtual Desktop link
    OS -.->|"isolated session"| VM

    %% Click Events
    click CLI_UFO "https://github.com/microsoft/ufo/blob/main/ufo/__main__.py"
    click CLI_Dataflow "https://github.com/microsoft/ufo/blob/main/dataflow/__main__.py"
    click CLI_Record "https://github.com/microsoft/ufo/blob/main/record_processor/__main__.py"
    click HostAgent "https://github.com/microsoft/ufo/blob/main/ufo/agents/agent/host_agent.py"
    click HostAgentProcessor "https://github.com/microsoft/ufo/blob/main/ufo/agents/processors/host_agent_processor.py"
    click HostAgentState "https://github.com/microsoft/ufo/blob/main/ufo/agents/states/host_agent_state.py"
    click DataflowController "https://github.com/microsoft/ufo/blob/main/dataflow/data_flow_controller.py"
    click AppAgent "https://github.com/microsoft/ufo/blob/main/ufo/agents/agent/app_agent.py"
    click AppAgentProcessor "https://github.com/microsoft/ufo/blob/main/ufo/agents/processors/app_agent_processor.py"
    click AppAgentState "https://github.com/microsoft/ufo/blob/main/ufo/agents/states/app_agent_state.py"
    click Prompter "https://github.com/microsoft/ufo/blob/main/ufo/prompter/agent_prompter.py"
    click Puppeteer "https://github.com/microsoft/ufo/blob/main/ufo/automator/puppeteer.py"
    click API_Clients "https://github.com/microsoft/ufo/tree/main/ufo/automator/app_apis/"
    click UI_Control "https://github.com/microsoft/ufo/tree/main/ufo/automator/ui_control/"
    click VectorDB "https://github.com/microsoft/ufo/tree/main/vectordb/"
    click Retriever "https://github.com/microsoft/ufo/blob/main/ufo/rag/retriever.py"
    click WebSearch "https://github.com/microsoft/ufo/blob/main/ufo/rag/web_search.py"
    click RecordProc "https://github.com/microsoft/ufo/tree/main/record_processor/"
    click Learner "https://github.com/microsoft/ufo/tree/main/learner/"

    %% Styles
    classDef cli fill:#d5e8d4,stroke:#82b366,color:#333;
    classDef orchestrator fill:#dae8fc,stroke:#6c8ebf,color:#333;
    classDef agent fill:#fff2cc,stroke:#d6b656,color:#333;
    classDef exec fill:#f8cecc,stroke:#b85450,color:#333;
    classDef os fill:#e1d5e7,stroke:#9673a6,color:#333;
    classDef storage fill:#e1eed4,stroke:#82a363,color:#333;
    classDef external fill:#f5f5f5,stroke:#999999,color:#333;
```
