How microsoft/ufo works
Project type and purpose
• UFO² is a Python-based “Desktop AgentOS” framework for Windows.
• It turns natural-language tasks into multi-application workflows via a multi-agent, LLM-backed automation pipeline.
File structure & README
• Top-level packages:
– “ufo/”: core multi-agent framework (agents, automator, llm, prompter, module, config, utils)
– “dataflow/”: generic workflow orchestration engine (instantiation, execution, prompter, data-flow controller)
– “record_processor/”, “learner/”, “model_worker/”, “vectordb/”: support tooling for RAG and offline knowledge processing
• Configuration: YAML templates under ufo/config and dataflow/config
• Entry points: CLI modules in ufo/__main__.py, dataflow/__main__.py, record_processor/__main__.py
• README’s Architecture Overview defines five key runtime components.
Main components & relationships
1. User CLI
• “python -m ufo” / dataflow / record_processor – accepts natural-language requests.
↓
2. HostAgent (ufo/agents/agent/host_agent.py)
• Parses the goal, launches apps, manages a global FSM.
• Coordinates workflows via the “dataflow” engine (data_flow_controller.py).
↓
3. AppAgents (ufo/agents/agent/app_agent.py + processors + prompter)
• One per target application.
• Runs a ReAct loop:
– Instantiation: builds prompt templates (dataflow/instantiation/… & ufo/prompter/…)
– Execution: calls LLM (ufo/llm/… → openai, qwen, etc.)
– Speculative Executor (batch action prediction & UIA state validation)
↓
4. Automator (ufo/automator/…)
• Puppeteer subsystem decides hybrid API vs. GUI actions.
• UI-control modules for Win32, UIA, vision.
↓
5. OS & Virtual Desktop (Picture-in-Picture)
• Isolated Windows session where actual clicks/keystrokes/APIs run.
↕
6. Knowledge Substrate (RAG)
• Offline docs (assets/, documents/docs/…) and external Bing search.
• Vector store (vectordb/) contains embeddings; served by retriever.
• record_processor/ and learner/ preprocess logs, experiences, help docs.
Architectural patterns & principles
• Multi-agent orchestration (HostAgent → AppAgents)
• Dataflow-driven FSM workflows
• ReAct loop in each agent (prompt → LLM → action → environment)
• Retrieval-augmented generation with a vector DB
• Plugin pattern for LLM backends & app-specific automations
• Clear separation of concerns: orchestration, prompting, LLM calls, UI execution, knowledge storage.
– User Interface (CLI)
– HostAgent / Dataflow Orchestrator
– AppAgents (with sub-components: Prompter, LLM Client, Speculative Executor)
– Automator (GUI/API executor)
– Windows OS sandbox
– Knowledge Substrate (vector DB, record_processor, learner, external search)
– CLI → HostAgent → Dataflow controller → instantiate workflows
– Workflows → AppAgents → Prompter → LLM → actions → Automator → OS
– OS state feedback → Speculative Executor → AppAgents
– AppAgents/HostAgent → RAG retriever → vectordb/record_processor/learner
– Rectangles for services/components (e.g., HostAgent, AppAgent, Automator)
– Clouds for external services (OpenAI / Bing)