# browser-use/macos-use architecture

> Architecture diagram of the GitHub repository browser-use/macos-use, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/browser-use/macos-use
- Repository: https://github.com/browser-use/macos-use

## Overview

##### Project Type and Purpose
- Type: Python-based AI-agent library with both CLI examples and a lightweight Gradio UI.
- Purpose: Let a language-model-powered “agent” drive macOS apps (and eventually other Apple devices) by issuing OS-level commands, with optional telemetry and pluggable inference backends (OpenAI, Anthropic, local MLX models).

##### File Structure
Top-level components:
- examples/ – CLI scripts that exercise the library (basic_agent, calculate, login_to_auth0, etc.).
- gradio_app/ – a small Gradio-based web UI (app.py + src folder with config, models, services, ui, utils).
- mlx_use/ – core library with sub-packages:
– agent ( message_manager, prompts, service, views )
– controller ( registry, service, views )
– mac ( actions, context, element, tree, views )
– telemetry (service, views)
– logging_config.py, utils.py
- Configuration & metadata: pyproject.toml, .env.example, pytest.ini, etc.

##### Extract Key Insights from the README
- AI-agent orchestrates macOS UI operations by interpreting user prompts.
- Supports multiple LLM backends via environment variables (.env).
- Includes telemetry, self-correction hooks, fallback behaviors.
- Future goal: local inference with MLX and MLX-VLM models.

A. Frontend Layer
– CLI Examples (examples/*.py)
– Gradio UI (gradio_app/app.py → src/ui/interface.py)
B. Application / Orchestration Layer
– Agent (mlx_use.agent)
- MessageManager Service
- Prompt builder (prompts.py)
– Controller (mlx_use.controller)
- Registry service & view adapters
– Telemetry (mlx_use.telemetry)
C. Integration / Infrastructure Layer
– macOS Integration (mlx_use.mac)
- Actions (element selection, events, keyboard/mouse injection)
- Context manager & UI tree abstractions
– Logging & Utils (logging_config.py, utils.py)
D. External Dependencies
– LLM Providers (OpenAI, Anthropic, Gemini) via HTTP API
– Local inference (future: MLX, MLX-VLM)
– OS environment (Python runtime, UV toolchain, Homebrew, .env)

##### Relationships & Data Flow
1. User issues prompt via CLI or Gradio →
2. Frontend invokes Agent API →
3. Agent builds prompt (prompts.py) →
4. Agent calls external LLM provider →
5. LLM response fed into MessageManager →
6. Controller looks up matching “action” in registry →
7. Controller dispatches to macOS Integration (actions/context) →
8. macOS modules simulate UI events, read UI state →
9. Results & telemetry logged →
10. Agent may loop / self-correct, then finally returns “done” status to frontend.

##### Architectural Patterns & Design Principles
- Layered architecture: clear separation of UI, orchestration, integration.
- Modular services: Agent, Controller, Telemetry can be extended independently.
- Registry pattern: controller.registry holds available “views” & “services.”
- Adapter/Facade for macOS UI: hides OS-specific details behind simple Python calls.
- Pluggable backends: LLM API clients and local inference handlers share the same interface.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend" 
        direction TB
        CLI["CLI Examples"]:::frontend
        Gradio["Gradio UI"]:::frontend
    end

    %% Orchestration Layer
    subgraph "Orchestration" 
        direction TB
        subgraph "Agent" 
            direction TB
            PromptBuilder["Prompt Builder\n(mlx_use/agent/prompts.py)"]:::orchestration
            MessageManagerService["MessageManager Service\n(service.py)"]:::orchestration
            MessageManagerViews["MessageManager Views\n(views.py)"]:::orchestration
            AgentCoreService["Agent Core Service\n(service.py)"]:::orchestration
            AgentCoreViews["Agent Core Views\n(views.py)"]:::orchestration
        end
        subgraph "Controller" 
            direction TB
            RegistryService["Registry Service\n(registry/service.py)"]:::orchestration
            RegistryViews["Registry Views\n(registry/views.py)"]:::orchestration
            ControllerService["Controller Core Service\n(service.py)"]:::orchestration
            ControllerViews["Controller Core Views\n(views.py)"]:::orchestration
        end
        subgraph "Telemetry" 
            direction TB
            TelemetryService["Telemetry Service\n(service.py)"]:::orchestration
            TelemetryViews["Telemetry Views\n(views.py)"]:::orchestration
        end
    end

    %% Integration Layer
    subgraph "Integration" 
        direction TB
        subgraph "macOS Integration" 
            direction TB
            Actions["Actions\n(actions.py)"]:::integration
            Context["Context Manager\n(context.py)"]:::integration
            Element["UI Element Abstractions\n(element.py)"]:::integration
            Tree["UI Tree Abstractions\n(tree.py)"]:::integration
            MacViews["macOS Views/Adapters\n(views.py)"]:::integration
        end
        LoggingConfig["Logging Config\n(logging_config.py)"]:::integration
        Utils["Utility Functions\n(utils.py)"]:::integration
    end

    %% External Dependencies
    subgraph "External" 
        direction TB
        LLM["LLM Providers\n(OpenAI/Anthropic/Gemini)"]:::external
        LocalInference["Local Inference\n(MLX/MLX-VLM)"]:::external
        OS["OS Environment\n(Python, Homebrew, .env)"]:::external
    end

    %% Data Flow
    CLI -->|"User Prompt"| AgentCoreService
    Gradio -->|"User Prompt"| AgentCoreService
    AgentCoreService -->|"builds prompt"| PromptBuilder
    PromptBuilder -->|"calls API"| LLM
    LLM -->|"response"| MessageManagerService
    MessageManagerService --> MessageManagerViews
    MessageManagerService --> RegistryService
    RegistryService --> RegistryViews
    RegistryService --> ControllerService
    ControllerService --> ControllerViews
    ControllerService --> Actions
    Actions --> Context
    Context --> Element
    Element --> Tree
    Tree --> MacViews
    MacViews --> LoggingConfig
    LoggingConfig --> TelemetryService
    Utils --> TelemetryService
    TelemetryService --> TelemetryViews
    TelemetryService --> AgentCoreService
    AgentCoreService -->|"done"| CLI
    AgentCoreService -->|"done"| Gradio

    %% Click Events
    click CLI "https://github.com/browser-use/macos-use/tree/main/examples/"
    click Gradio "https://github.com/browser-use/macos-use/blob/main/gradio_app/app.py"
    click PromptBuilder "https://github.com/browser-use/macos-use/blob/main/mlx_use/agent/prompts.py"
    click MessageManagerService "https://github.com/browser-use/macos-use/blob/main/mlx_use/agent/message_manager/service.py"
    click MessageManagerViews "https://github.com/browser-use/macos-use/blob/main/mlx_use/agent/message_manager/views.py"
    click AgentCoreService "https://github.com/browser-use/macos-use/blob/main/mlx_use/agent/service.py"
    click AgentCoreViews "https://github.com/browser-use/macos-use/blob/main/mlx_use/agent/views.py"
    click RegistryService "https://github.com/browser-use/macos-use/blob/main/mlx_use/controller/registry/service.py"
    click RegistryViews "https://github.com/browser-use/macos-use/blob/main/mlx_use/controller/registry/views.py"
    click ControllerService "https://github.com/browser-use/macos-use/blob/main/mlx_use/controller/service.py"
    click ControllerViews "https://github.com/browser-use/macos-use/blob/main/mlx_use/controller/views.py"
    click TelemetryService "https://github.com/browser-use/macos-use/blob/main/mlx_use/telemetry/service.py"
    click TelemetryViews "https://github.com/browser-use/macos-use/blob/main/mlx_use/telemetry/views.py"
    click Actions "https://github.com/browser-use/macos-use/blob/main/mlx_use/mac/actions.py"
    click Context "https://github.com/browser-use/macos-use/blob/main/mlx_use/mac/context.py"
    click Element "https://github.com/browser-use/macos-use/blob/main/mlx_use/mac/element.py"
    click Tree "https://github.com/browser-use/macos-use/blob/main/mlx_use/mac/tree.py"
    click MacViews "https://github.com/browser-use/macos-use/blob/main/mlx_use/mac/views.py"
    click LoggingConfig "https://github.com/browser-use/macos-use/blob/main/mlx_use/logging_config.py"
    click Utils "https://github.com/browser-use/macos-use/blob/main/mlx_use/utils.py"
    click LLM "https://github.com/browser-use/macos-use/tree/main/gradio_app/src/models/"
    click LocalInference "https://github.com/browser-use/macos-use/blob/main/gradio_app/src/services/google_form.py"
    click OS "https://github.com/browser-use/macos-use/blob/main/gradio_app/src/config/example_prompts.py"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#004085,color:#004085;
    classDef orchestration fill:#d4edda,stroke:#155724,color:#155724;
    classDef integration fill:#fff3cd,stroke:#856404,color:#856404;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
```
