# zihanddd/bimgent architecture

> Architecture diagram of the GitHub repository zihanddd/bimgent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zihanddd/bimgent
- Repository: https://github.com/zihanddd/bimgent

## Overview

##### Project Type and Purpose
- BIMgent is a Python-based, agentic framework for autonomous building modeling in the Vectorworks BIM tool.
- It orchestrates multiple “providers” (LLM, vision, builder) and a control loop to interpret sketches, generate instructions, and drive Vectorworks via simulated UI actions.

##### File-Structure Analysis
- Top-level files:
- agent_runner.py – entry point
- runner/vectorworks_runner.py – Vectorworks control logic
- conf/env_config_vectorworks.json – tool panel layout & model paths
- mouse_detector.py – panel-coordinate calibration
- Core package (BIMgent/):
- floorplan/ – sketch-to-floorplan processor
- memory/ – state storage abstractions (base, local)
- provider/ – four subdomains:
– Deep_fp_provider: floorplan segmentation & post-processing (TFRecord utils, model client endpoints)
– llm_provider: OpenAI/Gemini API wrappers
– builders_provider: mapping of high-level commands to Vectorworks primitives
– loop_providers: orchestrators (project_manager, skill_generator, skill_executor, design_interpreter, etc.)
– omni_provider: vision + parsing extension (OmniParser integration, Gradio/Omnibox demo UIs, VM scripts)
- ui_controller.py / screenshots_processor.py – image capture and UI-action glue
- utils/ – coordinate transforms, JSON/file helpers, IFC tracking, result phasing.

##### README Insights
- External dependencies: OpenAI & Google Gemini APIs, DeepFloorplan model, OmniParser model, Vectorworks application.

a. Agent Runner Layer
- agent_runner.py → loads configuration, initializes Vectorworks runner.
- runner/vectorworks_runner.py → UI-control entry point.
b. Core Orchestration Layer
- Loop Providers (project_manager → skill_generator_provider → skill_executor → design_interpreter → …)
- LLM Provider (OpenAI/Gemini)
- Deep Floorplan Provider (vision model inference + post-processing)
- Builder Provider (translating abstract instructions to Vectorworks API calls)
- Memory Module (local/session state)
c. Utility Layer
- UI Controller & Screenshot Processor
- Mouse Detector (panel calibration)
- Generic Utils (file, JSON, encoding, coordinate transforms)
d. External Services & Tools
- Vectorworks application (desktop)
- OpenAI & Gemini API endpoints
- DeepFloorplan model (downloaded externally)
- OmniParser model + Gradio/Omnibox demo servers
e. Configuration & Deployment
- conf/env_config_vectorworks.json, .env for API keys
- Requirements.txt for Python environment
- docs/ for visual guidance

##### Component Relationships & Data Flow
- agent_runner starts → initializes Vectorworks_runner → captures initial UI screenshot → passes image to DeepFloorplan Provider → produces floorplan structure → handed off to Loop Providers → Loop Providers request LLM responses (via llm_provider) and builder actions (via builders_provider) → ui_controller executes actions in Vectorworks → screenshots & state are logged to Memory → repeat until project completion → final model exported.
- Omni_provider can be invoked for alternative vision + parsing, exposed via REST (omni_endpoint) or Gradio/Omnibox UIs.

##### Architectural Patterns & Design Principles
- Modular “Provider” pattern: each concern is isolated (vision, LLM, builder, memory, loop orchestration).
- Controller/Loop orchestrator: central state machine (project_manager) calling skill generators and executors.
- Adapter/Wrappers for external APIs (OpenAI/Gemini, DeepFloorplan, OmniParser).
- Configuration-driven UI emulation for Vectorworks.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Entry & Runner Layer
    subgraph "Entry & Runner"
        direction TB
        AgentRunner["agent_runner.py"]:::core
        VectorworksRunner["vectorworks_runner.py"]:::core
    end

    AgentRunner -->|"initializes"| VectorworksRunner

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        ProjectManager["project_manager.py"]:::provider
        SkillGenerator["skill_generator_provider.py"]:::provider
        SkillExecutor["skill_executor.py"]:::provider
        DesignInterpreter["design_interpreter.py"]:::provider
        DesignerProvider["designer_provider.py"]:::provider
        ImageUnderstanding["image_understanding_provider.py"]:::provider
        TemplateProcessor["template_processor.py"]:::provider
    end

    %% Services Layer
    subgraph "Services Layer"
        direction TB
        LLMProvider["openai_provider.py"]:::provider
        DeepFPProvider["deep_floorplan_provider.py"]:::provider
        DeepFPEndpoint["deepfloorplan_endpoint.py"]:::provider
        DeepFPUtils["Deep_fp_provider utils"]:::provider
        BuilderProvider["builder_provider.py"]:::provider
        FloorplanPre["floorplan_processer_llm.py"]:::core
    end

    %% Utility & Memory Layer
    subgraph "Utility & Memory"
        direction TB
        UIController["ui_controller.py"]:::core
        ScreenshotProcessor["screenshots_processor.py"]:::core
        MouseDetector["mouse_detector.py"]:::util
        GenericUtils["utils/*.py"]:::util
        MemoryBase["base.py"]:::memory
        LocalMemory["local_memory.py"]:::memory
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        VectorworksApp["Vectorworks App"]:::external
        OpenAIAPI["OpenAI API"]:::external
        GeminiAPI["Google Gemini API"]:::external
        DeepFPModel["DeepFloorplan Model"]:::model
        OmniParserModel["OmniParser Model"]:::model
        GradioServer["Gradio Demo"]:::external
        OmniboxServer["Omnibox UI"]:::external
    end

    %% Data Flows
    VectorworksRunner -->|"captures UI"| UIController
    UIController -->|"captures|screenshots"| ScreenshotProcessor
    ScreenshotProcessor -->|"provides image"| DeepFPProvider
    DeepFPProvider --> DeepFPEndpoint
    DeepFPEndpoint -->|"inference"| DeepFPModel
    DeepFPProvider -->|"floorplan data"| ProjectManager

    ProjectManager --> SkillGenerator
    SkillGenerator --> SkillExecutor
    SkillExecutor --> DesignInterpreter
    DesignInterpreter --> DesignerProvider
    DesignerProvider --> TemplateProcessor
    TemplateProcessor --> ImageUnderstanding
    ImageUnderstanding --> ProjectManager

    ProjectManager -->|"LLM request"| LLMProvider
    LLMProvider --> OpenAIAPI
    LLMProvider --> GeminiAPI

    ProjectManager -->|"build commands"| BuilderProvider
    BuilderProvider --> UIController

    ProjectManager -->|"state read/write"| MemoryBase
    MemoryBase --> LocalMemory

    %% Click Events
    click AgentRunner "https://github.com/zihanddd/bimgent/blob/main/agent_runner.py"
    click VectorworksRunner "https://github.com/zihanddd/bimgent/blob/main/BIMgent/runner/vectorworks_runner.py"
    click ProjectManager "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/loop_providers/project_manager.py"
    click SkillGenerator "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/loop_providers/skill_generator_provider.py"
    click SkillExecutor "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/loop_providers/skill_executor.py"
    click DesignInterpreter "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/loop_providers/design_interpreter.py"
    click DesignerProvider "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/loop_providers/designer_provider.py"
    click ImageUnderstanding "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/loop_providers/image_understanding_provider.py"
    click TemplateProcessor "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/loop_providers/template_processor.py"
    click LLMProvider "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/llm_provider/openai_provider.py"
    click DeepFPProvider "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/Deep_fp_provider/deep_floorplan_provider.py"
    click DeepFPEndpoint "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/Deep_fp_provider/deepfloorplan_endpoint.py"
    click DeepFPUtils "https://github.com/zihanddd/bimgent/tree/main/BIMgent/provider/Deep_fp_provider/utils/"
    click BuilderProvider "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/builders_provider/builder_provider.py"
    click FloorplanPre "https://github.com/zihanddd/bimgent/blob/main/BIMgent/floorplan/floorplan_processer_llm.py"
    click MemoryBase "https://github.com/zihanddd/bimgent/blob/main/BIMgent/memory/base.py"
    click LocalMemory "https://github.com/zihanddd/bimgent/blob/main/BIMgent/memory/local_memory.py"
    click UIController "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/ui_controller.py"
    click ScreenshotProcessor "https://github.com/zihanddd/bimgent/blob/main/BIMgent/provider/screenshots_processor.py"
    click MouseDetector "https://github.com/zihanddd/bimgent/blob/main/mouse_detector.py"
    click GenericUtils "https://github.com/zihanddd/bimgent/tree/main/BIMgent/utils/"
    click EnvConfig "https://github.com/zihanddd/bimgent/blob/main/conf/env_config_vectorworks.json"
    click ConfigLoader "https://github.com/zihanddd/bimgent/blob/main/conf/config.py"
    click Requirements "https://github.com/zihanddd/bimgent/blob/main/requirements.txt"
    click Docs "https://github.com/zihanddd/bimgent/tree/main/docs/"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#005A9C,color:#000;
    classDef provider fill:#FFF2CC,stroke:#997300,color:#000;
    classDef util fill:#E8E8E8,stroke:#666,color:#000;
    classDef memory fill:#E2F0D9,stroke:#3C763D,color:#000;
    classDef external fill:#DFF0D8,stroke:#3C763D,color:#000;
    classDef model fill:#F2DCDB,stroke:#A94442,color:#000;
```
