How zihanddd/bimgent works
Project Type and Purpose
• BIMgent is a Python-based, agentic framework for autonomous building modeling in the Vectorworks BIM tool.
• It orchestrates multiple “providers” (LLM, vision, builder) and a control loop to interpret sketches, generate instructions, and drive Vectorworks via simulated UI actions.
File-Structure Analysis
• Top-level files:
• agent_runner.py – entry point
• runner/vectorworks_runner.py – Vectorworks control logic
• conf/env_config_vectorworks.json – tool panel layout & model paths
• mouse_detector.py – panel-coordinate calibration
• Core package (BIMgent/):
• floorplan/ – sketch-to-floorplan processor
• memory/ – state storage abstractions (base, local)
• provider/ – four subdomains:
– Deep_fp_provider: floorplan segmentation & post-processing (TFRecord utils, model client endpoints)
– llm_provider: OpenAI/Gemini API wrappers
– builders_provider: mapping of high-level commands to Vectorworks primitives
– loop_providers: orchestrators (project_manager, skill_generator, skill_executor, design_interpreter, etc.)
– omni_provider: vision + parsing extension (OmniParser integration, Gradio/Omnibox demo UIs, VM scripts)
• ui_controller.py / screenshots_processor.py – image capture and UI-action glue
• utils/ – coordinate transforms, JSON/file helpers, IFC tracking, result phasing.
README Insights
• External dependencies: OpenAI & Google Gemini APIs, DeepFloorplan model, OmniParser model, Vectorworks application.
a. Agent Runner Layer
• agent_runner.py → loads configuration, initializes Vectorworks runner.
• runner/vectorworks_runner.py → UI-control entry point.
b. Core Orchestration Layer
• Loop Providers (project_manager → skill_generator_provider → skill_executor → design_interpreter → …)
• LLM Provider (OpenAI/Gemini)
• Deep Floorplan Provider (vision model inference + post-processing)
• Builder Provider (translating abstract instructions to Vectorworks API calls)
• Memory Module (local/session state)
c. Utility Layer
• UI Controller & Screenshot Processor
• Mouse Detector (panel calibration)
• Generic Utils (file, JSON, encoding, coordinate transforms)
d. External Services & Tools
• Vectorworks application (desktop)
• OpenAI & Gemini API endpoints
• DeepFloorplan model (downloaded externally)
• OmniParser model + Gradio/Omnibox demo servers
e. Configuration & Deployment
• conf/env_config_vectorworks.json, .env for API keys
• Requirements.txt for Python environment
• docs/ for visual guidance
Component Relationships & Data Flow
• agent_runner starts → initializes Vectorworks_runner → captures initial UI screenshot → passes image to DeepFloorplan Provider → produces floorplan structure → handed off to Loop Providers → Loop Providers request LLM responses (via llm_provider) and builder actions (via builders_provider) → ui_controller executes actions in Vectorworks → screenshots & state are logged to Memory → repeat until project completion → final model exported.
• Omni_provider can be invoked for alternative vision + parsing, exposed via REST (omni_endpoint) or Gradio/Omnibox UIs.
Architectural Patterns & Design Principles
• Modular “Provider” pattern: each concern is isolated (vision, LLM, builder, memory, loop orchestration).
• Controller/Loop orchestrator: central state machine (project_manager) calling skill generators and executors.
• Adapter/Wrappers for external APIs (OpenAI/Gemini, DeepFloorplan, OmniParser).
• Configuration-driven UI emulation for Vectorworks.