# weixi-feng/layoutgpt architecture

> Architecture diagram of the GitHub repository weixi-feng/layoutgpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/weixi-feng/layoutgpt
- Repository: https://github.com/weixi-feng/layoutgpt

## Overview

##### Project Type & Purpose
- This is a research prototype/system for layout and scene generation.
- It combines large language models (e.g., GPT-4, Llama-2) with visual synthesis tools to generate and evaluate both 2D image layouts and 3D indoor environments.
- The system supports multi-modal generation, evaluation (using GLIP), and visualization (using Blender for 3D rendering).

##### Major System Components
a. Data & Configuration:
– Dataset directories (e.g., dataset/NSR-1K and dataset/3D) and config directories (e.g., ATISS/config, gligen/configs) hold ground truth data and parameter settings.
– YAML/CSV configuration files govern room setups, splits, and evaluation settings.

b. Layout and Scene Generation:
– Script layer: Command-line scripts (e.g., run_layoutgpt_2d.py, run_layoutgpt_3d.py) serve as entry points.
– 2D layout generation: Utilizes LLM-based on-context learning to generate image layouts.
– 3D scene synthesis: Leverages ATISS modules (found in ATISS/scene_synthesis) and additional scripts for scene data preprocessing, training, and scene completion.

c. Core Modules and Libraries:
– ATISS: Provides dataset handling, network definitions (e.g., autoregressive_transformer, feature extractors, losses), training utilities, and synthesis pipelines.
– GLIGEN: Manages diffusion models and layout/input processing (gligen/dataset, gligen/ldm, etc.) for integrating text/image conditions.
– GLIP (under eval_models/GLIP): Contains evaluation and inference modules for image evaluation and grounding.
– Utility scripts (e.g., utils.py, parse_llm_output.py) assist in integrating outputs across the pipeline.

d. Evaluation and Rendering:
– Evaluation scripts for 2D layouts (eval_counting_layout.py) and 3D scenes (eval_scene_layout.py) assess generated outputs.
– Blender integration: Scripts like render_from_files.py and render_with_blender.py produce scene visualizations.
– Additional external model checkpoints and dependencies are pulled in via instructions in the README.

##### Relationships & Data Flow
– The system starts with data ingestion: raw images, 3D scene datasets, and preprocessed splits.
– Configuration files drive dataset filtering and preprocessing – these feed into the scene synthesis modules (ATISS) and layout generation scripts.
– The LLM (via GPT4 or Llama-13B) is used for generating layouts which are saved in output directories (like llm_output).
– Downstream components (GLIGEN and GLIP) then take these outputs for image generation and evaluation.
– There is a clear separation between generation and evaluation, with pipelines for both 2D and 3D flow.
– Data is passed between modules via files (JSON layouts, CSV configurations, pickle files for scene objects).

##### Architectural Design Patterns & Technologies
– Modularity: Each major functionality (layout generation, synthesis, evaluation, rendering) is encapsulated in its own directory and set of scripts.
– Pipeline pattern: Data flows linearly from preprocessing to generation to evaluation; feedback loops are possible (e.g., training networks, refining outputs).
– Integration of external dependencies: The design accounts for installing and running third-party modules such as GLIP, GLIGEN, and Blender.
– Use of configuration-driven setups (several YAML and CSV config files) enables flexible experimentation and parameter tuning.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input & Configuration
    subgraph "Input & Configuration"
        ATISS_Config["ATISS Config"]
        Data3D["3D Dataset"]
        NSR1K["NSR-1K Dataset"]
        GLIGEN_Config["GLIGEN Config"]
    end

    %% Preprocessing & Utilities
    subgraph "Preprocessing & Utilities"
        Preprocess["Preprocessing Scripts"]
        Utilities["Utility Scripts"]
    end

    %% LLM-based Layout Generation Module
    subgraph "LLM-based Layout Generation"
        Run2D["Run LayoutGPT 2D"]
        Run3D["Run LayoutGPT 3D"]
        LLM_Output["LLM Output"]
    end

    %% Downstream: GLIGEN & GLIP
    subgraph "Image Generation & Evaluation"
        GLIGEN_Modules["GLIGEN Modules"]
        GLIP_Eval["GLIP Evaluation"]
        Eval2D["2D Layout Evaluation"]
        Eval3D["3D Scene Evaluation"]
    end

    %% ATISS Scene Synthesis Branch
    subgraph "ATISS Scene Synthesis"
        ATISS_Synthesis["ATISS Scene Synthesis"]
        ATISS_Training["ATISS Training & Processing"]
        ATISS_Core["ATISS Core Modules"]
    end

    %% Rendering & Visualization
    subgraph "Rendering & Visualization"
        Render_Files["Render from Files"]
        Render_Blender["Render with Blender"]
    end

    %% Connections from Input & Config to Preprocessing
    ATISS_Config --> Preprocess
    Data3D --> Preprocess
    NSR1K --> Preprocess
    GLIGEN_Config --> Preprocess

    %% Preprocessing & Utilities feed into LLM-based Generation
    Preprocess --> Run2D
    Preprocess --> Run3D
    Utilities --> Run2D
    Utilities --> Run3D

    %% LLM Modules flow
    Run2D --> LLM_Output
    Run3D --> LLM_Output

    %% LLM Output distributed to Generation & Evaluation
    LLM_Output --> GLIGEN_Modules
    LLM_Output --> GLIP_Eval

    %% GLIGEN and GLIP connect to Evaluation scripts
    GLIGEN_Modules --> Eval2D
    GLIGEN_Modules --> Eval3D
    GLIP_Eval --> Eval2D
    GLIP_Eval --> Eval3D

    %% ATISS branch: Input & Config to ATISS Core
    ATISS_Config --> ATISS_Core
    Utilities --> ATISS_Core

    %% ATISS Core flows into Scene Synthesis and Training
    ATISS_Core --> ATISS_Synthesis
    ATISS_Core --> ATISS_Training
    ATISS_Training --> ATISS_Synthesis

    %% ATISS Synthesis flows into Rendering
    ATISS_Synthesis --> Render_Files
    ATISS_Synthesis --> Render_Blender

    %% Click Events
    click ATISS_Config "https://github.com/weixi-feng/layoutgpt/tree/master/ATISS/config"
    click Data3D "https://github.com/weixi-feng/layoutgpt/tree/master/dataset/3D"
    click NSR1K "https://github.com/weixi-feng/layoutgpt/tree/master/dataset/NSR-1K"
    click GLIGEN_Config "https://github.com/weixi-feng/layoutgpt/tree/master/gligen/configs"
    click Run2D "https://github.com/weixi-feng/layoutgpt/blob/master/run_layoutgpt_2d.py"
    click Run3D "https://github.com/weixi-feng/layoutgpt/blob/master/run_layoutgpt_3d.py"
    click LLM_Output "https://github.com/weixi-feng/layoutgpt/tree/master/llm_output"
    click ATISS_Synthesis "https://github.com/weixi-feng/layoutgpt/tree/master/ATISS/scene_synthesis"
    click ATISS_Training "https://github.com/weixi-feng/layoutgpt/tree/master/ATISS/scripts"
    click ATISS_Core "https://github.com/weixi-feng/layoutgpt/blob/master/ATISS/setup.py"
    click GLIGEN_Modules "https://github.com/weixi-feng/layoutgpt/blob/master/gligen/main.py"
    click GLIP_Eval "https://github.com/weixi-feng/layoutgpt/tree/master/eval_models/GLIP"
    click Eval2D "https://github.com/weixi-feng/layoutgpt/blob/master/eval_counting_layout.py"
    click Eval3D "https://github.com/weixi-feng/layoutgpt/blob/master/eval_scene_layout.py"
    click Render_Files "https://github.com/weixi-feng/layoutgpt/blob/master/ATISS/scripts/render_from_files.py"
    click Render_Blender "https://github.com/weixi-feng/layoutgpt/blob/master/ATISS/scripts/render_with_blender.py"
    click Utilities "https://github.com/weixi-feng/layoutgpt/blob/master/utils.py"
    %% Note: parse_llm_output.py and ATISS/scripts/preprocess_data.py are represented within Utilities and Preprocessing Scripts respectively.

    %% Styles
    classDef config fill:#e0f7fa,stroke:#006064,stroke-width:2px;
    classDef process fill:#f1f8e9,stroke:#33691e,stroke-width:2px;
    classDef generation fill:#fff3e0,stroke:#e65100,stroke-width:2px;
    classDef evaluation fill:#fce4ec,stroke:#880e4f,stroke-width:2px;
    classDef synthesis fill:#ede7f6,stroke:#4527a0,stroke-width:2px;
    classDef rendering fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px;
    classDef utility fill:#f3e5f5,stroke:#4a148c,stroke-width:2px;

    %% Assign classes
    class ATISS_Config,Data3D,NSR1K,GLIGEN_Config config;
    class Preprocess,ATISS_Training process;
    class Run2D,Run3D,LLM_Output generation;
    class GLIGEN_Modules,GLIP_Eval,Eval2D,Eval3D evaluation;
    class ATISS_Synthesis,ATISS_Core synthesis;
    class Render_Files,Render_Blender rendering;
    class Utilities utility;
```
