How weixi-feng/layoutgpt works
Project Type & Purpose
• This is a research prototype/system for layout and scene generation.
• It combines large language models (e.g., GPT-4, Llama-2) with visual synthesis tools to generate and evaluate both 2D image layouts and 3D indoor environments.
• The system supports multi-modal generation, evaluation (using GLIP), and visualization (using Blender for 3D rendering).
Major System Components
a. Data & Configuration:
– Dataset directories (e.g., dataset/NSR-1K and dataset/3D) and config directories (e.g., ATISS/config, gligen/configs) hold ground truth data and parameter settings.
– YAML/CSV configuration files govern room setups, splits, and evaluation settings.
b. Layout and Scene Generation:
– Script layer: Command-line scripts (e.g., run_layoutgpt_2d.py, run_layoutgpt_3d.py) serve as entry points.
– 2D layout generation: Utilizes LLM-based on-context learning to generate image layouts.
– 3D scene synthesis: Leverages ATISS modules (found in ATISS/scene_synthesis) and additional scripts for scene data preprocessing, training, and scene completion.
c. Core Modules and Libraries:
– ATISS: Provides dataset handling, network definitions (e.g., autoregressive_transformer, feature extractors, losses), training utilities, and synthesis pipelines.
– GLIGEN: Manages diffusion models and layout/input processing (gligen/dataset, gligen/ldm, etc.) for integrating text/image conditions.
– GLIP (under eval_models/GLIP): Contains evaluation and inference modules for image evaluation and grounding.
– Utility scripts (e.g., utils.py, parse_llm_output.py) assist in integrating outputs across the pipeline.
d. Evaluation and Rendering:
– Evaluation scripts for 2D layouts (eval_counting_layout.py) and 3D scenes (eval_scene_layout.py) assess generated outputs.
– Blender integration: Scripts like render_from_files.py and render_with_blender.py produce scene visualizations.
– Additional external model checkpoints and dependencies are pulled in via instructions in the README.
Relationships & Data Flow
– The system starts with data ingestion: raw images, 3D scene datasets, and preprocessed splits.
– Configuration files drive dataset filtering and preprocessing – these feed into the scene synthesis modules (ATISS) and layout generation scripts.
– The LLM (via GPT4 or Llama-13B) is used for generating layouts which are saved in output directories (like llm_output).
– Downstream components (GLIGEN and GLIP) then take these outputs for image generation and evaluation.
– There is a clear separation between generation and evaluation, with pipelines for both 2D and 3D flow.
– Data is passed between modules via files (JSON layouts, CSV configurations, pickle files for scene objects).
Architectural Design Patterns & Technologies
– Modularity: Each major functionality (layout generation, synthesis, evaluation, rendering) is encapsulated in its own directory and set of scripts.
– Pipeline pattern: Data flows linearly from preprocessing to generation to evaluation; feedback loops are possible (e.g., training networks, refining outputs).
– Integration of external dependencies: The design accounts for installing and running third-party modules such as GLIP, GLIGEN, and Blender.
– Use of configuration-driven setups (several YAML and CSV config files) enables flexible experimentation and parameter tuning.