How yujiaoshi/sat2strpanoramasynthesis works
Project Overview
• This project is a research codebase implementing Geometry-Guided Street-View Panorama Synthesis from satellite imagery. It leverages a differentiable geometry-guided projection (S2SP module) together with a generator network and a VGG-based perceptual loss.
• The core idea is to explicitly model geometric transformations between the satellite view and the intended street-view panorama, enhancing the synthesis process compared to standard generative approaches.
Identifying Main Components
• VGG Module:
• Contains files like vgg.py and perceptual_loss.py (and its corresponding test file).
• Implements the VGG-based perceptual loss which is used during training to enforce image realism.
• Geometry Module:
• Files include Geometry.py, projector.py, test_geometry.py, and utils.py.
• Implements the geometric transformations (the S2SP module) that project satellite images into ground-level layout.
• Load Data Module:
• Contains several scripts in load_data for handling different datasets (e.g., CVACT, CVUSA, unaligned or half-size versions).
• This module is responsible for preprocessing and feeding data into the training/testing pipeline.
• Evaluation Metrics:
• The evaluation_metrics directory contains scripts to evaluate model outputs (for example, metrics_tf.py and metrics_tf_cvact.py), along with a shell script to run evaluations.
• Training/Execution Scripts:
• Under script3, several training related components exist (main.py, baseline scripts, model definitions, and shell scripts for running experiments).
• This is where the end-to-end pipeline is orchestrated, invoking data loading, geometric projection, and synthesis generation (using the learned generator network) during training and testing.
Relationships and Data Flow
• Data Input: The load_data module ingests satellite images from the CVUSA and CVACT datasets.
• Geometric Projection: The Geometry module processes the data to compute height maps and project the satellite imagery into approximate ground-view representations.
• Image Synthesis: The output of the geometric projection feeds into the synthesis generator (encoded in the script3 models), which then produces the street-view panorama.
• Perceptual Supervision: The VGG module applies a perceptual loss by comparing synthesized images against real street-view panoramas, ensuring high-level feature consistency.
• Evaluation: Once synthesis is complete, the evaluation_metrics module computes quantitative metrics (via TensorFlow-based functions) to help assess performance.
• Training Workflow: The main training loop (in script3/main.py) orchestrates these components, running them in sequence for both training and testing modes as specified in the README.
Architectural Patterns and Design Principles
• Modular Design: Each component (data loading, geometric transformation, synthesis network, perceptual loss, evaluation) is encapsulated in its own directory/module, making the architecture modular.
• End-to-End Differentiability: The design emphasizes differentiability so that the S2SP module and generator network are trained together.
• Separation of Concerns: Distinct responsibilities—data ingestion, transformation, image generation, and evaluation—are clearly separated which helps in isolating issues and further extension.