How esdolo/freevs works
Main Components
• Data Ingestion & Pre-processing:
– Components handling Waymo raw data inputs (e.g., TFRecord files).
– Scripts for extracting ground truth images (extract_gt_images.py) and generating pseudo-images (lidarproj_halfreso_multiframe.py and its mismatch-augmentation variant).
– Data processing scripts (e.g., waymo_data_generation_subsegbycampos_multiframe.py) that combine GT and pseudo images into pickle files for downstream consumption.
• Model & Pipeline Components:
– The core diffusion model components (e.g., models, pipelines, schedulers within diffusers/src/diffusers).
– Specific pipelines for stable video diffusion (pipeline_stable_video_diffusion) and additional free view synthesis extensions (inside examples/freevs and diffusers/examples/freevs).
– The “FreeVS” components including encoder modules (convnext, swin_transformer, unified_encoder) and custom UNet variants (unet_spatial_temporal_condition_custom) in the examples module.
• Training & Inference Modules:
– Training scripts (scripts/run_train.sh, run_train_onlyunet.sh) which show both joint training (encoder and diffusion) and training with a frozen pseudo-image encoder.
– Inference scripts (inference_svd.py) that load the pretrained or newly trained model to synthesize views.
• External Dependencies & Asset Management:
– Pretrained checkpoints (from Hugging Face hub) and local pretrained directories (pretrained/layout_encoder, pretrained for SVD).
– Integration with Hugging Face Accelerate and Hub modules for model/version management.
– External data sources like the Waymo Open Dataset.
Relationships & Data Flow
– Start with external inputs: Waymo dataset enters via data ingestion.
– The output (pickle files containing processed scene information) flows into the Training module.
– Training interacts with model pipelines: diffusion model architectures and scheduler modules (from diffusers/src/diffusers/models and schedulers).
– Finally, the trained model is used by the Inference module, which outputs synthesized views (rendered images/videos).
– Add connections to external services: Hugging Face hub for downloading pretrained models and executing CLI commands.
Architectural Patterns & Key Technologies
– Emphasize the modular design of the diffusers library: separation between model definitions (autoencoders, UNets, transformers) and pipelines.
– Show usage of a microservices-like or component-based approach; data processing, training, and inference are separate but interact via clear interfaces (e.g., pickle files, configuration YAMLs).
– Note extensions for video synthesis, where stable diffusion models are adapted for temporal (video) tasks.
– Underline that the project uses shell scripts, Python entry points (CLI commands), and configuration files (e.g., fsdp.yaml) to manage distributed training.