How zkawfanx/atlantis works
Project Type and Purpose
• This is a research‐oriented pipeline for underwater depth estimation and stylization, combining depth prediction (MiDaS), image captioning (LAVIS/BLIP2), and ControlNet training/inference on Stable Diffusion.
• It implements the CVPR2024 “Atlantis” framework: prepare {Underwater, Depth, Text} triplets → train a Depth2Underwater ControlNet → generate novel underwater scenes from terrestrial depths.
File Structure Analysis
• Top‐level:
• .gitmodules: submodules for MiDaS and Diffusers
• src: core scripts and vendored libraries
• Under src:
• MiDaS/: Depth estimation codebase (submodule)
• diffusers/: Hugging Face Diffusers (submodule)
• blip2_caption.py: text caption generation using LAVIS/BLIP2
• triplets.py: script to load and format {underwater, depth, text} dataset
• inference.py: uses trained ControlNet + Stable Diffusion to generate images
• No monolithic “training script” here; ControlNet training is done via the Diffusers examples folder referenced in README.
Key Architectural Insights from README
• Three sequential stages:
A. Triplet preparation (Depth via MiDaS + Text via BLIP2)
B. ControlNet training (Diffusers + accelerate + stable-diffusion-v1-5)
C. Inference/generation (trained ControlNet + input terrestrial depths)
• Heavy reliance on external frameworks: PyTorch, accelerate, cv2.
Main Components
• Data Preparation
• Input: Raw underwater images, terrestrial images
• Subcomponents: MiDaS depth estimator, BLIP2 captioner, triplet formatter
• Model Training
• Input: Prepared triplets
• Subcomponents: Hugging Face Diffusers ControlNet training script, Stable Diffusion v1.5 backbone, accelerate scheduler
• Output: Depth2Underwater ControlNet weights
• Inference
• Input: Terrestrial depth maps + ControlNet weights
• Subcomponents: inference.py, ControlNet + SD1.5 pipeline
• Output: Generated underwater images (“Atlantis”)
Component Interactions and Data Flow
• Raw Underwater → MiDaS → 8-bit Depth maps
• Raw Underwater → BLIP2 → Text descriptions
• Depth + Underwater + Text → triplets.py → Dataset structure
• Dataset → Diffusers/train_controlnet.py → ControlNet model
• Terrestrial Depths + ControlNet → inference.py → Stylized Underwater Outputs
Architectural Patterns and Principles
• Pipeline pattern: clear separation of data prep, training, inference
• Modular integration of third-party tools as submodules (MiDaS, Diffusers)
• Batch processing and GPU acceleration via accelerate and PyTorch