How picsart-ai-research/mi-gan works
Main Components
• Data/Configuration Management:
– Include the “configs” directory, which holds YAML files for datasets, experiments, and model configurations (like ffhq.yaml, places2.yaml, and various experiment configs).
– Represent the “lib” modules such as data_factory (for data loading, formatting, sampling) that prepare image/mask inputs.
• Model Components:
– Indicate the “lib/model_zoo” and “dnnlib” modules, which contain model definitions for MI-GAN and related architectures.
– Highlight that there are two main model types: MI-GAN and Co-Mod-GAN (supported through different config files and experiment scripts) with separate training and inference flows.
• Evaluation and Metrics:
– Include the “lib/evaluator” and “torch_utils” directories. The evaluator modules incorporate metrics (FID, LPIPS, PSNR, SSIM) and stylegan metrics.
• Training and Inference Pipeline:
– The “run.sh” script is a wrapper for launching distributed training on multiple GPUs.
• Pre- and Post-Processing and ONNX Pipeline:
– Represent the ONNX conversion (from scripts/create_onnx_pipeline.py) and the integration of pre-/postprocessing steps (such as image/mask conversion, cropping, and resizing).
• Examples and Assets:
– Mention the “examples” and “__assets__” folders that show sample datasets and images/masks used in demonstrations and validation of results.
Relationships and Data Flow
• How the data flows into the Data Factory (lib/data_factory) and then into the Training module, which utilizes configuration (from “configs”) and model definitions (in “lib/model_zoo” and “dnnlib”).
• A separate inference path where the outputs are generated via scripts (demo.py for inpainting results) and then optionally exported via the ONNX pipeline.
Architectural Patterns and Design Principles
• The architecture exhibits a modular design – configuration-driven experiments (separating dataset, experiment, and model configurations) and clearly demarcated data preprocessing, training, inference, and evaluation modules.
• Separation of concerns with well-organized directories for data handling, model definitions, custom GPU operations, and experiment execution.
• How external pre-trained models and checkpoints are managed (via the “models” directory externally referenced in training/inference commands).
Technologies, Frameworks, and Key Patterns
• Python is the primary language, relying on PyTorch (evident from file names and torch_utils modules).
• The system is built with GPU-acceleration and distributed training in mind.
• YAML-based configuration guides experiments and training routines.
• There is a focus on converting the pipeline to ONNX for streamlined mobile deployment.
Elements
• Each component’s label (e.g., “Data Loader (lib/data_factory)”, “Configuration Manager (configs/)”, “Model Zoo (lib/model_zoo)”, “GPU Ops (torch_utils)”, “Evaluation Engine (lib/evaluator)”).
• Annotations for key processes like “Training”, “Inference”, “Evaluation (FID/LPIPS)”, and “ONNX Export Pipeline”.