How sail-sg/sewformer works
Project Type and Purpose
• Sewformer is a research‐oriented deep learning system for reconstructing garment sewing patterns from single images.
• It comprises two major subsystems:
a) SewFactory – a data‐generation and simulation toolkit that assembles synthetic garment meshes, patterns, and scene materials via Maya‐based scripts.
b) Sewformer – a PyTorch‐based training and inference pipeline implementing the transformer‐based model described in SIGGRAPH Asia 2023.
High‐Level Architecture Components
A. SewFactory (Data Generation & Simulation)
• Data Generators (data_generator/) – Python scripts (data_sim.py, deepfashion_sim.py, pattern_gen.py, fbx_anime.py) that produce 2D panels, 3D meshes, animations, and simulation inputs.
• Meta-Information (meta_infos/) – JSON/YAML catalogs defining sewing patterns, material configs, FBX metadata, scene materials, and simulation parameters.
• MayaQLTools & Pattern Packages (packages/) – reusable Python modules wrapping Maya commands, Qualoth cloth simulator, panel‐rotation logic, core pattern operations, and utility functions.
• Output Dataset (SewFactory folder) – Stores synthetic images, ground‐truth pattern files, and Maya scene exports for training and evaluation.
B. Sewformer (Model Training & Inference)
• Configuration (configs/, system.json, environment.yaml, requirements.txt) – Defines training and testing hyperparameters (train.yaml, test.yaml), dataset paths, environment dependencies, and checkpoint locations.
• Data Layer (data/) – Dataset wrappers, SMPL body model integration (human_body_prior/), data transforms, panel‐class converters, and unified data loader exposing batches to the model.
• Models (models/) – Backbone CNNs, transformer encoder/decoder, positional encodings, garment‐specific detection heads (garment_detr_2d.py), and utility functions.
• Training Engine (train.py, trainer.py, metrics/) – Training loop orchestrating forward/backward passes, loss computation (composed_loss.py, eval_detr_metrics.py), scheduler (warm_cosine_scheduler.py), logging, and checkpoint saving.
• Inference & Evaluation (inference.py, experiment.py, metrics/) – Scripts to run inference on real/synthetic images, compute evaluation metrics, and export predicted panels for downstream simulation.
Component Relationships and Data Flow
• SewFactory → Produces a structured dataset on disk (images + pattern labels + simulation inputs).
• Sewformer Data Layer → Reads synthetic/real data using paths from system.json and YAML configs.
• Model → Receives image batch → Backbone (CNN) → Transformer → Panel detection & parameter regression heads → Output pattern predictions.
• Trainer → Calculates losses, updates model weights, stores checkpoints.
• Inference → Loads trained checkpoint, runs on new images, dumps predicted patterns.
• Simulation (optional) → Feeds predicted patterns back into SewFactory’s Maya‐based simulators (deepfashion_sim.py) to generate rendered results.
Key Architectural Patterns and Design Principles
• Modular separation: Data generation (SewFactory) vs. modeling (Sewformer) lives in separate directories.
• Config‐driven workflows: Almost every script reads from YAML/JSON configs, enabling reproducibility and hyperparameter management.
• Layered design:
• Data layer abstracts dataset formats (SMPL, panels, images).
• Model layer encapsulates neural architecture.
• Training/inference layer orchestrates execution and metrics.
• Reusability: MayaQLTools and Pattern packages serve as internal SDKs for garment operations.