How hmrishavbandy/dy3d works
File-Structure Analysis
• Top-level modules:
• data/ → dataset definitions, sample data, file-listing JSONs
• models/ → neural-network definitions (ResNet, transformer, latent diffusion model)
• train/ → Python training entry-points (segment_train.py, diffusion_train.py, precompute_encodings.py)
• scripts/ → Bash wrappers to launch the training and preprocessing pipelines
• utils/ → general helper functions, and a submodule utils_spaghetti/ that interfaces with SPAGHETTI
• assets/, README, LICENSE, etc.
• The directory layout reflects a clear separation of: data ingestion, model definitions, training logic, and utility code.
README Insights
• The system has three sequential phases:
a. Dataset synthesis (inversion with SPAGHETTI → aligned latents → sketch renderings + segment maps)
b. Sketch segmentation network training (ResNet-based CNN)
c. Conditional diffusion model training (LDM) preceded by sketch-encoding precomputation
a. Main Components
1. External Tools:
– SPAGHETTI library (latent inversion)
– CLIPasso (sketch edgemap generation)
2. Data‐Synthesis Module (data/):
– Invert meshes → produce m×d latents
– Align latents → samples_final.npy
– Render sketches & segment overlays → segment maps (.exr) + sketch crop params
3. Precompute Module (train/precompute_encodings.py + scripts/precompute.sh):
– Load sample sketches & segment maps
– Generate fixed sketch embeddings for diffusion network
4. Segmentation‐Training Module (train/segment_train.py + scripts/segment_train.sh):
– Model: models/resnet_model.py
– Input: sketch images + segment maps
– Output: segmentation checkpoint
5. Diffusion‐Training Module (train/diffusion_train.py + scripts/diffusion_train.sh):
– Model: models/ldm.py, models/transformer.py
– Input: (precomputed sketch encodings, aligned latents)
– Output: trained diffusion model checkpoint
6. Utilities (utils/, utils_spaghetti/):
– Latent & mesh processing, meshing (mcubes_meshing), file I/O, CLI parsing
b. Relationships & Data Flow
• Data-Synthesis → outputs raw dataset
• Segmentation training consumes raw sketches & segment maps → saves segm. model
• Precompute module uses raw sketches + segment model → sketch encodings
• Diffusion training consumes sketch encodings + aligned latents → diffusion model
• At inference (not detailed in code): feed new sketch → segmentation net → encoding → diffusion net → latent → SPAGHETTI decoder → mesh output
c. Architectural Patterns & Principles
• Modular pipeline: each stage isolated (data, segmentation, diffusion)
• Reusable components: common utils, model definitions separate from training code
• External-tool integration: wrappers in data and utils_spaghetti
d. Key Technologies
• Python 3.x, PyTorch (implicit in model code)
• Bash scripts for orchestration
• SPAGHETTI library (external)
• CLIPasso (external)
• File formats: .npy, .exr, JSON, text
Essential Elements to Capture
• Explicit module boundaries (data/, models/, train/, utils/)
• Sequence of pipeline stages
• External libraries and I/O artifacts
• Flow of intermediate artifacts: .npy latents, .exr segment maps, model checkpoints, precomputed encodings