# hmrishavbandy/dy3d architecture

> Architecture diagram of the GitHub repository hmrishavbandy/dy3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hmrishavbandy/dy3d
- Repository: https://github.com/hmrishavbandy/dy3d

## Overview

##### File-Structure Analysis
- Top-level modules:
- data/        → dataset definitions, sample data, file-listing JSONs
- models/      → neural-network definitions (ResNet, transformer, latent diffusion model)
- train/       → Python training entry-points (segment_train.py, diffusion_train.py, precompute_encodings.py)
- scripts/     → Bash wrappers to launch the training and preprocessing pipelines
- utils/       → general helper functions, and a submodule utils_spaghetti/ that interfaces with SPAGHETTI
- assets/, README, LICENSE, etc.
- The directory layout reflects a clear separation of: data ingestion, model definitions, training logic, and utility code.

##### README Insights
- The system has three sequential phases:
a. Dataset synthesis (inversion with SPAGHETTI → aligned latents → sketch renderings + segment maps)
b. Sketch segmentation network training (ResNet-based CNN)
c. Conditional diffusion model training (LDM) preceded by sketch-encoding precomputation

a. Main Components
1. External Tools:
– SPAGHETTI library (latent inversion)
– CLIPasso (sketch edgemap generation)
2. Data‐Synthesis Module (data/):
– Invert meshes → produce m×d latents
– Align latents → samples_final.npy
– Render sketches & segment overlays → segment maps (.exr) + sketch crop params
3. Precompute Module (train/precompute_encodings.py + scripts/precompute.sh):
– Load sample sketches & segment maps
– Generate fixed sketch embeddings for diffusion network
4. Segmentation‐Training Module (train/segment_train.py + scripts/segment_train.sh):
– Model: models/resnet_model.py
– Input: sketch images + segment maps
– Output: segmentation checkpoint
5. Diffusion‐Training Module (train/diffusion_train.py + scripts/diffusion_train.sh):
– Model: models/ldm.py, models/transformer.py
– Input: (precomputed sketch encodings, aligned latents)
– Output: trained diffusion model checkpoint
6. Utilities (utils/, utils_spaghetti/):
– Latent & mesh processing, meshing (mcubes_meshing), file I/O, CLI parsing
b. Relationships & Data Flow
- Data-Synthesis → outputs raw dataset
- Segmentation training consumes raw sketches & segment maps → saves segm. model
- Precompute module uses raw sketches + segment model → sketch encodings
- Diffusion training consumes sketch encodings + aligned latents → diffusion model
- At inference (not detailed in code): feed new sketch → segmentation net → encoding → diffusion net → latent → SPAGHETTI decoder → mesh output
c. Architectural Patterns & Principles
- Modular pipeline: each stage isolated (data, segmentation, diffusion)
- Reusable components: common utils, model definitions separate from training code
- External-tool integration: wrappers in data and utils_spaghetti
d. Key Technologies
- Python 3.x, PyTorch (implicit in model code)
- Bash scripts for orchestration
- SPAGHETTI library (external)
- CLIPasso (external)
- File formats: .npy, .exr, JSON, text
##### Essential Elements to Capture
- Explicit module boundaries (data/, models/, train/, utils/)
- Sequence of pipeline stages
- External libraries and I/O artifacts
- Flow of intermediate artifacts: .npy latents, .exr segment maps, model checkpoints, precomputed encodings

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Dependencies
    subgraph "External Tools"
        direction TB
        SPAGHETTI["SPAGHETTI Library"]:::ext
        CLIPasso["CLIPasso Edgemap Gen"]:::ext
    end

    %% Data & Preprocessing Layer
    subgraph "Data & Preprocessing" 
        direction TB
        DataSynthesis["Data-Synthesis Module"]:::data
        RawLatents["Raw Latents (.npy)"]:::dataStore
        SegmentMaps["Segment Maps (.exr)"]:::dataStore
        SketchParams["Sketch Params (.txt)"]:::dataStore
    end

    %% Segmentation Training Layer
    subgraph "Segmentation Training" 
        direction TB
        SegmentTrain["segment_train.py"]:::train
        ResNetModel["resnet_model.py"]:::model
        SegCheckpoint["Segmentation Checkpoint"]:::dataStore
    end

    %% Precompute Layer
    subgraph "Precompute Encodings"
        direction TB
        Precompute["precompute_encodings.py"]:::train
        Encodings["Sketch Encodings (.npy)"]:::dataStore
    end

    %% Diffusion Training Layer
    subgraph "Diffusion Training"
        direction TB
        DiffusionTrain["diffusion_train.py"]:::train
        LDM["ldm.py"]:::model
        Transformer["transformer.py"]:::model
        DiffCheckpoint["Diffusion Checkpoint"]:::dataStore
    end

    %% Utilities
    subgraph "Utilities"
        direction TB
        BaseUtils["base_utils.py"]:::util
        SpaghettiUtils["spaghetti_utils.py"]:::util
        UtilsSpaghetti["utils_spaghetti/"]:::util
    end

    %% Data Stores for Dataset Definitions
    subgraph "Dataset Files" 
        direction TB
        DsetDiff["dset_diffusion.py"]:::data
        DsetSeg["dset_segment.py"]:::data
        Lists["data/lists/"]:::data
        SampleSegment["data/sample/segment/"]:::dataStore
        SampleSketch["data/sample/sketch/"]:::dataStore
    end

    %% Scripts
    subgraph "Scripts"
        direction TB
        ScriptPrecompute["scripts/precompute.sh"]:::script
        ScriptSeg["scripts/segment_train.sh"]:::script
        ScriptDiff["scripts/diffusion_train.sh"]:::script
    end

    %% Flow
    SPAGHETTI -->|invert meshes| DataSynthesis
    CLIPasso -->|generate edgemaps| DataSynthesis

    DataSynthesis -->|produce latents| RawLatents
    DataSynthesis -->|produce segments| SegmentMaps
    DataSynthesis -->|produce params| SketchParams

    RawLatents --> DiffusionTrain
    SegmentMaps --> SegmentTrain
    SketchParams --> SegmentTrain

    SegmentTrain -->|save checkpoint| SegCheckpoint
    SegCheckpoint --> Precompute

    Precompute -->|save encodings| Encodings
    Encodings --> DiffusionTrain

    DiffusionTrain -->|save checkpoint| DiffCheckpoint

    %% Script orchestration
    ScriptPrecompute --> Precompute
    ScriptSeg --> SegmentTrain
    ScriptDiff --> DiffusionTrain

    %% Modules and Models usage
    SegmentTrain --> ResNetModel
    DiffusionTrain --> LDM
    DiffusionTrain --> Transformer

    %% Utilities usage
    DataSynthesis --> BaseUtils
    DataSynthesis --> UtilsSpaghetti
    SegmentTrain --> BaseUtils
    Precompute --> BaseUtils
    DiffusionTrain --> BaseUtils
    DiffusionTrain --> SpaghettiUtils

    %% Click Events
    click DataSynthesis "https://github.com/hmrishavbandy/dy3d/tree/main/data/"
    click DsetDiff "https://github.com/hmrishavbandy/dy3d/blob/main/data/dset_diffusion.py"
    click DsetSeg "https://github.com/hmrishavbandy/dy3d/blob/main/data/dset_segment.py"
    click Lists "https://github.com/hmrishavbandy/dy3d/blob/main/data/lists/ShapeNet.json"
    click SampleSegment "https://github.com/hmrishavbandy/dy3d/tree/main/data/sample/segment/"
    click SampleSketch "https://github.com/hmrishavbandy/dy3d/tree/main/data/sample/sketch/"
    click Precompute "https://github.com/hmrishavbandy/dy3d/blob/main/train/precompute_encodings.py"
    click ScriptPrecompute "https://github.com/hmrishavbandy/dy3d/blob/main/scripts/precompute.sh"
    click SegmentTrain "https://github.com/hmrishavbandy/dy3d/blob/main/train/segment_train.py"
    click ScriptSeg "https://github.com/hmrishavbandy/dy3d/blob/main/scripts/segment_train.sh"
    click ResNetModel "https://github.com/hmrishavbandy/dy3d/blob/main/models/resnet_model.py"
    click DiffusionTrain "https://github.com/hmrishavbandy/dy3d/blob/main/train/diffusion_train.py"
    click ScriptDiff "https://github.com/hmrishavbandy/dy3d/blob/main/scripts/diffusion_train.sh"
    click LDM "https://github.com/hmrishavbandy/dy3d/blob/main/models/ldm.py"
    click Transformer "https://github.com/hmrishavbandy/dy3d/blob/main/models/transformer.py"
    click BaseUtils "https://github.com/hmrishavbandy/dy3d/blob/main/utils/base_utils.py"
    click SpaghettiUtils "https://github.com/hmrishavbandy/dy3d/blob/main/utils/spaghetti_utils.py"
    click UtilsSpaghetti "https://github.com/hmrishavbandy/dy3d/tree/main/utils/utils_spaghetti/"

    %% Styling
    classDef ext fill:#ccc,stroke:#333;
    classDef data fill:#bbf,stroke:#333;
    classDef dataStore fill:#bbf,stroke:#333,stroke-dasharray: 5 5;
    classDef train fill:#afa,stroke:#333;
    classDef model fill:#f96,stroke:#333;
    classDef util fill:#eee,stroke:#333;
    classDef script fill:#ddf,stroke:#333;
```
