# biansy000/chatgarment architecture

> Architecture diagram of the GitHub repository biansy000/chatgarment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/biansy000/chatgarment
- Repository: https://github.com/biansy000/chatgarment

## Overview

##### Project type and purpose
- ChatGarment is a full-stack machine-learning application that leverages large vision-language models (VLMs) to automate 3D-garment estimation, generation, and editing from image or text prompts.
- The core functionality lives in the “llava” Python package (model building, inference, serving, training, evaluation), with auxiliary scripts for end-to-end workflows (evaluation, postprocessing, 3D draping).

##### File structure
- Top-level directories
- llava/             → main code: model, utils, serve, train, eval
- scripts/           → shell and Python scripts for evaluation, postprocess, finetuning
- example_data/      → sample images, JSON prompts, sewing patterns
- docs/              → user docs, installation, postprocess instructions, prompt templates
- Configuration
- pyproject.toml, cog.yaml → project metadata, deployment config
- .devcontainer/           → development Docker setup
- .dockerignore, .gitignore
- Deployment & Serving
- llava/serve/ → CLI and Gradio-based web server, controller & worker registration

##### Key insights from README
- Three main inference use cases: image-based reconstruction, text-based generation, garment editing.
- Outputs: 2D sewing patterns (GarmentCode), JSON configs; optional 3D draping via run_garmentcode_sim.py.
- Relies on external repositories: GarmentCodeRC (pattern generation) and ContourCraft-CG (simulation).

##### Main components
A. Frontend / Entry points
- CLI scripts (scripts/v1_5/*.sh, run_garmentcode_sim.py)
- Gradio web UI (llava/serve/gradio_web_server.py)
B. Serving & Orchestration
- llava/serve/controller.py          – central message router
- llava/serve/register_worker.py     – service registry
- llava/serve/model_worker.py        – hosts the ChatGarment model
- llava/serve/sglang_worker.py       – SGLang adapter worker
- llava/serve/cli.py                 – command‐line interface
C. Core Model Stack
- llava/model/builder.py             – fuses language model + vision adapter
- llava/model/language_model/*       – base LLM wrappers (LLaMA, Mistral, MPT)
- llava/model/multimodal_encoder/*   – image encoder (CLIP)
- llava/model/multimodal_projector/* – adapter layers into LLM
- llava/model/apply_delta.py, make_delta.py, consolidate.py – LoRA/delta tuning
D. Inference Pipeline
- llava/conversation.py              – CoT multi-turn logic
- llava/prompts_utils.py             – prompt templates, chaining
- llava/garment_utils_v2.py, garmentcode_utils.py – parse & format garment code
E. Evaluation & Benchmarking
- llava/eval/…                       – QA, VQA, science QA, textVQA, table benchmarks, webpage generator
F. Postprocessing & 3D Simulation
- scripts/postprocess/*.py           – refine garment sizes
- run_garmentcode_sim.py              – stitches patterns into 3D via PyTorch3D
G. Data & Configuration
- example_data/                       – sample inputs and expected outputs
- docs/                               – installation, usage, prompts, images

##### Relationships and data flow
1. User uploads image/text → Frontend (CLI or Gradio)
2. Controller routes request → model_worker
3. model_worker invokes builder → loads LLM + vision encoders + adapters
4. Inference pipeline (conversation)
- Image passed to CLIP encoder → embedded
- Prompt templating → fused into LLM context → generates garment instructions
5. Output garment code JSON → garmentcode_utils formats sewing patterns
6. Optional: run_garmentcode_sim.py consumes JSON → produces 3D mesh
7. Postprocess scripts adjust dimensions if necessary

##### Architectural patterns & design principles
- Modular separation: Serving layer, Model layer, Evaluation layer, Postprocess layer
- LoRA/delta tuning methodology for efficient fine-tuning
- Adapter pattern for multimodal fusion (projector modules inject vision features into LLM)
- Service registry for worker discovery (register_worker/controller)
- Clear separation of concerns: data ingestion, model inference, result formatting, simulation

##### Technologies & frameworks
- Python 3.10+, PyTorch, LoRA (delta), HuggingFace-style model loading
- CLIP-based vision encoder, LLaMA/Mistral/MPT language models
- Gradio for web UI, Docker/devcontainer for reproducibility
- PyTorch3D for 3D garment simulation

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External User
    EX["User"]:::external

    %% Frontend
    subgraph "Frontend" 
        direction TB
        CLI1["evaluate_garment_v2_demo_edit.sh"]:::frontend
        CLI2["evaluate_garment_v2_eva_edit.sh"]:::frontend
        CLI3["evaluate_garment_v2_imggen_2step.sh"]:::frontend
        CLI4["evaluate_garment_v2_textgen.sh"]:::frontend
        CLI5["evaluate_garment_v2_textgen_fromimg.sh"]:::frontend
        CLI6["finetune_task_lora_garmentcode_outfit.sh"]:::frontend
        SIM["run_garmentcode_sim.py"]:::frontend
        GR["Gradio Web UI"]:::frontend
    end

    %% Serving & Orchestration
    subgraph "Serving & Orchestration" 
        direction TB
        CONT["Controller"]:::serving
        REG["Service Registry"]:::serving
        MW["Model Worker"]:::serving
        SG["SGLang Adapter Worker"]:::serving
        CLI_SERV["CLI Interface"]:::serving
    end

    %% Core Model & Inference
    subgraph "Core Model & Inference" 
        direction TB
        BUIL["Model Builder"]:::model
        LLM["Language Model Wrappers"]:::model
        ENC["Multimodal Encoder"]:::model
        PROJ["Multimodal Projector"]:::model
        DEL1["apply_delta.py"]:::model
        DEL2["make_delta.py"]:::model
        DEL3["consolidate.py"]:::model
        CONV["Conversation Logic"]:::model
        PROM["Prompt Utilities"]:::model
        GARV["garment_utils_v2.py"]:::model
        GCOD["garmentcode_utils.py"]:::model
        OUTPUT_JSON["Garment Code JSON"]:::data
    end

    %% Evaluation & Benchmarking
    subgraph "Evaluation & Benchmarking" 
        direction TB
        EVAL["llava/eval"]:::eval
    end

    %% Postprocess & Simulation
    subgraph "Postprocess & Simulation" 
        direction TB
        PPS["Postprocess Scripts"]:::post
        MESH["3D Mesh Output"]:::post
    end

    %% Connections
    EX -->|invoke|CLI1
    EX -->|invoke|CLI2
    EX -->|invoke|CLI3
    EX -->|invoke|CLI4
    EX -->|invoke|CLI5
    EX -->|invoke|CLI6
    EX -->|invoke|GR
    EX -->|invoke|SIM

    CLI1 -->|request|CONT
    CLI2 -->|request|CONT
    CLI3 -->|request|CONT
    CLI4 -->|request|CONT
    CLI5 -->|request|CONT
    CLI6 -->|request|CONT
    GR -->|request|CONT

    CONT -->|route to|MW
    REG -->|registers|MW
    MW -->|build model|BUIL
    BUIL -->|uses|LLM
    BUIL -->|uses|ENC
    BUIL -->|uses|PROJ
    BUIL -->|uses|DEL1
    BUIL -->|uses|DEL2
    BUIL -->|uses|DEL3

    ENC -->|embed image|CONV
    PROM -->|template prompts|CONV
    CONV -->|produce instructions|GARV
    GARV -->|format code|GCOD
    GCOD -->|output|OUTPUT_JSON

    OUTPUT_JSON -.->|optional|SIM
    OUTPUT_JSON -.->|optional|PPS
    PPS -.->|refine dims|MESH

    %% Click Events
    click CLI1 "https://github.com/biansy000/chatgarment/blob/main/scripts/v1_5/evaluate_garment_v2_demo_edit.sh"
    click CLI2 "https://github.com/biansy000/chatgarment/blob/main/scripts/v1_5/evaluate_garment_v2_eva_edit.sh"
    click CLI3 "https://github.com/biansy000/chatgarment/blob/main/scripts/v1_5/evaluate_garment_v2_imggen_2step.sh"
    click CLI4 "https://github.com/biansy000/chatgarment/blob/main/scripts/v1_5/evaluate_garment_v2_textgen.sh"
    click CLI5 "https://github.com/biansy000/chatgarment/blob/main/scripts/v1_5/evaluate_garment_v2_textgen_fromimg.sh"
    click CLI6 "https://github.com/biansy000/chatgarment/blob/main/scripts/v1_5/finetune_task_lora_garmentcode_outfit.sh"
    click SIM "https://github.com/biansy000/chatgarment/blob/main/run_garmentcode_sim.py"
    click GR "https://github.com/biansy000/chatgarment/blob/main/llava/serve/gradio_web_server.py"
    click CONT "https://github.com/biansy000/chatgarment/blob/main/llava/serve/controller.py"
    click REG "https://github.com/biansy000/chatgarment/blob/main/llava/serve/register_worker.py"
    click MW "https://github.com/biansy000/chatgarment/blob/main/llava/serve/model_worker.py"
    click SG "https://github.com/biansy000/chatgarment/blob/main/llava/serve/sglang_worker.py"
    click CLI_SERV "https://github.com/biansy000/chatgarment/blob/main/llava/serve/cli.py"
    click BUIL "https://github.com/biansy000/chatgarment/blob/main/llava/model/builder.py"
    click LLM "https://github.com/biansy000/chatgarment/tree/main/llava/model/language_model/"
    click ENC "https://github.com/biansy000/chatgarment/tree/main/llava/model/multimodal_encoder/"
    click PROJ "https://github.com/biansy000/chatgarment/tree/main/llava/model/multimodal_projector/"
    click DEL1 "https://github.com/biansy000/chatgarment/blob/main/llava/model/apply_delta.py"
    click DEL2 "https://github.com/biansy000/chatgarment/blob/main/llava/model/make_delta.py"
    click DEL3 "https://github.com/biansy000/chatgarment/blob/main/llava/model/consolidate.py"
    click CONV "https://github.com/biansy000/chatgarment/blob/main/llava/conversation.py"
    click PROM "https://github.com/biansy000/chatgarment/blob/main/llava/prompts_utils.py"
    click GARV "https://github.com/biansy000/chatgarment/blob/main/llava/garment_utils_v2.py"
    click GCOD "https://github.com/biansy000/chatgarment/blob/main/llava/garmentcode_utils.py"
    click EVAL "https://github.com/biansy000/chatgarment/tree/main/llava/eval/"
    click PPS "https://github.com/biansy000/chatgarment/tree/main/scripts/postprocess/"
    click MESH "https://github.com/biansy000/chatgarment/blob/main/run_garmentcode_sim.py"

    %% Styles
    classDef external fill:#ffffff,stroke:#000,stroke-width:1px
    classDef frontend fill:#ccffcc,stroke:#069,stroke-width:1px
    classDef serving fill:#ccddff,stroke:#036,stroke-width:1px
    classDef model fill:#ffddaa,stroke:#963,stroke-width:1px
    classDef data fill:#eeeeee,stroke:#666,stroke-width:1px,shape:cylinder
    classDef eval fill:#cccccc,stroke:#555,stroke-width:1px
    classDef post fill:#e0ccff,stroke:#606,stroke-width:1px
```
