How biansy000/chatgarment works
Project type and purpose
• ChatGarment is a full-stack machine-learning application that leverages large vision-language models (VLMs) to automate 3D-garment estimation, generation, and editing from image or text prompts.
• The core functionality lives in the “llava” Python package (model building, inference, serving, training, evaluation), with auxiliary scripts for end-to-end workflows (evaluation, postprocessing, 3D draping).
File structure
• Top-level directories
• llava/ → main code: model, utils, serve, train, eval
• scripts/ → shell and Python scripts for evaluation, postprocess, finetuning
• example_data/ → sample images, JSON prompts, sewing patterns
• docs/ → user docs, installation, postprocess instructions, prompt templates
• Configuration
• pyproject.toml, cog.yaml → project metadata, deployment config
• .devcontainer/ → development Docker setup
• .dockerignore, .gitignore
• Deployment & Serving
• llava/serve/ → CLI and Gradio-based web server, controller & worker registration
Key insights from README
• Three main inference use cases: image-based reconstruction, text-based generation, garment editing.
• Outputs: 2D sewing patterns (GarmentCode), JSON configs; optional 3D draping via run_garmentcode_sim.py.
• Relies on external repositories: GarmentCodeRC (pattern generation) and ContourCraft-CG (simulation).
Main components
A. Frontend / Entry points
• CLI scripts (scripts/v1_5/*.sh, run_garmentcode_sim.py)
• Gradio web UI (llava/serve/gradio_web_server.py)
B. Serving & Orchestration
• llava/serve/controller.py – central message router
• llava/serve/register_worker.py – service registry
• llava/serve/model_worker.py – hosts the ChatGarment model
• llava/serve/sglang_worker.py – SGLang adapter worker
• llava/serve/cli.py – command‐line interface
C. Core Model Stack
• llava/model/builder.py – fuses language model + vision adapter
• llava/model/language_model/* – base LLM wrappers (LLaMA, Mistral, MPT)
• llava/model/multimodal_encoder/* – image encoder (CLIP)
• llava/model/multimodal_projector/* – adapter layers into LLM
• llava/model/apply_delta.py, make_delta.py, consolidate.py – LoRA/delta tuning
D. Inference Pipeline
• llava/conversation.py – CoT multi-turn logic
• llava/prompts_utils.py – prompt templates, chaining
• llava/garment_utils_v2.py, garmentcode_utils.py – parse & format garment code
E. Evaluation & Benchmarking
• llava/eval/… – QA, VQA, science QA, textVQA, table benchmarks, webpage generator
F. Postprocessing & 3D Simulation
• scripts/postprocess/*.py – refine garment sizes
• run_garmentcode_sim.py – stitches patterns into 3D via PyTorch3D
G. Data & Configuration
• example_data/ – sample inputs and expected outputs
• docs/ – installation, usage, prompts, images
Relationships and data flow
1. User uploads image/text → Frontend (CLI or Gradio)
2. Controller routes request → model_worker
3. model_worker invokes builder → loads LLM + vision encoders + adapters
4. Inference pipeline (conversation)
• Image passed to CLIP encoder → embedded
• Prompt templating → fused into LLM context → generates garment instructions
5. Output garment code JSON → garmentcode_utils formats sewing patterns
6. Optional: run_garmentcode_sim.py consumes JSON → produces 3D mesh
7. Postprocess scripts adjust dimensions if necessary
Architectural patterns & design principles
• Modular separation: Serving layer, Model layer, Evaluation layer, Postprocess layer
• LoRA/delta tuning methodology for efficient fine-tuning
• Adapter pattern for multimodal fusion (projector modules inject vision features into LLM)
• Service registry for worker discovery (register_worker/controller)
• Clear separation of concerns: data ingestion, model inference, result formatting, simulation
Technologies & frameworks
• Python 3.10+, PyTorch, LoRA (delta), HuggingFace-style model loading
• CLIP-based vision encoder, LLaMA/Mistral/MPT language models
• Gradio for web UI, Docker/devcontainer for reproducibility
• PyTorch3D for 3D garment simulation