How microsoft/cadfusion works
CADFusion turns text descriptions into CAD command sequences, then improves generation through an alternating visual-feedback loop. The documented workflow prepares and renders CAD data, captions it for sequential learning, generates candidate models, evaluates their rendered images, builds preference pairs, and fine-tunes again. The graph also includes CAD-to-solid conversion and evaluation capabilities. Several links follow the README’s documented workflow; unsampled implementation details are not inferred beyond those statements.
Main entry points: make_dpo_dataset.py Visual scoring · parser_visual.py Solid conversion · parser.py Sequence parser · convert.py SL conversion
Components 21
Data preparation
- SL conversion
src/data_preprocessing/convert.py - Provided data conversion
data/sl_data/convert.py - LLM captioning
src/data_preprocessing/captioning.py - Azure API client
src/data_preprocessing/call_openai.py
CAD rendering
- Sequence parser
src/rendering_utils/parser.py - OBJ geometry parser
src/rendering_utils/geometry/obj_parser.py - Curve primitives
src/rendering_utils/geometry - Image renderer
src/rendering_utils/img_renderer.py - Solid conversion
src/rendering_utils/parser_visual.py - OBJ reconverter
src/rendering_utils/utils/obj_reconverter.py
Model learning
- SL dataset
src/train/CAD_dataset.py - Sequential fine-tuning
src/train/llama_finetune.py - CAD generation
src/test/inference.py - DPO fine-tuning
src/train/dpo.py
Feedback and evaluation
- Visual scoring
src/dpo/make_dpo_dataset.py - LLaVA scoring
src/dpo/llava_utils.py - GPT scoring
src/dpo/openai_utils.py - Preference pairs
src/dpo/make_dpo_dataset.py - Generation evaluation
src/test/dist_eval.py
Other components
- Researcher
- Azure OpenAI
Connections 23
- Researcher → Provided data conversion: converts data
README.md - Researcher → SL conversion: prepares data
README.md - SL conversion → Sequence parser: renders sequences
README.md - Sequence parser → Image renderer: renders images
README.md - Sequence parser → Solid conversion: converts CAD
README.md - OBJ geometry parser → Curve primitives: uses curves
src/rendering_utils/geometry/obj_parser.py - Solid conversion → OBJ geometry parser: parses OBJ
src/rendering_utils/parser_visual.py - Solid conversion → OBJ reconverter: builds solids
src/rendering_utils/parser_visual.py - Researcher → LLM captioning: captions images
README.md - LLM captioning → Azure API client: calls API
src/data_preprocessing/captioning.py - Azure API client → Azure OpenAI: requests completions
src/data_preprocessing/call_openai.py - SL dataset → Sequential fine-tuning: supplies examples
src/train/llama_finetune.py - Researcher → Sequential fine-tuning: trains model
README.md - Researcher → CAD generation: generates samples
README.md - CAD generation → Solid conversion: renders models
README.md - Researcher → Visual scoring: scores generations
README.md - Visual scoring → LLaVA scoring: uses local model
src/dpo/make_dpo_dataset.py - Visual scoring → GPT scoring: uses GPT option
src/dpo/make_dpo_dataset.py - GPT scoring → Azure OpenAI: requests scores
- Visual scoring → Preference pairs: builds pairs
src/dpo/make_dpo_dataset.py - Researcher → DPO fine-tuning: fine-tunes model
README.md - Preference pairs → DPO fine-tuning: trains on pairs
README.md - Researcher → Generation evaluation: evaluates samples
README.md