How microsoft/cad-editor works
Project type and purpose
• This is a research‐grade machine‐learning system for text‐based CAD editing. It implements a “locate‐then‐infill” framework (two‐stage transformer finetuning on CAD sequences) plus an automated data‐synthesis pipeline and evaluation suite.
File structure
Top‐level directories and their roles:
• data/ – raw and intermediate data generation and pairing scripts
• finetune/ – training & sampling scripts, DeepSpeed config, mask/infill tasks
• hnc-cad/ – hooks into external CAD sample generator (customized ac_gen.py)
• prompt/ – zero‐shot and few‐shot GPT‐4 baseline implementations
• utils/ – CAD parsing, conversion, rendering, sampling, evaluation helpers
Other files:
• environment.yaml – conda environment
• README.md – installation, pipeline, training, inference, evaluation
README for additional insights
• Data preparation: 3 sub‐pipelines (paired CAD generation, instruction generation, merge/filter)
• Training: two stages (locate → infill), multi‐GPU via accelerate + DeepSpeed
• Inference: llama_sample.py for mask & infill
• Evaluation: parse_seq2obj, visual_obj, cad_img, sample_points, eval_cad, eval_dclip
Main components
A. Data Generation & Preprocessing
• External hnc-cad generator (ac_gen.py)
• obj → sequence parsing (utils/parse_obj2seq.py)
• sequence conversion & pairing (utils/convert.py, data/pair.py)
• Instruction synthesis – image captions (data/caption_image.py ➔ OpenAI API),
sequence captions (data/caption_sequence.py)
• Merging & filtering scripts (data/merge.py, data/filter_sequence.py)
B. Training Pipeline
1. Locate Stage
– create_mask.py ➔ masked sequences
– llama_finetune.py --task_type mask (DeepSpeed config, multi‐GPU)
2. Infill Stage
– llama_finetune.py --task_type infill / infill_selective
– selective data enhancement
C. Inference Pipeline
• llama_sample.py for mask generation
• llama_sample.py for infill generation
• Model checkpoints from HuggingFace (locate_stage, infill_stage)
D. Evaluation Suite
• parse_seq2obj.py ➔ CAD obj
• visual_obj.py + cad_img.py ➔ render & images
• sample_points.py ➔ point clouds
• eval_cad.py ➔ 3D geometry metrics
• eval_dclip.py ➔ directional CLIP score
E. Baseline Prompting
• prompt/gpt4_basic.py, prompt/gpt4_fs3.py
Relationships & Data Flow
Data Gen → Preprocessing → Training (Locate → Infill)
Training → Model Registry (HuggingFace checkpoints)
Model Registry + Test Data → Inference → Edited CAD sequences
Edited CAD + GT → Evaluation Suite → Metrics
• External services: OpenAI API (captioning), HuggingFace model hub
Architectural patterns & design principles
• Pipeline architecture: sequential ETL → train → infer → evaluate
• Two‐stage modular ML workflow (locate vs. infill)
• Script‐driven orchestration (Python CLIs)
• Reusable utilities under utils/ for parsing, rendering, evaluation
Relevant technologies & frameworks
• Python 3, Conda, DeepSpeed/Accelerate
• LLaMA transformer via HuggingFace Transformers
• OpenAI API for text captioning
• OBJ file parsing & geometry utilities
• PyTorch (implied by accelerate)