How crockwell/cap3d works
Project type and purpose
• From the file tree and README, this is a research codebase for “Scalable 3D Captioning with Pretrained Models” and associated text-to-3D evaluation/fine-tuning.
• It is not a traditional frontend/backend web app, but a collection of Python pipelines and libraries that ingest 3D object datasets, render multi-view images, run pretrained captioning models (BLIP2/CLIP+GPT), and evaluate or fine-tune 3D generative models (Point-E and Shap-E).
File structure
• Top-level directories:
• captioning_pipeline – scripts to render 3D models, extract multi-view images, run captioning.
• text-to-3D – evaluation and fine-tuning scripts for text-to-3D, divided into two subpackages: point-e and shap-e.
• Each subpackage (point-e, shap-e) is structured like a standalone library: diffusion modules, model definitions, evaluators, example notebooks, rendering scripts, setup.py.
• Configuration and orchestration scripts: render_script.py, caption_blip2.py, caption_clip_gpt.py, evaluate.py, finetune_pointE.py, finetune_shapE.py.
README for additional insights
• The captioning pipeline ingests 3D object files (GLBs from Objaverse/ABO), renders images with Blender + PyTorch3D, then passes those images through BLIP2 or CLIP to extract features, and finally prompts an LLM (e.g., GPT) for the final caption.
• The text-to-3D folder provides evaluation pipelines (evaluate.py) and fine-tuning on two generative architectures: Point-E and Shap-E, leveraging diffusion models and rendering utilities.
Detail without over-splitting
• Logical step (rendering, feature extraction, LLM inference, diffusion sampling, mesh conversion).
• Notes on configuration scripts (e.g., setup.py in subpackages) to show how each library is packaged.