# crockwell/cap3d architecture

> Architecture diagram of the GitHub repository crockwell/cap3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/crockwell/cap3d
- Repository: https://github.com/crockwell/cap3d

## Overview

##### Project type and purpose
- From the file tree and README, this is a research codebase for “Scalable 3D Captioning with Pretrained Models” and associated text-to-3D evaluation/fine-tuning.
- It is not a traditional frontend/backend web app, but a collection of Python pipelines and libraries that ingest 3D object datasets, render multi-view images, run pretrained captioning models (BLIP2/CLIP+GPT), and evaluate or fine-tune 3D generative models (Point-E and Shap-E).

##### File structure
- Top-level directories:
- captioning_pipeline – scripts to render 3D models, extract multi-view images, run captioning.
- text-to-3D – evaluation and fine-tuning scripts for text-to-3D, divided into two subpackages: point-e and shap-e.
- Each subpackage (point-e, shap-e) is structured like a standalone library: diffusion modules, model definitions, evaluators, example notebooks, rendering scripts, setup.py.
- Configuration and orchestration scripts: render_script.py, caption_blip2.py, caption_clip_gpt.py, evaluate.py, finetune_pointE.py, finetune_shapE.py.

##### README for additional insights
- The captioning pipeline ingests 3D object files (GLBs from Objaverse/ABO), renders images with Blender + PyTorch3D, then passes those images through BLIP2 or CLIP to extract features, and finally prompts an LLM (e.g., GPT) for the final caption.
- The text-to-3D folder provides evaluation pipelines (evaluate.py) and fine-tuning on two generative architectures: Point-E and Shap-E, leveraging diffusion models and rendering utilities.

##### Detail without over-splitting
- Logical step (rendering, feature extraction, LLM inference, diffusion sampling, mesh conversion).
- Notes on configuration scripts (e.g., setup.py in subpackages) to show how each library is packaged.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Data Sources"
        Objaverse["Objaverse GLB Dataset"]:::external
        ABO["ABO GLB Dataset"]:::external
        Example3D["Example 3D Inputs"]:::external
    end

    %% Captioning Pipeline
    subgraph "Captioning Pipeline"
        Renderer["Renderer<br/>(Blender + PyTorch3D)"]:::pipeline
        BLIP2["BLIP2 Feature Extractor"]:::pipeline
        CLIPGPT["CLIP+GPT Captioner"]:::pipeline
        JSONOut["Caption Output<br/>(captions.json)"]:::output
    end

    %% Shared Utilities & External Services
    subgraph "Shared Utilities & External Services"
        BlenderSvc["Blender"]:::service
        PyTorch3DSvc["PyTorch3D"]:::service
        PyTorchSvc["PyTorch"]:::service
        HFDS["Hugging Face Datasets"]:::service
        LLMSvc["LLM Service (GPT)"]:::service
        Storage["Storage<br/>(Local/S3/HF)"]:::service
    end

    %% Text-to-3D Pipeline
    subgraph "Text-to-3D Pipeline"
        Prompt["Text Prompt"]:::external
        Evaluate["Evaluation Module"]:::pipeline
        PointEFine["Fine-tune Point-E"]:::pipeline
        ShapEFine["Fine-tune Shap-E"]:::pipeline
        Vis["Rendering & Visualization"]:::pipeline
        MeshOut["Generated Meshes"]:::output
        Report["Metric Reports"]:::output

        Prompt -->|"Text →"| PointEFine
        Prompt -->|"Text →"| ShapEFine
        PointEFine -->|"Point Cloud →"| Vis
        ShapEFine -->|"Point Cloud →"| Vis
        Vis -->|"Mesh →"| MeshOut
        MeshOut -->|"Evaluate Inputs"| Evaluate
        Evaluate -->|"Metrics →"| Report
    end

    %% Point-E Submodule
    subgraph "Point-E Submodule"
        PESetup["setup.py"]:::library
        PERender["render_script_pointE.py"]:::library
        PEText2PLY["text2ply_pointE.py"]:::library
        PECore["Core Library"]:::library
    end
    PointEFine --> PESetup
    PointEFine --> PERender
    PointEFine --> PEText2PLY
    PointEFine --> PECore

    %% Shap-E Submodule
    subgraph "Shap-E Submodule"
        SESetup["setup.py"]:::library
        SEExtract["extract_pointcloud.py"]:::library
        SERender["render_script_shapE.py"]:::library
        SEText2IMG["text2img_shapE.py"]:::library
        SEText2PLY["text2ply_shapE.py"]:::library
        SECore["Core Library"]:::library
    end
    ShapEFine --> SESetup
    ShapEFine --> SEExtract
    ShapEFine --> SERender
    ShapEFine --> SEText2IMG
    ShapEFine --> SEText2PLY
    ShapEFine --> SECore

    %% Connections between sources and captioning
    Objaverse -->|"GLB → Images"| Renderer
    ABO -->|"GLB → Images"| Renderer
    Example3D --> Renderer
    Renderer -->|"Images → Features"| BLIP2
    BLIP2 -->|"Features → Captions"| CLIPGPT
    CLIPGPT --> JSONOut

    %% Connections to external services
    Renderer --> BlenderSvc
    Renderer --> PyTorch3DSvc
    BLIP2 --> PyTorchSvc
    BLIP2 --> HFDS
    CLIPGPT --> LLMSvc
    JSONOut --> Storage
    PointEFine --> PyTorchSvc
    ShapEFine --> PyTorchSvc
    Evaluate --> PyTorchSvc
    Evaluate --> HFDS
    Vis --> BlenderSvc

    %% Click Events
    click Renderer "https://github.com/crockwell/cap3d/blob/main/captioning_pipeline/render_script.py"
    click BLIP2 "https://github.com/crockwell/cap3d/blob/main/captioning_pipeline/caption_blip2.py"
    click CLIPGPT "https://github.com/crockwell/cap3d/blob/main/captioning_pipeline/caption_clip_gpt.py"
    click Example3D "https://github.com/crockwell/cap3d/tree/main/captioning_pipeline/example_material/glbs/"
    click Evaluate "https://github.com/crockwell/cap3d/blob/main/text-to-3D/evaluate.py"
    click PointEFine "https://github.com/crockwell/cap3d/blob/main/text-to-3D/finetune_pointE.py"
    click ShapEFine "https://github.com/crockwell/cap3d/blob/main/text-to-3D/finetune_shapE.py"
    click PESetup "https://github.com/crockwell/cap3d/blob/main/text-to-3D/point-e/setup.py"
    click PERender "https://github.com/crockwell/cap3d/blob/main/text-to-3D/point-e/render_script_pointE.py"
    click PEText2PLY "https://github.com/crockwell/cap3d/blob/main/text-to-3D/point-e/text2ply_pointE.py"
    click PECore "https://github.com/crockwell/cap3d/tree/main/text-to-3D/point-e/point_e/"
    click SESetup "https://github.com/crockwell/cap3d/blob/main/text-to-3D/shap-e/setup.py"
    click SEExtract "https://github.com/crockwell/cap3d/blob/main/text-to-3D/shap-e/extract_pointcloud.py"
    click SERender "https://github.com/crockwell/cap3d/blob/main/text-to-3D/shap-e/render_script_shapE.py"
    click SEText2IMG "https://github.com/crockwell/cap3d/blob/main/text-to-3D/shap-e/text2img_shapE.py"
    click SEText2PLY "https://github.com/crockwell/cap3d/blob/main/text-to-3D/shap-e/text2ply_shapE.py"
    click SECore "https://github.com/crockwell/cap3d/tree/main/text-to-3D/shap-e/shap_e/"

    %% Styles
    classDef pipeline fill:#D0E8FF,stroke:#0366d6,color:#0366d6
    classDef external fill:#E8E8E8,stroke:#666,color:#333
    classDef service fill:#D0FFD0,stroke:#2E7D32,color:#2E7D32
    classDef output fill:#FFE0B2,stroke:#E65100,color:#E65100
    classDef library fill:#FFF2CC,stroke:#BF360C,color:#BF360C
```
