# openai/shap-e architecture

> Architecture diagram of the GitHub repository openai/shap-e, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openai/shap-e
- Repository: https://github.com/openai/shap-e

## Overview

##### Project Type & Purpose
- Type: Open-source machine-learning library for generating conditional 3D implicit functions.
- Purpose: Take a text prompt or image, run it through diffusion and encoding/decoding pipelines, and produce a 3D model or mesh that can be rendered as GIFs or other formats.

##### High-Level Components
Four main layers and their subcomponents:
A. Client Layer (Notebooks & CLI)
– sample_text_to_3d.ipynb
– sample_image_to_3d.ipynb
– encode_model.ipynb
– Python API entry points via `shap_e/diffusion/sample.py` and utility scripts.
B. Conditioning & Sampling
– Text/Image Encoder (uses pretrained CLIP, multiview render + point cloud encoder in `shap_e/models/transmitter/`)
– Diffusion Sampler (`shap_e/diffusion/gaussian_diffusion.py`, `k_diffusion.py`)
C. 3D Implicit Function Generation
– Latent‐diffusion model (`shap_e/models/generation/latent_diffusion.py`)
– Perceiver/Transformer/MLP cores (`perceiver.py`, `transformer.py`, `pooled_mlp.py`)
– Implicit representation classes (`shap_e/models/volume.py`, `nerf`, `stf`, `nerstf` modules)
D. Rendering & Output
– In-Python renderer: PyTorch3D helpers (`pytorch3d_util.py`), mesh/point_cloud routines
– Blender-based renderer (`shap_e/rendering/blender/render.py`, `blender_script.py`)
– Output artifacts: `.gif`, `.obj` + `.mtl` in `samples/`

##### Relationships & Data Flow
- Within Sampling: text/image → encoder → latent noise scheduling → diffusion steps → latent code
- Latent code → “3D Implicit Function Generation” → volume or neural field
- Implicit field → “Rendering & Output” → mesh extraction / raycasting (`rendering/raycast/`) → final rendered files
- External calls: model download via Hugging Face API (`shap_e/models/download.py`), Blender executable (env `BLENDER_PATH`)

##### Important Patterns & Principles
– Modular separation: diffusion vs. model vs. rendering modules
– Abstract “renderer” interface to support both CPU/PyTorch3D and Blender back ends
– Clear I/O utility layer (`shap_e/util/io.py`, `image_util.py`, `data_util.py`)

##### Essential Elements Checklist
[ ] External dependencies: Hugging Face model hub, Blender executable

## Diagram (Mermaid)

```mermaid
graph TD
  %% Client Layer
  subgraph "Client Layer"
    ST3D["sample_text_to_3d.ipynb"]:::util
    SI3D["sample_image_to_3d.ipynb"]:::util
    EM["encode_model.ipynb"]:::util
    SAMPLE["sample.py"]:::util
  end

  %% Conditioning & Sampling
  subgraph "Conditioning & Sampling"
    subgraph "Text/Image Encoders" 
      BASE["base.py"]:::ml
      MV["multiview_encoder.py"]:::ml
      PC["pc_encoder.py"]:::ml
      CH["channels_encoder.py"]:::ml
      BN["bottleneck.py"]:::ml
      PP["params_proj.py"]:::ml
    end
    subgraph "Diffusion Sampler"
      GD["gaussian_diffusion.py"]:::ml
      KD["k_diffusion.py"]:::ml
    end
    HF["Hugging Face Model Hub"]:::external
  end

  %% 3D Implicit Function Generation
  subgraph "3D Implicit Function Generation"
    subgraph "Latent Diffusion"
      LD["latent_diffusion.py"]:::ml
      PCV["perceiver.py"]:::ml
      TRF["transformer.py"]:::ml
      MLP["pooled_mlp.py"]:::ml
      CLIPG["pretrained_clip.py"]:::ml
    end
    subgraph "Implicit Representations"
      VOL["volume.py"]:::ml
      subgraph "NeRF"
        NMF["nerf/model.py"]:::ml
        NMR["nerf/ray.py"]:::ml
        NR["nerf/renderer.py"]:::ml
      end
      subgraph "STF"
        STF1["stf/base.py"]:::ml
        STF2["stf/mlp.py"]:::ml
        STF3["stf/renderer.py"]:::ml
      end
      subgraph "NeRSTF"
        NRM["nerstf/mlp.py"]:::ml
        NRR["nerstf/renderer.py"]:::ml
      end
    end
  end

  %% Rendering & Output
  subgraph "Rendering & Output"
    subgraph "PyTorch3D & Mesh/PointCloud"
      P3D["pytorch3d_util.py"]:::rendering
      MESH["mesh.py"]:::rendering
      TMESH["torch_mesh.py"]:::rendering
      PCLOUD["point_cloud.py"]:::rendering
      PLY["ply_util.py"]:::rendering
      MC["mc.py"]:::rendering
      MCT["_mc_table.py"]:::rendering
    end
    subgraph "Raycasting"
      RC["cast.py"]:::rendering
      RR["render.py"]:::rendering
      RU["_utils.py"]:::rendering
      RT["types.py"]:::rendering
    end
    subgraph "Blender Backend"
      BR["render.py"]:::rendering
      BS["blender_script.py"]:::rendering
      BC["constants.py"]:::rendering
      BV["view_data.py"]:::rendering
      BLB["Blender Binary"]:::cloud
    end
    SAMPLES["samples/"]:::external
  end

  %% Utilities
  subgraph "Utilities (I/O, Data & Image Helpers)"
    IO["io.py"]:::util
    IMG["image_util.py"]:::util
    DATA["data_util.py"]:::util
    NB["notebooks.py"]:::util
    COL["collections.py"]:::util
  end

  %% Data Flows
  ST3D -->|"invoke"| BASE
  SI3D --> BASE
  EM --> BASE
  SAMPLE --> GD
  SAMPLE --> KD
  SAMPLE ---|uses| IO

  BASE -->|encodes| LD
  MV --> LD
  PC --> LD
  CH --> LD
  BN --> LD
  PP --> LD

  GD --> LD
  KD --> LD

  LD --> VOL
  LD --> NMF
  LD --> STF1

  VOL --> P3D
  NMR --> RC
  STF3 --> RR
  NRR --> RR

  RC -->|raycast| RR
  RR --> SAMPLES
  P3D --> MESH
  MESH --> SAMPLES
  TMESH --> SAMPLES
  PCLOUD --> SAMPLES
  PLY --> SAMPLES
  MC --> SAMPLES

  BR -.->|CLI| BLB
  BS -.-> BLB
  BR --> SAMPLES

  HF --> GD
  HF --> BASE

  %% Utilities connections
  IO --> GD
  IMG --> BASE
  DATA --> KD
  NB --> ST3D
  COL --> MV

  %% Click Events
  click ST3D "https://github.com/openai/shap-e/blob/main/shap_e/examples/sample_text_to_3d.ipynb"
  click SI3D "https://github.com/openai/shap-e/blob/main/shap_e/examples/sample_image_to_3d.ipynb"
  click EM "https://github.com/openai/shap-e/blob/main/shap_e/examples/encode_model.ipynb"
  click SAMPLE "https://github.com/openai/shap-e/blob/main/shap_e/diffusion/sample.py"
  click GD "https://github.com/openai/shap-e/blob/main/shap_e/diffusion/gaussian_diffusion.py"
  click KD "https://github.com/openai/shap-e/blob/main/shap_e/diffusion/k_diffusion.py"
  click BASE "https://github.com/openai/shap-e/blob/main/shap_e/models/transmitter/base.py"
  click MV "https://github.com/openai/shap-e/blob/main/shap_e/models/transmitter/multiview_encoder.py"
  click PC "https://github.com/openai/shap-e/blob/main/shap_e/models/transmitter/pc_encoder.py"
  click CH "https://github.com/openai/shap-e/blob/main/shap_e/models/transmitter/channels_encoder.py"
  click BN "https://github.com/openai/shap-e/blob/main/shap_e/models/transmitter/bottleneck.py"
  click PP "https://github.com/openai/shap-e/blob/main/shap_e/models/transmitter/params_proj.py"
  click LD "https://github.com/openai/shap-e/blob/main/shap_e/models/generation/latent_diffusion.py"
  click PCV "https://github.com/openai/shap-e/blob/main/shap_e/models/generation/perceiver.py"
  click TRF "https://github.com/openai/shap-e/blob/main/shap_e/models/generation/transformer.py"
  click MLP "https://github.com/openai/shap-e/blob/main/shap_e/models/generation/pooled_mlp.py"
  click CLIPG "https://github.com/openai/shap-e/blob/main/shap_e/models/generation/pretrained_clip.py"
  click VOL "https://github.com/openai/shap-e/blob/main/shap_e/models/volume.py"
  click NMF "https://github.com/openai/shap-e/blob/main/shap_e/models/nerf/model.py"
  click NMR "https://github.com/openai/shap-e/blob/main/shap_e/models/nerf/ray.py"
  click NR "https://github.com/openai/shap-e/blob/main/shap_e/models/nerf/renderer.py"
  click STF1 "https://github.com/openai/shap-e/blob/main/shap_e/models/stf/base.py"
  click STF2 "https://github.com/openai/shap-e/blob/main/shap_e/models/stf/mlp.py"
  click STF3 "https://github.com/openai/shap-e/blob/main/shap_e/models/stf/renderer.py"
  click NRM "https://github.com/openai/shap-e/blob/main/shap_e/models/nerstf/mlp.py"
  click NRR "https://github.com/openai/shap-e/blob/main/shap_e/models/nerstf/renderer.py"
  click P3D "https://github.com/openai/shap-e/blob/main/shap_e/rendering/pytorch3d_util.py"
  click MESH "https://github.com/openai/shap-e/blob/main/shap_e/rendering/mesh.py"
  click TMESH "https://github.com/openai/shap-e/blob/main/shap_e/rendering/torch_mesh.py"
  click PCLOUD "https://github.com/openai/shap-e/blob/main/shap_e/rendering/point_cloud.py"
  click PLY "https://github.com/openai/shap-e/blob/main/shap_e/rendering/ply_util.py"
  click MC "https://github.com/openai/shap-e/blob/main/shap_e/rendering/mc.py"
  click MCT "https://github.com/openai/shap-e/blob/main/shap_e/rendering/_mc_table.py"
  click RC "https://github.com/openai/shap-e/blob/main/shap_e/rendering/raycast/cast.py"
  click RR "https://github.com/openai/shap-e/blob/main/shap_e/rendering/raycast/render.py"
  click RU "https://github.com/openai/shap-e/blob/main/shap_e/rendering/raycast/_utils.py"
  click RT "https://github.com/openai/shap-e/blob/main/shap_e/rendering/raycast/types.py"
  click BR "https://github.com/openai/shap-e/blob/main/shap_e/rendering/blender/render.py"
  click BS "https://github.com/openai/shap-e/blob/main/shap_e/rendering/blender/blender_script.py"
  click BC "https://github.com/openai/shap-e/blob/main/shap_e/rendering/blender/constants.py"
  click BV "https://github.com/openai/shap-e/blob/main/shap_e/rendering/blender/view_data.py"
  click IO "https://github.com/openai/shap-e/blob/main/shap_e/util/io.py"
  click IMG "https://github.com/openai/shap-e/blob/main/shap_e/util/image_util.py"
  click DATA "https://github.com/openai/shap-e/blob/main/shap_e/util/data_util.py"
  click NB "https://github.com/openai/shap-e/blob/main/shap_e/util/notebooks.py"
  click COL "https://github.com/openai/shap-e/blob/main/shap_e/util/collections.py"

  %% Styles
  classDef ml fill:#ADD8E6,stroke:#000,stroke-width:1px
  classDef rendering fill:#90EE90,stroke:#000,stroke-width:1px
  classDef util fill:#D3D3D3,stroke:#000,stroke-width:1px
  classDef external fill:#FFA500,stroke:#000,stroke-width:1px
  classDef cloud fill:#F08080,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5
```
