# apple/ml-stable-diffusion architecture

> Architecture diagram of the GitHub repository apple/ml-stable-diffusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/ml-stable-diffusion
- Repository: https://github.com/apple/ml-stable-diffusion

## Overview

##### Main Components
- “PyTorch Model Source” – This represents the original Stable Diffusion models (for example, from Hugging Face) used as inputs.
- “Python Conversion & Optimization” – Encompasses modules in the python_coreml_stable_diffusion package with conversion scripts (torch2coreml), activation_quantization, mixed_bit_compression_pre_analysis, mixed_bit_compression_apply, controlnet handling, and pipeline modules. This component is responsible for:
– Converting PyTorch checkpoint files to Core ML (.mlpackage or compiled .mlmodelc) assets.
– Performing weight compression (quantization to 6, 4, or even lower bits, via both post-training palettization and advanced recipes for MBP).
– Running activation quantization analysis and creating quantized models.
- “Core ML Model Assets & Resources” – The output of the conversion process (including the Unet, VAE Decoder/Encoder, Text Encoder, Safety Checker, tokenizer assets, and optional refiner and ControlNet models). These are bundled either for the Swift CLI or Python-based inference.
- “Swift Inference / Integration” – Implemented in the Swift package (StableDiffusion) and Swift CLI. This component loads the previously converted Core ML models (with the necessary tokenization, resource handling, and pipeline logic) and invokes image generation. It includes a subcomponent for the tokenizer (in the swift/StableDiffusion/tokenizer directory) and test modules (in swift/StableDiffusionTests).
- “Python Inference Pipeline” – A separate pathway (located in tests and the pipeline module) that allows for image generation on macOS via a diffusers-like interface.
- “Assets & Benchmarks” – The assets folder (containing images, benchmark visuals, ControlNet sample visuals, and MBP result images) serves as a visual aid when evaluating performance and image quality.

##### Relationships & Interactions
- The conversion process that outputs standard Core ML models (with default float16) and the alternative branch that performs weight/activation quantization and advanced MBP to reduce model size.
- A note that the conversion process leverages coremltools and depends on external configurations (e.g., attention-implementation options, compute unit choices).
- Optionally, include feedback loops from “Testing & Benchmarking” environments (tests folder, and benchmark results as seen in the README) to guide performance tuning in the conversion and inference phases.

##### Architectural Patterns & Design Principles
- Modularization: Both modules (Python and Swift) are composed of several independent submodules (for example, pipeline, tokenizer, controlnet, quantization) that are integrated via clearly defined resource bundles.

##### Additional Elements
- Clear labeling of components (e.g., “Model Conversion & Optimization”, “Core ML Models”, “Inference – Swift CLI/Library”, “Inference – Python Pipeline”).
- Conversion scripts (Python-based) – perhaps blue rectangles.
- External resource downloads (Hugging Face models, assets) – possibly a different symbol.
- Annotations for special architectural patterns like “just-in-time decompression”, “chunked Unet processing for mobile deployment”, and “advanced quantization techniques” to capture the nuances in the model conversion process.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["PyTorch Model Source"]:::external
    B["Python Conversion & Optimization\n(torch2coreml, activation_quantization, mixed_bit_compression, ControlNet)"]:::conversion
    C["Core ML Model Assets & Resources"]:::asset

    subgraph "Swift Inference / Integration"
        D1["Swift Package"]:::swift
        D2["Tokenizer"]:::swift
        D3["CLI"]:::swift
        D4["Test Modules"]:::swift
    end

    subgraph "Python Inference Pipeline"
        E1["Diffusers-like Interface"]:::python
        E2["Pipeline Module"]:::python
    end

    A -->|"downloads+transforms"| B
    B -->|"outputs"| C
    C -->|"loaded_by"| D1
    C -->|"loaded_by"| D2
    C -->|"loaded_by"| D3
    C -->|"loaded_by"| D4
    C -->|"loaded_by"| E1
    C -->|"loaded_by"| E2
    D1 -->|"feedback"| B
    E1 -->|"feedback"| B

    click B "https://github.com/apple/ml-stable-diffusion/tree/main/python_coreml_stable_diffusion"
    click C "https://github.com/apple/ml-stable-diffusion/tree/main/assets"
    click D1 "https://github.com/apple/ml-stable-diffusion/tree/main/swift/StableDiffusion"
    click D3 "https://github.com/apple/ml-stable-diffusion/tree/main/swift/StableDiffusionCLI"
    click E1 "https://github.com/apple/ml-stable-diffusion/tree/main/tests"
    click E2 "https://github.com/apple/ml-stable-diffusion/blob/main/python_coreml_stable_diffusion/pipeline.py"

    classDef external fill:#F4A460,stroke:#000,stroke-width:2px;
    classDef conversion fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef asset fill:#D3D3D3,stroke:#000,stroke-width:2px;
    classDef swift fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef python fill:#FFDAB9,stroke:#000,stroke-width:2px;
```
