# apple/coreai-models architecture

> Architecture diagram of the GitHub repository apple/coreai-models, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/coreai-models
- Repository: https://github.com/apple/coreai-models

## Overview

Core AI Models combines Python model-authoring and export tools with Swift on-device runtime libraries and command-line runners. Its principal workflow is selecting a supported model, exporting a Core AI asset bundle and resources, then loading them in an app or runner for inference and task-specific postprocessing. Major branches include language and vision inference, image/video diffusion, speech recognition, detection and segmentation, plus agent skills. Edges reflect README-documented workflows and sampled orchestration; unsampled module wiring is intentionally left out.

Main entry points: [`swift/Sources/CoreAILanguageModels/LanguageModel/CoreAILanguageModel.swift`](https://github.com/apple/coreai-models/blob/main/swift/Sources/CoreAILanguageModels/LanguageModel/CoreAILanguageModel.swift) (Language model API), [`python/src/coreai_models/model_registry.py`](https://github.com/apple/coreai-models/blob/main/python/src/coreai_models/model_registry.py) (Model registry), [`python/src/coreai_models/export/pipeline.py`](https://github.com/apple/coreai-models/blob/main/python/src/coreai_models/export/pipeline.py) (Model export), [`python/src/coreai_models/primitives`](https://github.com/apple/coreai-models/tree/main/python/src/coreai_models/primitives) (Python primitives)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_export["Authoring and export"]
  node_catalog["Model recipes"]
  node_registry["Model registry<br/>[model_registry.py]"]
  node_primitives["Python primitives"]
  node_model_export["Model export<br/>[pipeline.py]"]
  node_diffusion_export["Diffusion export<br/>[pipeline.py]"]
  node_seg_export["Segmentation export<br/>[video_pipeline.py]"]
end

subgraph group_language["Language runtime"]
  node_llm["Language model API"]
  node_engine["Inference engines"]
  node_decode["Token decoding"]
  node_server["LLM server"]
end

subgraph group_vision["Vision runtime"]
  node_image_seg["Image segmentation"]
  node_video_seg["Video segmentation"]
  node_detector["Object detection"]
end

subgraph group_media["Media runtime"]
  node_diffusion["Image diffusion"]
  node_video_diffusion["Video diffusion"]
  node_speech["Speech recognition"]
end

subgraph group_skills["Agent skills"]
  node_skills["Agent skills"]
end

node_author(("Model author"))
node_asset["Core AI bundles"]
node_app(("App or runner"))
node_result["Inference results"]

node_author -->|"selects model"| node_registry
node_catalog -->|"lists supported"| node_registry
node_registry -->|"resolves model"| node_model_export
node_author -->|"authors with"| node_primitives
node_primitives -->|"supports export"| node_model_export
node_author -->|"exports pipeline"| node_diffusion_export
node_author -->|"exports segmentation"| node_seg_export
node_model_export -->|"writes model"| node_asset
node_diffusion_export -->|"writes pipeline"| node_asset
node_seg_export -->|"writes bundle"| node_asset
node_app -->|"requests generation"| node_llm
node_llm -->|"runs inference"| node_engine
node_engine -->|"generates tokens"| node_decode
node_decode -->|"returns output"| node_llm
node_app -->|"sends requests"| node_server
node_server -->|"dispatches completion"| node_llm
node_app -->|"submits image"| node_image_seg
node_image_seg -->|"returns masks"| node_result
node_app -->|"submits video"| node_video_seg
node_video_seg -->|"returns tracks"| node_result
node_app -->|"submits image"| node_detector
node_detector -->|"returns detections"| node_result
node_app -->|"requests image"| node_diffusion
node_diffusion -->|"returns image"| node_result
node_app -->|"requests video"| node_video_diffusion
node_video_diffusion -->|"returns frames"| node_result
node_app -->|"submits audio"| node_speech
node_speech -->|"returns transcript"| node_result
node_author -.->|"uses guidance"| node_skills

click node_catalog "https://github.com/apple/coreai-models/tree/main/models"
click node_registry "https://github.com/apple/coreai-models/blob/main/python/src/coreai_models/model_registry.py"
click node_primitives "https://github.com/apple/coreai-models/tree/main/python/src/coreai_models/primitives"
click node_model_export "https://github.com/apple/coreai-models/blob/main/python/src/coreai_models/export/pipeline.py"
click node_diffusion_export "https://github.com/apple/coreai-models/blob/main/python/src/coreai_models/diffusion/pipeline.py"
click node_seg_export "https://github.com/apple/coreai-models/blob/main/python/src/coreai_models/segmentation/video_pipeline.py"
click node_llm "https://github.com/apple/coreai-models/blob/main/swift/Sources/CoreAILanguageModels/LanguageModel/CoreAILanguageModel.swift"
click node_engine "https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAILanguageModels/InferenceEngines"
click node_decode "https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAILanguageModels/DecodingStrategies"
click node_server "https://github.com/apple/coreai-models/tree/main/swift/Sources/Tools/llm-server"
click node_image_seg "https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAIImageSegmenter"
click node_video_seg "https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAIVideoSegmenter"
click node_detector "https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAIObjectDetector"
click node_diffusion "https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAIDiffusionPipeline/Pipelines"
click node_video_diffusion "https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAIVideoDiffusionPipeline/Pipelines"
click node_speech "https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAISpeech"
click node_skills "https://github.com/apple/coreai-models/tree/main/skills"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_catalog,node_registry,node_primitives,node_model_export,node_diffusion_export,node_seg_export toneBlue
class node_llm,node_engine,node_decode,node_server toneAmber
class node_image_seg,node_video_seg,node_detector toneMint
class node_diffusion,node_video_diffusion,node_speech toneRose
class node_skills,node_author,node_asset,node_app,node_result toneIndigo
```

## Components

### Authoring and export

- [Model recipes](https://github.com/apple/coreai-models/tree/main/models): `models`
- [Model registry](https://github.com/apple/coreai-models/blob/main/python/src/coreai_models/model_registry.py): `python/src/coreai_models/model_registry.py`
- [Python primitives](https://github.com/apple/coreai-models/tree/main/python/src/coreai_models/primitives): `python/src/coreai_models/primitives`
- [Model export](https://github.com/apple/coreai-models/blob/main/python/src/coreai_models/export/pipeline.py): `python/src/coreai_models/export/pipeline.py`
- [Diffusion export](https://github.com/apple/coreai-models/blob/main/python/src/coreai_models/diffusion/pipeline.py): `python/src/coreai_models/diffusion/pipeline.py`
- [Segmentation export](https://github.com/apple/coreai-models/blob/main/python/src/coreai_models/segmentation/video_pipeline.py): `python/src/coreai_models/segmentation/video_pipeline.py`

### Language runtime

- [Language model API](https://github.com/apple/coreai-models/blob/main/swift/Sources/CoreAILanguageModels/LanguageModel/CoreAILanguageModel.swift): `swift/Sources/CoreAILanguageModels/LanguageModel/CoreAILanguageModel.swift`
- [Inference engines](https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAILanguageModels/InferenceEngines): `swift/Sources/CoreAILanguageModels/InferenceEngines`
- [Token decoding](https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAILanguageModels/DecodingStrategies): `swift/Sources/CoreAILanguageModels/DecodingStrategies`
- [LLM server](https://github.com/apple/coreai-models/tree/main/swift/Sources/Tools/llm-server): `swift/Sources/Tools/llm-server`

### Vision runtime

- [Image segmentation](https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAIImageSegmenter): `swift/Sources/CoreAIImageSegmenter`
- [Video segmentation](https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAIVideoSegmenter): `swift/Sources/CoreAIVideoSegmenter`
- [Object detection](https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAIObjectDetector): `swift/Sources/CoreAIObjectDetector`

### Media runtime

- [Image diffusion](https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAIDiffusionPipeline/Pipelines): `swift/Sources/CoreAIDiffusionPipeline/Pipelines`
- [Video diffusion](https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAIVideoDiffusionPipeline/Pipelines): `swift/Sources/CoreAIVideoDiffusionPipeline/Pipelines`
- [Speech recognition](https://github.com/apple/coreai-models/tree/main/swift/Sources/CoreAISpeech): `swift/Sources/CoreAISpeech`

### Agent skills

- [Agent skills](https://github.com/apple/coreai-models/tree/main/skills): `skills`

### Other components

- **Model author**
- **Core AI bundles**
- **App or runner**
- **Inference results**

## Connections

- Model author → Model registry: selects model
- Model recipes → Model registry: lists supported
- Model registry → Model export: resolves model
- Model author → Python primitives: authors with
- Python primitives → Model export: supports export
- Model author → Diffusion export: exports pipeline
- Model author → Segmentation export: exports segmentation
- Model export → Core AI bundles: writes model
- Diffusion export → Core AI bundles: writes pipeline
- Segmentation export → Core AI bundles: writes bundle
- App or runner → Language model API: requests generation
- Language model API → Inference engines: runs inference
- Inference engines → Token decoding: generates tokens
- Token decoding → Language model API: returns output
- App or runner → LLM server: sends requests
- LLM server → Language model API: dispatches completion
- App or runner → Image segmentation: submits image
- Image segmentation → Inference results: returns masks
- App or runner → Video segmentation: submits video
- Video segmentation → Inference results: returns tracks
- App or runner → Object detection: submits image
- Object detection → Inference results: returns detections
- App or runner → Image diffusion: requests image
- Image diffusion → Inference results: returns image
- App or runner → Video diffusion: requests video
- Video diffusion → Inference results: returns frames
- App or runner → Speech recognition: submits audio
- Speech recognition → Inference results: returns transcript
- Model author → Agent skills: uses guidance