# microsoft/foundry-local architecture

> Architecture diagram of the GitHub repository microsoft/foundry-local, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/foundry-local
- Repository: https://github.com/microsoft/foundry-local

## Overview

Foundry Local provides embedded, on-device model management and inference through language SDKs, with an optional local OpenAI-compatible server. The principal workflow is to initialize a manager, find and acquire a model, load a hardware-suitable variant, then make chat, embedding, or audio requests. The graph emphasizes the v2 native runtime and bindings; the README documents additional capabilities, but sampled source does not establish every internal connection. Dashed edges mark documented or optional paths whose implementation wiring was not shown.

Main entry points: [`sdk_v2/cpp/src/manager.cc`](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/manager.cc) (Runtime manager), [`sdk_v2/cpp/src/contracts`](https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/contracts) (OpenAI API clients), [`sdk_v2/cpp/src/inferencing/session/session_manager.cc`](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/inferencing/session/session_manager.cc) (Inference sessions), [`sdk_v2/cpp/src/service/embeddings_handler.cc`](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/service/embeddings_handler.cc) (Embedding route)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sdk["SDKs and API"]
  node_bindings["Language bindings"]
  node_manager["Runtime manager<br/>[manager.cc]"]
  node_openai["OpenAI API clients"]
end

subgraph group_lifecycle["Model lifecycle"]
  node_catalog["Model catalog"]
  node_catalog_service["Catalog service"]
  node_download["Model downloads"]
  node_cache[("Local model cache<br/>[catalog_cache.cc]")]
  node_hardware["Hardware detection"]
  node_model_load["Model loading"]
end

subgraph group_runtime["Inference runtime"]
  node_session["Inference sessions<br/>[session_manager.cc]"]
  node_chat["Chat generation"]
  node_embeddings["Embeddings"]
  node_audio["Audio inference"]
  node_responses["Responses API"]
end

subgraph group_server["Local server"]
  node_local_server["Local web server<br/>[web_service.cc]"]
  node_embedding_route["Embedding route"]
  node_other_routes["Other API routes"]
end

node_developer(("Application developer"))
node_openai_app(("OpenAI-compatible client"))
node_registry[("Model registry")]
node_device{{"Device hardware"}}

node_developer -.->|"uses SDK"| node_bindings
node_bindings -.->|"initializes"| node_manager
node_manager -->|"creates"| node_catalog
node_catalog -.->|"queries catalog"| node_catalog_service
node_manager -->|"manages downloads"| node_download
node_download -.->|"fetches model data"| node_registry
node_download -.->|"stores models"| node_cache
node_manager -->|"detects providers"| node_hardware
node_hardware -.->|"detects capabilities"| node_device
node_manager -->|"loads models"| node_model_load
node_model_load -.->|"selects variant"| node_hardware
node_developer -.->|"selects model"| node_catalog
node_developer -.->|"requests load"| node_model_load
node_openai_app -.->|"sends requests"| node_local_server
node_developer -.->|"creates clients"| node_openai
node_local_server -.->|"routes requests"| node_embedding_route
node_local_server -.->|"routes requests"| node_other_routes
node_embedding_route -->|"resolves model"| node_catalog
node_embedding_route -->|"gets loaded model"| node_model_load
node_embedding_route -->|"processes request"| node_session
node_session -.->|"dispatches inference"| node_embeddings
node_session -.->|"dispatches inference"| node_chat
node_session -.->|"dispatches inference"| node_audio
node_session -.->|"dispatches inference"| node_responses
node_openai -.->|"requests completion"| node_chat
node_openai -.->|"requests vectors"| node_embeddings
node_openai -.->|"requests transcription"| node_audio
node_openai -.->|"uses format"| node_responses

click node_bindings "https://github.com/microsoft/foundry-local/tree/main/sdk_v2"
click node_manager "https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/manager.cc"
click node_openai "https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/contracts"
click node_catalog "https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/catalog"
click node_catalog_service "https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/catalog/azure_catalog_client.cc"
click node_download "https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/download"
click node_cache "https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/catalog/catalog_cache.cc"
click node_hardware "https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/ep_detection"
click node_model_load "https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/inferencing/model_load_manager.cc"
click node_session "https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/inferencing/session/session_manager.cc"
click node_chat "https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.cc"
click node_embeddings "https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.cc"
click node_audio "https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/inferencing/generative/audio"
click node_responses "https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/inferencing/generative/openresponses"
click node_local_server "https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/service/web_service.cc"
click node_embedding_route "https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/service/embeddings_handler.cc"
click node_other_routes "https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/service"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bindings,node_manager,node_openai,node_openai_app toneBlue
class node_catalog,node_catalog_service,node_download,node_cache,node_hardware,node_model_load,node_registry toneAmber
class node_session,node_chat,node_embeddings,node_audio,node_responses toneMint
class node_local_server,node_embedding_route,node_other_routes toneRose
class node_developer,node_device toneIndigo
```

## Components

### SDKs and API

- [Language bindings](https://github.com/microsoft/foundry-local/tree/main/sdk_v2): `sdk_v2`
- [Runtime manager](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/manager.cc): `sdk_v2/cpp/src/manager.cc`
- [OpenAI API clients](https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/contracts): `sdk_v2/cpp/src/contracts`

### Model lifecycle

- [Model catalog](https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/catalog): `sdk_v2/cpp/src/catalog`
- [Catalog service](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/catalog/azure_catalog_client.cc): `sdk_v2/cpp/src/catalog/azure_catalog_client.cc`
- [Model downloads](https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/download): `sdk_v2/cpp/src/download`
- [Local model cache](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/catalog/catalog_cache.cc): `sdk_v2/cpp/src/catalog/catalog_cache.cc`
- [Hardware detection](https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/ep_detection): `sdk_v2/cpp/src/ep_detection`
- [Model loading](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/inferencing/model_load_manager.cc): `sdk_v2/cpp/src/inferencing/model_load_manager.cc`

### Inference runtime

- [Inference sessions](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/inferencing/session/session_manager.cc): `sdk_v2/cpp/src/inferencing/session/session_manager.cc`
- [Chat generation](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.cc): `sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.cc`
- [Embeddings](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.cc): `sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.cc`
- [Audio inference](https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/inferencing/generative/audio): `sdk_v2/cpp/src/inferencing/generative/audio`
- [Responses API](https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/inferencing/generative/openresponses): `sdk_v2/cpp/src/inferencing/generative/openresponses`

### Local server

- [Local web server](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/service/web_service.cc): `sdk_v2/cpp/src/service/web_service.cc`
- [Embedding route](https://github.com/microsoft/foundry-local/blob/main/sdk_v2/cpp/src/service/embeddings_handler.cc): `sdk_v2/cpp/src/service/embeddings_handler.cc`
- [Other API routes](https://github.com/microsoft/foundry-local/tree/main/sdk_v2/cpp/src/service): `sdk_v2/cpp/src/service`

### Other components

- **Application developer**
- **OpenAI-compatible client**
- **Model registry**
- **Device hardware**

## Connections

- Application developer → Language bindings: uses SDK (evidence: `README.md`)
- Language bindings → Runtime manager: initializes (evidence: `sdk_v2/cpp/src/manager.cc`)
- Runtime manager → Model catalog: creates (evidence: `sdk_v2/cpp/src/manager.cc`)
- Model catalog → Catalog service: queries catalog
- Runtime manager → Model downloads: manages downloads (evidence: `sdk_v2/cpp/src/manager.cc`)
- Model downloads → Model registry: fetches model data (evidence: `README.md`)
- Model downloads → Local model cache: stores models (evidence: `README.md`)
- Runtime manager → Hardware detection: detects providers (evidence: `sdk_v2/cpp/src/manager.cc`)
- Hardware detection → Device hardware: detects capabilities (evidence: `README.md`)
- Runtime manager → Model loading: loads models (evidence: `sdk_v2/cpp/src/manager.cc`)
- Model loading → Hardware detection: selects variant (evidence: `README.md`)
- Application developer → Model catalog: selects model (evidence: `README.md`)
- Application developer → Model loading: requests load (evidence: `README.md`)
- OpenAI-compatible client → Local web server: sends requests (evidence: `README.md`)
- Application developer → OpenAI API clients: creates clients (evidence: `README.md`)
- Local web server → Embedding route: routes requests
- Local web server → Other API routes: routes requests (evidence: `README.md`)
- Embedding route → Model catalog: resolves model (evidence: `sdk_v2/cpp/src/service/embeddings_handler.cc`)
- Embedding route → Model loading: gets loaded model (evidence: `sdk_v2/cpp/src/service/embeddings_handler.cc`)
- Embedding route → Inference sessions: processes request (evidence: `sdk_v2/cpp/src/service/embeddings_handler.cc`)
- Inference sessions → Embeddings: dispatches inference
- Inference sessions → Chat generation: dispatches inference
- Inference sessions → Audio inference: dispatches inference
- Inference sessions → Responses API: dispatches inference
- OpenAI API clients → Chat generation: requests completion (evidence: `README.md`)
- OpenAI API clients → Embeddings: requests vectors (evidence: `README.md`)
- OpenAI API clients → Audio inference: requests transcription (evidence: `README.md`)
- OpenAI API clients → Responses API: uses format (evidence: `README.md`)