How microsoft/foundry-local works
Foundry Local provides embedded, on-device model management and inference through language SDKs, with an optional local OpenAI-compatible server. The principal workflow is to initialize a manager, find and acquire a model, load a hardware-suitable variant, then make chat, embedding, or audio requests. The graph emphasizes the v2 native runtime and bindings; the README documents additional capabilities, but sampled source does not establish every internal connection. Dashed edges mark documented or optional paths whose implementation wiring was not shown.
Main entry points: manager.cc Runtime manager · contracts OpenAI API clients · session_manager.cc Inference sessions · embeddings_handler.cc Embedding route
Components 21
SDKs and API
- Language bindings
sdk_v2 - Runtime manager
sdk_v2/cpp/src/manager.cc - OpenAI API clients
sdk_v2/cpp/src/contracts
Model lifecycle
- Model catalog
sdk_v2/cpp/src/catalog - Catalog service
sdk_v2/cpp/src/catalog/azure_catalog_client.cc - Model downloads
sdk_v2/cpp/src/download - Local model cache
sdk_v2/cpp/src/catalog/catalog_cache.cc - Hardware detection
sdk_v2/cpp/src/ep_detection - Model loading
sdk_v2/cpp/src/inferencing/model_load_manager.cc
Inference runtime
- Inference sessions
sdk_v2/cpp/src/inferencing/session/session_manager.cc - Chat generation
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.cc - Embeddings
sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.cc - Audio inference
sdk_v2/cpp/src/inferencing/generative/audio - Responses API
sdk_v2/cpp/src/inferencing/generative/openresponses
Local server
- Local web server
sdk_v2/cpp/src/service/web_service.cc - Embedding route
sdk_v2/cpp/src/service/embeddings_handler.cc - Other API routes
sdk_v2/cpp/src/service
Other components
- Application developer
- OpenAI-compatible client
- Model registry
- Device hardware
Connections 28
- Application developer → Language bindings: uses SDK
README.md - Language bindings → Runtime manager: initializes
sdk_v2/cpp/src/manager.cc - Runtime manager → Model catalog: creates
sdk_v2/cpp/src/manager.cc - Model catalog → Catalog service: queries catalog
- Runtime manager → Model downloads: manages downloads
sdk_v2/cpp/src/manager.cc - Model downloads → Model registry: fetches model data
README.md - Model downloads → Local model cache: stores models
README.md - Runtime manager → Hardware detection: detects providers
sdk_v2/cpp/src/manager.cc - Hardware detection → Device hardware: detects capabilities
README.md - Runtime manager → Model loading: loads models
sdk_v2/cpp/src/manager.cc - Model loading → Hardware detection: selects variant
README.md - Application developer → Model catalog: selects model
README.md - Application developer → Model loading: requests load
README.md - OpenAI-compatible client → Local web server: sends requests
README.md - Application developer → OpenAI API clients: creates clients
README.md - Local web server → Embedding route: routes requests
- Local web server → Other API routes: routes requests
README.md - Embedding route → Model catalog: resolves model
sdk_v2/cpp/src/service/embeddings_handler.cc - Embedding route → Model loading: gets loaded model
sdk_v2/cpp/src/service/embeddings_handler.cc - Embedding route → Inference sessions: processes request
sdk_v2/cpp/src/service/embeddings_handler.cc - Inference sessions → Embeddings: dispatches inference
- Inference sessions → Chat generation: dispatches inference
- Inference sessions → Audio inference: dispatches inference
- Inference sessions → Responses API: dispatches inference
- OpenAI API clients → Chat generation: requests completion
README.md - OpenAI API clients → Embeddings: requests vectors
README.md - OpenAI API clients → Audio inference: requests transcription
README.md - OpenAI API clients → Responses API: uses format
README.md