# tensorflow/serving architecture

> Architecture diagram of the GitHub repository tensorflow/serving, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tensorflow/serving
- Repository: https://github.com/tensorflow/serving

## Overview

TensorFlow Serving accepts versioned model requests over gRPC or HTTP, resolves a model version, executes inference, and returns results. In parallel, configured storage sources discover model versions and the serving core loads, tracks, and makes those versions available. The README documents batching, model management, multiple versions, and extensible servables; the sampled source confirms filesystem version discovery and server endpoint registration. Some internal wiring is inferred from module roles and the documented architecture because those implementations were not sampled.

Main entry points: [`tensorflow_serving/model_servers/prediction_service_impl.cc`](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/prediction_service_impl.cc) (gRPC services), [`tensorflow_serving/servables/tensorflow/predict_impl.cc`](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/predict_impl.cc) (Prediction execution), [`tensorflow_serving/model_servers/server_core.cc`](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/server_core.cc) (Server core), [`tensorflow_serving/model_servers/http_rest_api_handler.cc`](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/http_rest_api_handler.cc) (HTTP REST API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving endpoints"]
  node_server["Server<br/>[server.cc]"]
  node_grpc["gRPC services"]
  node_rest["HTTP REST API"]
  node_api["Serving API messages"]
  node_management["Model management"]
end

subgraph group_lifecycle["Model lifecycle"]
  node_core["Server core<br/>[server_core.cc]"]
  node_source["Filesystem version source"]
  node_sourceconfig["Version policies"]
  node_manager["Version manager"]
  node_loader["SavedModel loader"]
end

subgraph group_inference["Inference"]
  node_servable["TensorFlow servable<br/>[servable.cc]"]
  node_predict["Prediction execution<br/>[predict_impl.cc]"]
  node_specialized["Task inference<br/>[multi_inference.cc]"]
  node_batch["Batch scheduler"]
end

subgraph group_operations["Runtime operations"]
  node_metadata["Model metadata"]
  node_status["Model status"]
  node_logging["Request logging<br/>[request_logger.cc]"]
end

node_client(("Inference client"))
node_filesystem[("Model storage")]

node_client -->|"sends gRPC"| node_grpc
node_client -->|"sends HTTP"| node_rest
node_grpc -->|"uses messages"| node_api
node_server -->|"registers services"| node_grpc
node_server -->|"starts endpoint"| node_rest
node_server -->|"initializes"| node_core
node_management -->|"manages models"| node_core
node_sourceconfig -->|"configures"| node_source
node_source -->|"polls versions"| node_filesystem
node_source -->|"aspires versions"| node_manager
node_manager -->|"loads versions"| node_loader
node_loader -->|"creates"| node_servable
node_core -->|"coordinates"| node_manager
node_grpc -->|"looks up model"| node_core
node_rest -->|"looks up model"| node_core
node_grpc -->|"dispatches inference"| node_predict
node_rest -->|"dispatches inference"| node_predict
node_predict -->|"executes on"| node_servable
node_predict -->|"dispatches tasks"| node_specialized
node_predict -.->|"submits batches"| node_batch
node_grpc -->|"handles metadata"| node_metadata
node_grpc -->|"handles status"| node_status
node_predict -.->|"logs requests"| node_logging

click node_server "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/server.cc"
click node_grpc "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/prediction_service_impl.cc"
click node_rest "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/http_rest_api_handler.cc"
click node_api "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_service.proto"
click node_management "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/model_service_impl.cc"
click node_core "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/server_core.cc"
click node_source "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc"
click node_sourceconfig "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/config/file_system_storage_path_source.proto"
click node_manager "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/core/aspired_versions_manager.cc"
click node_loader "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/saved_model_bundle_source_adapter.cc"
click node_servable "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/servable.cc"
click node_predict "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/predict_impl.cc"
click node_specialized "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/multi_inference.cc"
click node_batch "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/batching/batching_session.cc"
click node_metadata "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/get_model_metadata_impl.cc"
click node_status "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/get_model_status_impl.cc"
click node_logging "https://github.com/tensorflow/serving/blob/master/tensorflow_serving/core/request_logger.cc"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_grpc,node_rest,node_api,node_management,node_client toneBlue
class node_core,node_source,node_sourceconfig,node_manager,node_loader,node_filesystem toneAmber
class node_servable,node_predict,node_specialized,node_batch toneMint
class node_metadata,node_status,node_logging toneRose
```

## Components

### Serving endpoints

- [Server](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/server.cc): `tensorflow_serving/model_servers/server.cc`
- [gRPC services](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/prediction_service_impl.cc): `tensorflow_serving/model_servers/prediction_service_impl.cc`
- [HTTP REST API](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/http_rest_api_handler.cc): `tensorflow_serving/model_servers/http_rest_api_handler.cc`
- [Serving API messages](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_service.proto): `tensorflow_serving/apis/prediction_service.proto`
- [Model management](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/model_service_impl.cc): `tensorflow_serving/model_servers/model_service_impl.cc`

### Model lifecycle

- [Server core](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/server_core.cc): `tensorflow_serving/model_servers/server_core.cc`
- [Filesystem version source](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc): `tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc`
- [Version policies](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/config/file_system_storage_path_source.proto): `tensorflow_serving/config/file_system_storage_path_source.proto`
- [Version manager](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/core/aspired_versions_manager.cc): `tensorflow_serving/core/aspired_versions_manager.cc`
- [SavedModel loader](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/saved_model_bundle_source_adapter.cc): `tensorflow_serving/servables/tensorflow/saved_model_bundle_source_adapter.cc`

### Inference

- [TensorFlow servable](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/servable.cc): `tensorflow_serving/servables/tensorflow/servable.cc`
- [Prediction execution](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/predict_impl.cc): `tensorflow_serving/servables/tensorflow/predict_impl.cc`
- [Task inference](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/multi_inference.cc): `tensorflow_serving/servables/tensorflow/multi_inference.cc`
- [Batch scheduler](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/batching/batching_session.cc): `tensorflow_serving/batching/batching_session.cc`

### Runtime operations

- [Model metadata](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/servables/tensorflow/get_model_metadata_impl.cc): `tensorflow_serving/servables/tensorflow/get_model_metadata_impl.cc`
- [Model status](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/get_model_status_impl.cc): `tensorflow_serving/model_servers/get_model_status_impl.cc`
- [Request logging](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/core/request_logger.cc): `tensorflow_serving/core/request_logger.cc`

### Other components

- **Inference client**
- **Model storage**

## Connections

- Inference client → gRPC services: sends gRPC
- Inference client → HTTP REST API: sends HTTP
- gRPC services → Serving API messages: uses messages
- Server → gRPC services: registers services
- Server → HTTP REST API: starts endpoint
- Server → Server core: initializes
- Model management → Server core: manages models
- Version policies → Filesystem version source: configures
- Filesystem version source → Model storage: polls versions
- Filesystem version source → Version manager: aspires versions
- Version manager → SavedModel loader: loads versions
- SavedModel loader → TensorFlow servable: creates
- Server core → Version manager: coordinates
- gRPC services → Server core: looks up model
- HTTP REST API → Server core: looks up model
- gRPC services → Prediction execution: dispatches inference
- HTTP REST API → Prediction execution: dispatches inference
- Prediction execution → TensorFlow servable: executes on
- Prediction execution → Task inference: dispatches tasks
- Prediction execution → Batch scheduler: submits batches
- gRPC services → Model metadata: handles metadata
- gRPC services → Model status: handles status
- Prediction execution → Request logging: logs requests