How tensorflow/serving works
TensorFlow Serving accepts versioned model requests over gRPC or HTTP, resolves a model version, executes inference, and returns results. In parallel, configured storage sources discover model versions and the serving core loads, tracks, and makes those versions available. The README documents batching, model management, multiple versions, and extensible servables; the sampled source confirms filesystem version discovery and server endpoint registration. Some internal wiring is inferred from module roles and the documented architecture because those implementations were not sampled.
Main entry points: prediction_service_impl.cc gRPC services · predict_impl.cc Prediction execution · server_core.cc Server core · http_rest_api_handler.cc HTTP REST API
Components 19
Serving endpoints
- Server
tensorflow_serving/model_servers/server.cc - gRPC services
tensorflow_serving/model_servers/prediction_service_impl.cc - HTTP REST API
tensorflow_serving/model_servers/http_rest_api_handler.cc - Serving API messages
tensorflow_serving/apis/prediction_service.proto - Model management
tensorflow_serving/model_servers/model_service_impl.cc
Model lifecycle
- Server core
tensorflow_serving/model_servers/server_core.cc - Filesystem version source
tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc - Version policies
tensorflow_serving/config/file_system_storage_path_source.proto - Version manager
tensorflow_serving/core/aspired_versions_manager.cc - SavedModel loader
tensorflow_serving/servables/tensorflow/saved_model_bundle_source_adapter.cc
Inference
- TensorFlow servable
tensorflow_serving/servables/tensorflow/servable.cc - Prediction execution
tensorflow_serving/servables/tensorflow/predict_impl.cc - Task inference
tensorflow_serving/servables/tensorflow/multi_inference.cc - Batch scheduler
tensorflow_serving/batching/batching_session.cc
Runtime operations
- Model metadata
tensorflow_serving/servables/tensorflow/get_model_metadata_impl.cc - Model status
tensorflow_serving/model_servers/get_model_status_impl.cc - Request logging
tensorflow_serving/core/request_logger.cc
Other components
- Inference client
- Model storage
Connections 23
- Inference client → gRPC services: sends gRPC
- Inference client → HTTP REST API: sends HTTP
- gRPC services → Serving API messages: uses messages
- Server → gRPC services: registers services
- Server → HTTP REST API: starts endpoint
- Server → Server core: initializes
- Model management → Server core: manages models
- Version policies → Filesystem version source: configures
- Filesystem version source → Model storage: polls versions
- Filesystem version source → Version manager: aspires versions
- Version manager → SavedModel loader: loads versions
- SavedModel loader → TensorFlow servable: creates
- Server core → Version manager: coordinates
- gRPC services → Server core: looks up model
- HTTP REST API → Server core: looks up model
- gRPC services → Prediction execution: dispatches inference
- HTTP REST API → Prediction execution: dispatches inference
- Prediction execution → TensorFlow servable: executes on
- Prediction execution → Task inference: dispatches tasks
- Prediction execution → Batch scheduler: submits batches
- gRPC services → Model metadata: handles metadata
- gRPC services → Model status: handles status
- Prediction execution → Request logging: logs requests