How jpmorganchase/inference-server works
Project Type and Purpose
• This is a Python SDK/CLI library—inference-server—whose sole responsibility is to package, deploy, and invoke ML models on Amazon SageMaker.
• It supports two deployment modes: Real-Time Inference and Batch Transform, via a plugin interface that allows custom implementations or the provided AWS-backed default.
File Structure
• Top-level “src/inference_server” contains the library code.
• _plugin.py defines the abstract plugin interface.
• default_plugin.py implements it using boto3 to call SageMaker.
• testing.py provides an in-process “mock” plugin for local unit and integration tests.
• “tests/” holds a suite that exercises both the core API and the testing plugin.
• “docs/” and “README.md” document usage, extension points, and deployment guides.
• CI/CD manifests under “.github/workflows” automate linting, testing, security scanning, and packaging.
Extract Insights from README
• Confirms it’s a drop-in pip/conda package.
• Directs users to commands (e.g. inference-server deploy …) and to ReadTheDocs for deeper coverage.
• No built-in UI or front end—interaction is purely code/CLI driven.
Main Components
a. Developer Environment
• Local machine or build server.
• Contains user’s ML model (Dockerfile or serialized artifact).
• Has the inference-server package installed (editable via tox/dev).
b. inference-server Core
• CLI/SDK Layer: entry point for commands (deploy, batch-transform, invoke).
• Plugin Manager: loads either the default_plugin or a custom plugin.
• API Abstractions: real-time vs batch interfaces.
c. Default Plugin (AWS SageMaker)
• Uses boto3 under the hood.
• Calls SageMaker CreateEndpointConfig, CreateEndpoint, InvokeEndpoint (real-time).
• Calls CreateTransformJob, DescribeTransformJob (batch).
d. Testing Plugin
• Local server stub that mimics AWS endpoints for offline testing.
• Used by tests/test_inference_server.py.
e. AWS SageMaker (external service)
• Real-Time Endpoint
• Batch Transform Job
f. CI/CD Pipelines
• GitHub Actions workflows for CodeQL, unit tests, packaging, release.
g. Documentation Site
• ReadTheDocs (built from docs/*.rst via Sphinx).
Relationships and Data Flows
• Developer issues CLI command → CLI/SDK layer → Plugin Manager
• Plugin Manager → Default Plugin → AWS SDK calls → AWS SageMaker
• For local tests: Plugin Manager → Testing Plugin → Local stub
• CI pipelines trigger tests and release packaging based on changes
Architectural Patterns & Principles
• Plugin Pattern: clear extension point for new deployment back-ends.
• Separation of Concerns: distinct layers for CLI, core API, plugin implementations, and tests.
• Infrastructure as Code: Sphinx docs, tox, GitHub Actions for reproducible builds & tests.
Essential Details
• Configuration inputs (Docker image URI, model parameters).
• How the plugin manager dynamically loads entry points (via _plugin.py).
• AWS calls with the boto3 client.
• Denote the documentation build and CI pipelines as separate but related flows.