# jpmorganchase/inference-server architecture

> Architecture diagram of the GitHub repository jpmorganchase/inference-server, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jpmorganchase/inference-server
- Repository: https://github.com/jpmorganchase/inference-server

## Overview

##### Project Type and Purpose
- This is a Python SDK/CLI library—**inference-server**—whose sole responsibility is to package, deploy, and invoke ML models on Amazon SageMaker.
- It supports two deployment modes: **Real-Time Inference** and **Batch Transform**, via a plugin interface that allows custom implementations or the provided AWS-backed default.

##### File Structure
- Top-level “src/inference_server” contains the library code.
- `_plugin.py` defines the abstract plugin interface.
- `default_plugin.py` implements it using boto3 to call SageMaker.
- `testing.py` provides an in-process “mock” plugin for local unit and integration tests.
- “tests/” holds a suite that exercises both the core API and the testing plugin.
- “docs/” and “README.md” document usage, extension points, and deployment guides.
- CI/CD manifests under “.github/workflows” automate linting, testing, security scanning, and packaging.

##### Extract Insights from README
- Confirms it’s a drop-in pip/conda package.
- Directs users to commands (e.g. `inference-server deploy …`) and to ReadTheDocs for deeper coverage.
- No built-in UI or front end—interaction is purely code/CLI driven.

##### Main Components
a. **Developer Environment**
- Local machine or build server.
- Contains user’s ML model (Dockerfile or serialized artifact).
- Has the `inference-server` package installed (editable via tox/dev).
b. **inference-server Core**
- **CLI/SDK Layer:** entry point for commands (`deploy`, `batch-transform`, `invoke`).
- **Plugin Manager:** loads either the `default_plugin` or a custom plugin.
- **API Abstractions:** real-time vs batch interfaces.
c. **Default Plugin (AWS SageMaker)**
- Uses boto3 under the hood.
- Calls SageMaker CreateEndpointConfig, CreateEndpoint, InvokeEndpoint (real-time).
- Calls CreateTransformJob, DescribeTransformJob (batch).
d. **Testing Plugin**
- Local server stub that mimics AWS endpoints for offline testing.
- Used by `tests/test_inference_server.py`.
e. **AWS SageMaker** (external service)
- Real-Time Endpoint
- Batch Transform Job
f. **CI/CD Pipelines**
- GitHub Actions workflows for CodeQL, unit tests, packaging, release.
g. **Documentation Site**
- ReadTheDocs (built from docs/*.rst via Sphinx).

##### Relationships and Data Flows
- Developer issues CLI command → CLI/SDK layer → Plugin Manager
- Plugin Manager → Default Plugin → AWS SDK calls → AWS SageMaker
- For local tests: Plugin Manager → Testing Plugin → Local stub
- CI pipelines trigger tests and release packaging based on changes

##### Architectural Patterns & Principles
- **Plugin Pattern:** clear extension point for new deployment back-ends.
- **Separation of Concerns:** distinct layers for CLI, core API, plugin implementations, and tests.
- **Infrastructure as Code:** Sphinx docs, tox, GitHub Actions for reproducible builds & tests.

##### Essential Details
- Configuration inputs (Docker image URI, model parameters).
- How the plugin manager dynamically loads entry points (via `_plugin.py`).
- AWS calls with the boto3 client.
- Denote the documentation build and CI pipelines as separate but related flows.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Developer Environment
    Developer["Developer Environment\n(Local Machine / Build Server)"]:::external
    Developer -->|issue CLI command| CLI["CLI/SDK Layer\n(deploy, batch-transform, invoke)"]:::internal

    %% Core Layer
    CLI -->|calls| PM["Plugin Manager\n(_plugin.py)"]:::internal

    subgraph "Plugin Implementations"
        direction TB
        Default["Default Plugin (AWS SageMaker)\n(default_plugin.py)"]:::internal
        Testing["Testing Plugin (Local Stub)\n(testing.py)"]:::testing
    end

    PM -->|loads| Default
    PM -->|loads| Testing

    %% AWS SageMaker External Service
    Default -->|boto3: CreateEndpointConfig,\nCreateEndpoint, InvokeEndpoint,\nCreateTransformJob, DescribeTransformJob| AWS["AWS SageMaker\n(Real-Time Endpoint & Batch Transform)"]:::external

    Testing -->|mimics AWS endpoints| Stub["Local Stub\n(for offline testing)"]:::testing

    %% Core Configuration
    CLI --> Config["Core Config & Typing\n(pyproject.toml, src/inference_server/py.typed)"]:::internal

    %% Tests
    Developer -->|runs tests| Tests["Unit & Integration Tests\ntests/test_inference_server.py"]:::testing
    Tests --> Testing

    %% Documentation & CI/CD
    subgraph "Documentation & CI/CD"
        direction TB
        Docs["Sphinx Docs\n(docs/, docs/conf.py)"]:::docsci
        Rtd["ReadTheDocs Config\n(.readthedocs.yaml)"]:::docsci
        CI1["CodeQL Workflow\n(.github/workflows/codeql.yml)"]:::docsci
        CI2["Test & Packaging Workflow\n(.github/workflows/test-package.yml)"]:::docsci
        CI3["Release & Publish Workflow\n(.github/workflows/release-package.yml)"]:::docsci
        Lint["Linting & Hooks\n(.pre-commit-config.yaml, .flake8, tox.ini)"]:::docsci
    end

    CI1 --> Tests
    CI2 --> Tests
    CI2 --> Config
    CI3 --> Config

    %% Click Events
    click CLI "https://github.com/jpmorganchase/inference-server/blob/main/src/inference_server/__init__.py"
    click PM "https://github.com/jpmorganchase/inference-server/blob/main/src/inference_server/_plugin.py"
    click Default "https://github.com/jpmorganchase/inference-server/blob/main/src/inference_server/default_plugin.py"
    click Testing "https://github.com/jpmorganchase/inference-server/blob/main/src/inference_server/testing.py"
    click Config "https://github.com/jpmorganchase/inference-server/blob/main/pyproject.toml"
    click Config "https://github.com/jpmorganchase/inference-server/blob/main/src/inference_server/py.typed"
    click Tests "https://github.com/jpmorganchase/inference-server/blob/main/tests/test_inference_server.py"
    click Docs "https://github.com/jpmorganchase/inference-server/tree/main/docs/"
    click Docs "https://github.com/jpmorganchase/inference-server/blob/main/docs/conf.py"
    click Rtd "https://github.com/jpmorganchase/inference-server/blob/main/.readthedocs.yaml"
    click CI1 "https://github.com/jpmorganchase/inference-server/blob/main/.github/workflows/codeql.yml"
    click CI2 "https://github.com/jpmorganchase/inference-server/blob/main/.github/workflows/test-package.yml"
    click CI3 "https://github.com/jpmorganchase/inference-server/blob/main/.github/workflows/release-package.yml"
    click Lint "https://github.com/jpmorganchase/inference-server/blob/main/.pre-commit-config.yaml"
    click Lint "https://github.com/jpmorganchase/inference-server/blob/main/.flake8"
    click Lint "https://github.com/jpmorganchase/inference-server/blob/main/tox.ini"

    %% Styles
    classDef internal fill:#D0E8FF,stroke:#0366d6,color:#0366d6;
    classDef external fill:#FFD2A6,stroke:#d96604,color:#b33a00;
    classDef testing fill:#DFFFD6,stroke:#2e7d32,color:#2e7d32;
    classDef docsci fill:#F0F0F0,stroke:#666666,color:#333333;
```
