# tuttlebr/triton-server-demo architecture

> Architecture diagram of the GitHub repository tuttlebr/triton-server-demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tuttlebr/triton-server-demo
- Repository: https://github.com/tuttlebr/triton-server-demo

## Overview

##### Main Components
a. Triton Inference Server: Central to the application, this component is deployed via Docker and managed on Kubernetes. It handles the loading and serving of multiple deep learning models.
b. Model Repository: A structured directory (“model_repository”) that houses various model configurations and executables. Each subdirectory (e.g., densenet-onnx, inception-graphdef) represents a model with its own configuration file and, in some cases, model files stored with Git LFS.
c. Kubernetes Cluster: Includes several nodes (NVIDIA GPU nodes) with the necessary GPU Operator ensuring that GPUs are allocated to the Triton Server appropriately.
d. Load Balancer & Ingress: Traefik (configured via Helm charts under the “tritoninferenceserver” directory) routes incoming requests to the appropriate Triton pods.
f. Supporting Scripts and Tools: Startup and monitoring scripts (00-start.sh, 01-querytriton.sh, 02-perfanalyzer.sh) that automate image building, deployment, server querying, and performance analysis.

##### Relationships and Interactions
- The Docker images are built (using Dockerfile and docker-compose.yaml) and pushed to a repository accessible to Kubernetes nodes.
- The Kubernetes cluster deploys the Triton Inference Server via a Helm chart (located in the tritoninferenceserver directory) that sets up the deployment, Horizontal Pod Autoscaler (HPA), ingress, and persistent volumes.
- The Triton Server pulls the models from the “model_repository” which is locally mounted or accessed as part of the container’s file system.
- External clients submit inference requests to the load balancer (Traefik), which routes them to the Triton Server.
- Monitoring tools (Prometheus and Grafana) collect and display performance metrics from the Triton Metrics Server.

##### Architectural Patterns and Technologies
- Containerized Deployment: Using Docker images and Docker Compose along with Kubernetes for orchestration.
- Microservices-like Approach: Although it is a single server, the separation between inference (Triton), orchestration (Kubernetes), and monitoring (Prometheus/Grafana) reinforces modular design.
- Infrastructure as Code: Helm charts for deploying the Triton Server components and associated configurations.
- External Integration: Load balancing through Traefik and GPU management via NVIDIA GPU Operator.

##### Elements
- Labels for each component: Triton Inference Server, Model Repository, Kubernetes Cluster, Traefik Load Balancer, NVIDIA GPU Operator, Prometheus, and Grafana.
- Containers/Server Components (e.g., Triton) could be depicted as rectangles.
- External/monitoring systems (Traefik, Grafana, Prometheus) should be clearly separated.
- Annotations explaining key scripts that drive the setup and diagnostics, e.g., “00-start.sh” (builds and deploys), “01-querytriton.sh” (status checking), “02-perfanalyzer.sh” (performance metrics).

- Triton Inference Server Pods (with labels indicating the GPU resource management thanks to NVIDIA GPU Operator).
- An attached “Model Repository” volume that is populated with the detailed directories for different models.
- Additionally, set aside a section for “Monitoring & Metrics” indicating Prometheus collecting metrics and Grafana using the “triton-server-dashboard.json.”
- Optionally, include an “Infrastructure Setup” section representing CI/CD pipelines with Docker build scripts and Helm chart deployments.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["External User/Client"]:::external
    A -->|"sends_requests"| B["Traefik Load Balancer"]:::lb
    B -->|"routes_to"| K["Kubernetes Cluster"]:::kube

    subgraph "Kubernetes Cluster"
        direction TB
        T["Triton Inference Server"]:::server
        TConfig["Triton Deployment Config"]:::config
        TConfig --> T
        G["NVIDIA GPU Operator"]:::gpu
        M["Model Repository"]:::repo
        KC["Kubernetes Deployment Config"]:::config
        KC --> T
        %% Model Repository subdirectories
        Mdensenet["densenet-onnx"]:::repo
        Minception["inception-graphdef"]:::repo
        M --- Mdensenet
        M --- Minception
    end

    subgraph "Monitoring & Metrics"
        direction TB
        P["Prometheus"]:::monitor
        Gf["Grafana Dashboard"]:::monitor
        MC["Monitoring Config"]:::config
        MC --> P
        MC --> Gf
    end

    subgraph "CI/CD & Supporting Scripts"
        direction TB
        S["Supporting Scripts & Tools"]:::scripts
    end

    %% Connections between major components
    T -->|"emits_metrics"| P
    T -->|"emits_metrics"| Gf
    S -->|"deploys_updates"| K

    %% Click Events
    click TConfig "https://github.com/tuttlebr/triton-server-demo/blob/main/Dockerfile,docker-compose.yaml,tritoninferenceserver/templates/deployment.yaml"
    click M "https://github.com/tuttlebr/triton-server-demo/tree/main/model_repository"
    click Mdensenet "https://github.com/tuttlebr/triton-server-demo/tree/main/model_repository/densenet-onnx"
    click Minception "https://github.com/tuttlebr/triton-server-demo/tree/main/model_repository/inception-graphdef"
    click KC "https://github.com/tuttlebr/triton-server-demo/tree/main/tritoninferenceserver"
    click B "https://github.com/tuttlebr/triton-server-demo/blob/main/tritoninferenceserver/charts/traefik-10.6.2.tgz,tritoninferenceserver/templates/ingressroute.yaml"
    click MC "https://github.com/tuttlebr/triton-server-demo/blob/main/triton-server-dashboard.json,tritoninferenceserver/charts/prometheus-adapter-3.0.3.tgz"
    click S "https://github.com/tuttlebr/triton-server-demo/blob/main/00-start.sh,01-querytriton.sh,02-perfanalyzer.sh,fetch_models.sh"

    %% Styles
    classDef external fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef lb fill:#66ccff,stroke:#333,stroke-width:2px;
    classDef kube fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef server fill:#ff9999,stroke:#333,stroke-width:2px;
    classDef gpu fill:#ffccff,stroke:#333,stroke-width:2px;
    classDef repo fill:#ccccff,stroke:#333,stroke-width:2px;
    classDef config fill:#eeeeee,stroke:#333,stroke-dasharray: 5 5;
    classDef monitor fill:#c0c0c0,stroke:#333,stroke-width:2px;
    classDef scripts fill:#d9ead3,stroke:#333,stroke-width:2px;
```
