# vu0607/llms-serving architecture

> Architecture diagram of the GitHub repository vu0607/llms-serving, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vu0607/llms-serving
- Repository: https://github.com/vu0607/llms-serving

## Overview

##### Project Type and Purpose
- From the project name and structure, it appears to be a backend service for serving large language models (LLMs). The single main Python entry point (app/main.py) implies the key processing happens there, likely handling API calls or inference requests.
- Deployment-related files (Dockerfile, Kubernetes manifests, and CI/CD pipeline definitions in Jenkinsfile and GitHub workflows) indicate that the service is containerized and orchestrated in a modern CI/CD environment.

##### File Structure
- Top-level folders and files:
– The “app” directory houses the application’s code, with main.py as a potential entry point.
– “requirements.txt” lists dependencies for the Python service.
– “Dockerfile” suggests the service is containerized.
– “k8s” contains Kubernetes manifests including deployment, service definitions, and ArgoCD configuration, indicating K8s is used for orchestration and continuous deployment.
– CI/CD directories (.github/workflows and the Jenkinsfile) show integration pipelines for automated build, test, and deployment.
- This structure hints at a microservice architecture for hosting the LLM-serving capability, with clear separation between application logic, deployment configuration, and CI/CD processes.

a. Main Components:
– Source Code Repository: Contains the application code (app/main.py), requirements.txt, Dockerfile, CI/CD configuration (GitHub workflows and Jenkinsfile).
– Build & CI/CD Pipelines: Represent Jenkins and GitHub actions that are triggered for build, test, and deployment.
– Containerization: Dockerfile builds the container image for the backend service.
– Orchestration Layer: Kubernetes manifests (in directory k8s) define the Deployment and Service, with ArgoCD integration for continuous deployment.
– The Core Application Service: The Python service that serves LLMs (likely an API endpoint or inference engine) running within the container.
– External Interactions: Although not explicitly shown in the file tree, consider where the service might get user input or communicate with external systems (e.g., clients sending inference requests).

b. Relationships and Interactions:
– The Source Code Repository feeds into the CI/CD pipelines.
– The CI/CD pipelines (Jenkins & GitHub workflows) build the application and create a Docker image.
– The Docker image is then deployed to Kubernetes via ArgoCD (as set in k8s/argocd-app.yaml) and the standard Kubernetes deployment/service manifests.
– The Kubernetes Service exposes the application to handle external requests.
– There might be integration with additional services (e.g., monitoring or logs), which could be highlighted if further details emerge later.

c. Architectural Patterns and Design Principles:
– Emphasis on containerization and orchestration illustrates a microservices and cloud-native approach.
– The separation between build (CI/CD pipelines), container image creation (Dockerfile), and deployment (Kubernetes) encourages a modular and reproducible deployment process.
– The use of ArgoCD points toward a GitOps workflow for deployment management.

d. Relevant Technologies and Frameworks:
– Python (for the backend service)
– Docker (for containerization)
– Kubernetes (for container orchestration)
– ArgoCD (for continuous deployment)
– Jenkins and GitHub Actions (for CI/CD pipelines)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Code Layer"
        SRC["Source Repository"]:::code
    end

    subgraph "CI/CD Layer"
        GHA["GitHub Actions (.github/workflows)"]:::cicd
        JEN["Jenkins Pipeline (Jenkinsfile)"]:::cicd
    end

    subgraph "Containerization Layer"
        DOCKER["Docker Build (Dockerfile)"]:::container
    end

    subgraph "Orchestration Layer"
        ARGO["ArgoCD (GitOps)"]:::orchestration
        K8S["Kubernetes Deployment & Service (k8s)"]:::orchestration
    end

    subgraph "Application Layer"
        APP["LLM Serving Application (app/main.py)"]:::application
    end

    EX["External Clients"]:::external

    SRC --> GHA
    SRC --> JEN
    GHA --> DOCKER
    JEN --> DOCKER
    DOCKER --> ARGO
    ARGO --> K8S
    K8S --> APP
    APP --> EX

    click SRC "https://github.com/vu0607/llms-serving/tree/main/./"
    click GHA "https://github.com/vu0607/llms-serving/blob/main/.github/workflows/deploy.yml"
    click JEN "https://github.com/vu0607/llms-serving/tree/main/Jenkinsfile"
    click DOCKER "https://github.com/vu0607/llms-serving/tree/main/Dockerfile"
    click ARGO "https://github.com/vu0607/llms-serving/blob/main/k8s/argocd-app.yaml"
    click K8S "https://github.com/vu0607/llms-serving/tree/main/k8s"
    click APP "https://github.com/vu0607/llms-serving/blob/main/app/main.py"

    classDef code fill:#cce5ff,stroke:#004085,color:#002752;
    classDef cicd fill:#d4edda,stroke:#155724,color:#0c5460;
    classDef container fill:#fff3cd,stroke:#856404,color:#856404;
    classDef orchestration fill:#d1ecf1,stroke:#0c5460,color:#0c5460;
    classDef application fill:#f8d7da,stroke:#721c24,color:#721c24;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#383d41;
```
