# winstxnhdw/llm-api architecture

> Architecture diagram of the GitHub repository winstxnhdw/llm-api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/winstxnhdw/llm-api
- Repository: https://github.com/winstxnhdw/llm-api

## Overview

##### Project type and purpose
- Project type: A backend API service (FastAPI) for serving Llama-3.2 inference via HTTP.
- Purpose: Expose a `/api/v1/chat` endpoint that accepts user messages, runs them through a CTranslate2-backed model, and streams back answers. Hosted in a Docker container on Hugging Face Spaces, with optional Consul registration for service discovery.

##### File structure
- Top-level files:
– Dockerfile* & Dockerfile.build – containerization
– pyproject.toml – Python dependencies & metadata
– README.md – usage & deployment info
- `server/`: application code
– `app.py` – FastAPI application factory & route registration
– `api/` – HTTP route handlers (`health.py` & `v1/chat.py`)
– `features/chat/model.py` – business logic for chat inference
– `lifespans/` – startup/shutdown hooks (model loading, Consul registration)
– `schemas/` – Pydantic request/response models
– `config.py` – environment-driven configuration
– `typedefs/` & `typings/` – type definitions

##### README for additional insights
- Inference engine: CTranslate2
- Hosting: Hugging Face Spaces (container exposes Swagger UI)
- Deployment workflows via GitHub Actions

##### Main components & their interactions
a. Client (curl, any HTTP client)
b. Reverse proxy / Load balancer (Hugging Face Spaces infra)
c. API Server (FastAPI app inside Docker container)
d. Lifespan managers
– ChatModelLoader: loads CTranslate2 model into memory
– ConsulRegistrar: optionally registers service to Consul
e. Chat Model Engine (CTranslate2 + model artifacts)
f. Configuration store (environment variables via `config.py`)
g. (Optional) External services: Consul for service discovery

Interactions:
1. Client → HTTP(S) → FastAPI Server
2. On startup, Server lifespans hook loads model & registers to Consul
3. `/api/v1/chat` handler calls `features/chat/model.py` which runs inference through CTranslate2
4. Model returns streaming tokens → handler serializes via Pydantic schemas → HTTP response

##### Architectural patterns & principles
- Layered architecture:
– Presentation (API endpoints)
– Business logic (chat feature module)
– Infrastructure (CTranslate2 inference engine)
- Dependency injection via lifespans hooks (startup/shutdown)
- Schema validation (Pydantic)
- Containerized deployment with clear build vs runtime images

##### Final checklist
- All components clearly labeled
- Key technologies called out (FastAPI, CTranslate2, Docker, Consul)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Entities
    Client["Client (curl or HTTP client)"]:::external
    Proxy["Reverse Proxy / Load Balancer\n(Hugging Face Spaces Infra)"]:::infra
    Consul["Consul Service Discovery (Optional)"]:::infra

    %% Docker Container Group
    subgraph "Docker Container" 
        direction TB
        Dockerfiles["Docker Build & Containerization"]:::code
        PyProject["Python Dependencies & Metadata"]:::code
        README["Project Documentation"]:::code

        Config["Config Store\n(env vars via config.py)"]:::code
        API["API Server\n(FastAPI + Uvicorn)"]:::compute

        subgraph "Lifespans Hooks"
            direction TB
            ChatLoader["Chat Model Loader\n(CTranslate2 Setup)"]:::code
            ConsulReg["Consul Service Registrar"]:::code
        end

        ModelEngine["Chat Model Engine\n(CTranslate2 + Artifacts)"]:::compute
        Schemas["Schemas & Types\n(Pydantic Models, Typedefs, Typings)"]:::code
    end

    %% Flows
    Client -->|"HTTP/HTTPS"| Proxy
    Proxy -->|"forwards request"| API
    API -->|"load config"| Config
    API -->|"startup"| ChatLoader
    API -->|"startup"| ConsulReg
    ChatLoader -->|"loads model into memory"| ModelEngine
    API -->|"calls inference"| ModelEngine
    ModelEngine -->|"stream tokens"| API
    API -->|"responds"| Client
    ConsulReg -->|"register service"| Consul

    %% Click Events
    click Dockerfiles "https://github.com/winstxnhdw/llm-api/tree/main/Dockerfile"
    click PyProject "https://github.com/winstxnhdw/llm-api/blob/main/pyproject.toml"
    click README "https://github.com/winstxnhdw/llm-api/blob/main/README.md"
    click Config "https://github.com/winstxnhdw/llm-api/blob/main/server/config.py"
    click API "https://github.com/winstxnhdw/llm-api/blob/main/server/app.py"
    click ChatLoader "https://github.com/winstxnhdw/llm-api/blob/main/server/lifespans/chat_model.py"
    click ConsulReg "https://github.com/winstxnhdw/llm-api/blob/main/server/lifespans/consul_register.py"
    click ModelEngine "https://github.com/winstxnhdw/llm-api/blob/main/server/features/chat/model.py"
    click Schemas "https://github.com/winstxnhdw/llm-api/blob/main/server/schemas/v1/query.py"
    click Schemas "https://github.com/winstxnhdw/llm-api/blob/main/server/schemas/v1/answer.py"
    click Schemas "https://github.com/winstxnhdw/llm-api/blob/main/server/schemas/v1/benchmark.py"
    click Schemas "https://github.com/winstxnhdw/llm-api/blob/main/server/typedefs/message.py"
    click Schemas "https://github.com/winstxnhdw/llm-api/blob/main/server/typedefs/state.py"
    click Schemas "https://github.com/winstxnhdw/llm-api/blob/main/server/typings/ctranslate2/__init__.pyi"
    click Schemas "https://github.com/winstxnhdw/llm-api/blob/main/server/typings/transformers/models/llama/__init__.pyi"

    %% Classes
    classDef compute fill:#E0F7FA,stroke:#0288D1,color:#000
    classDef infra fill:#FFF9C4,stroke:#F9A825,color:#000
    classDef code fill:#F1F8E9,stroke:#43A047,color:#000
    classDef external fill:#ECEFF1,stroke:#90A4AE,color:#000
```
