How winstxnhdw/llm-api works
Project type and purpose
• Project type: A backend API service (FastAPI) for serving Llama-3.2 inference via HTTP.
• Purpose: Expose a /api/v1/chat endpoint that accepts user messages, runs them through a CTranslate2-backed model, and streams back answers. Hosted in a Docker container on Hugging Face Spaces, with optional Consul registration for service discovery.
File structure
• Top-level files:
– Dockerfile* & Dockerfile.build – containerization
– pyproject.toml – Python dependencies & metadata
– README.md – usage & deployment info
• server/: application code
– app.py – FastAPI application factory & route registration
– api/ – HTTP route handlers (health.py & v1/chat.py)
– features/chat/model.py – business logic for chat inference
– lifespans/ – startup/shutdown hooks (model loading, Consul registration)
– schemas/ – Pydantic request/response models
– config.py – environment-driven configuration
– typedefs/ & typings/ – type definitions
README for additional insights
• Inference engine: CTranslate2
• Hosting: Hugging Face Spaces (container exposes Swagger UI)
• Deployment workflows via GitHub Actions
Main components & their interactions
a. Client (curl, any HTTP client)
b. Reverse proxy / Load balancer (Hugging Face Spaces infra)
c. API Server (FastAPI app inside Docker container)
d. Lifespan managers
– ChatModelLoader: loads CTranslate2 model into memory
– ConsulRegistrar: optionally registers service to Consul
e. Chat Model Engine (CTranslate2 + model artifacts)
f. Configuration store (environment variables via config.py)
g. (Optional) External services: Consul for service discovery
Interactions:
1. Client → HTTP(S) → FastAPI Server
2. On startup, Server lifespans hook loads model & registers to Consul
3. /api/v1/chat handler calls features/chat/model.py which runs inference through CTranslate2
4. Model returns streaming tokens → handler serializes via Pydantic schemas → HTTP response
Architectural patterns & principles
• Layered architecture:
– Presentation (API endpoints)
– Business logic (chat feature module)
– Infrastructure (CTranslate2 inference engine)
• Dependency injection via lifespans hooks (startup/shutdown)
• Schema validation (Pydantic)
• Containerized deployment with clear build vs runtime images
Final checklist
• All components clearly labeled
• Key technologies called out (FastAPI, CTranslate2, Docker, Consul)