How winstxnhdw/nllb-api works
Project Type and Purpose
• This is a microservice-style REST API for machine translation built on Meta’s NLLB distilled 1.3B 8-bit quantised model.
• Implements language detection and translation endpoints, served via FastAPI/uvicorn.
• Provides a Rust client crate (and Python packaging wrapper) for easy integration.
File Structure
• Top‐level artifacts:
• Dockerfiles & Makefile for container build and dev workflow
• client/ folder: Rust crate (with Cargo.toml, build.rs) plus Python packaging metadata
• server/ folder: Python FastAPI application with subfolders for API endpoints, feature logic, guards, lifespan hooks, schemas, utils, and type definitions
• tests/: pytest tests for API and utils
• Lifespan hooks load models and register the service in Consul at startup.
• CI/CD workflows (GitHub Actions) handle building, testing, CUDA builds, linting, and deployment to Hugging Face Spaces.
README for Additional Insights
• Tech stack: Python (FastAPI), Rust (client), CTranslate2 (inference engine), Hugging Face Hub (model download), Consul (service discovery), Docker.
• Exposes endpoints:
• /api/v4/language → language detection
• /api/v4/translator (+ /stream) → translation (SSE for streaming)
• Environment-driven config for ports, CORS, threading (OMP_NUM_THREADS, TRANSLATOR_THREADS), worker count, and optional CUDA support.
• CI/CD pushes container images to GHCR and auto-deploys to Hugging Face Spaces.
Main Components & Relationships
a. Client Library
• Rust crate + Python wrapper exposing AsyncTranslatorClient
• Makes HTTP(S) requests to the API server
b. API Server (Python)
• FastAPI app (server/app.py)
• API layer (server/api/v4/...)
• Feature/business logic (server/features/...)
• Lifespan hooks (server/lifespans/...): load models, register with Consul
• Guards (server/guards/...): secret-based access control
• Schemas (server/schemas/v1/...): Pydantic request/response models
• Utilities (server/utils/...): downloading from HF, internet check
• Typings & typedefs for CTranslate2, tokenizers, FastText bindings
c. Inference Engine
• CTranslate2 native library invoked via Python bindings
d. Model Repository
• Hugging Face Hub for downloading the quantised NLLB model
e. External Services
• Consul for service registration/discovery
• (Optional) GPU/CUDA if built with the USE_CUDA flag
f. Infrastructure & CI/CD
• Docker build (CPU vs CUDA)
• GitHub Actions workflows for build/test/deploy
• Container Registry (GHCR) → Hugging Face Spaces
Data Flow & Interaction
• Client → HTTP request → API Server
• Server startup → download model from HF Hub → load into CTranslate2
• Request enters API layer → feature logic → CTranslate2 inference → response
• Lifespan hook registers service address in Consul
• CI/CD pipeline → Docker image → deploy to Spaces