How devansh-365/freellm works
FreeLLM is an OpenAI-compatible gateway that accepts chat requests, chooses a provider/model strategy, fails over when needed, and returns normalized responses. Its primary workflow passes through authentication and chat routing into provider adapters; streaming requests use a separate normalization path. The README also documents caching, privacy and strict routing, usage/status reporting, scoped keys, browser tokens, and a dashboard. Some of these subsystems are unsampled, so their connections are shown only where the README or excerpts establish them.
Main entry points: router.ts Model selection and failover · app.ts HTTP application · chat.ts Chat endpoint · auth.ts Authentication
Components 23
Client experience
- Dashboard UI
apps/dashboard/src/App.tsx - Dashboard API hooks
apps/dashboard/src/api/hooks.ts
Gateway API
- HTTP application
apps/gateway/src/app.ts - Chat endpoint
apps/gateway/src/routes/v1/chat.ts - Models endpoint
apps/gateway/src/routes/v1/models.ts - Status endpoint
apps/gateway/src/routes/v1/status.ts - Token issuance
apps/gateway/src/routes/v1/tokens.ts - JSON mode
apps/gateway/src/features/json-mode.ts - Stream normalization
apps/gateway/src/streaming/pipeline.ts - Error responses
apps/gateway/src/middleware/error-handler.ts
Routing and providers
- Model selection and failover
apps/gateway/src/routing/router.ts - Provider registry
apps/gateway/src/routing/registry.ts - Response cache
apps/gateway/src/routing/cache.ts - Privacy routing
apps/gateway/src/routing/privacy.ts - Strict mode
apps/gateway/src/routing/strict.ts - Provider adapters
apps/gateway/src/providers/base.ts
Access and observability
- Authentication
apps/gateway/src/middleware/auth.ts - Virtual keys
apps/gateway/src/features/virtual-keys.ts - Browser tokens
apps/gateway/src/features/browser-tokens.ts - Per-user limits
apps/gateway/src/features/identifier-limiter.ts - Usage and request logs
apps/gateway/src/observability/usage-tracker.ts
Other components
- SDK / app
- LLM providers
Connections 24
- SDK / app → HTTP application: sends API requests
README.md - Dashboard UI → Dashboard API hooks: queries gateway data
apps/dashboard/src/api/hooks.ts - Dashboard API hooks → HTTP application: calls API
- HTTP application → Authentication: authenticates requests
apps/gateway/src/app.ts - HTTP application → Per-user limits: limits identifiers
apps/gateway/src/app.ts - HTTP application → Chat endpoint: mounts routes
apps/gateway/src/app.ts - HTTP application → Error responses: handles errors
apps/gateway/src/app.ts - Authentication → Virtual keys: checks credentials
apps/gateway/src/middleware/auth.ts - Authentication → Browser tokens: verifies tokens
apps/gateway/src/middleware/auth.ts - Chat endpoint → Model selection and failover: routes completion
apps/gateway/src/routes/v1/chat.ts - Chat endpoint → JSON mode: translates JSON mode
apps/gateway/src/routes/v1/chat.ts - Chat endpoint → Stream normalization: normalizes streams
apps/gateway/src/routes/v1/chat.ts - Chat endpoint → Virtual keys: checks and records caps
apps/gateway/src/routes/v1/chat.ts - Model selection and failover → Provider registry: selects providers
apps/gateway/src/routing/index.ts - Model selection and failover → Response cache: reads and writes cache
apps/gateway/src/routing/router.ts - Model selection and failover → Privacy routing: filters providers
apps/gateway/src/routing/router.ts - Model selection and failover → Strict mode: enforces strict mode
apps/gateway/src/routing/router.ts - Model selection and failover → Provider adapters: dispatches completion
apps/gateway/src/routing/router.ts - Provider adapters → LLM providers: sends model requests
README.md - Model selection and failover → Usage and request logs: records outcomes
apps/gateway/src/routing/router.ts - Models endpoint → Provider registry: reports available models
- Status endpoint → Usage and request logs: reports usage and health
- Token issuance → Browser tokens: issues tokens
- Error responses → Model selection and failover: builds retry advice
apps/gateway/src/middleware/error-handler.ts