How maximhq/bifrost works
Purpose and Entry Points
Bifrost is a Go-based AI gateway that normalizes inference across many LLM and multimodal providers behind OpenAI-compatible and provider-compatible HTTP APIs. It can also be embedded as a Go SDK.
• HTTP server executable: transports/bifrost-http/main.go
• Core SDK façade: core/bifrost.go
• Interactive/local CLI: cli/main.go
• Browser administration UI is a separate TypeScript application served by the gateway: ui/app/main.tsx, transports/bifrost-http/handlers/ui.go
HTTP Gateway and API Compatibility
The HTTP transport owns server initialization, routing, authentication/middleware, request validation, streaming/websocket handling, inference endpoints, and management APIs. It is the boundary between external clients and the core runtime.
• Server composition and lifecycle: transports/bifrost-http/server/server.go
• Canonical inference request handlers: transports/bifrost-http/handlers/inference.go
• SDK-compatibility routing for OpenAI, Anthropic, Bedrock, GenAI, LangChain, LiteLLM, and others: transports/bifrost-http/integrations/router.go, transports/bifrost-http/integrations/openai.go
• Public API contract is maintained under docs/openapi/openapi.yaml and docs/openapi/paths/inference/
Core Inference Runtime
core defines shared schemas and orchestrates request execution. It transforms normalized requests into provider-specific API calls, selects configured keys, handles errors/streams, and returns normalized responses. Provider modules implement the external-service boundary rather than exposing provider details to transport callers.
• Primary runtime and shared API: core/bifrost.go, core/schemas/bifrost.go
• Provider interface/schema layer: core/schemas/provider.go
• Representative provider adapters: core/providers/openai/openai.go, core/providers/anthropic/anthropic.go, core/providers/bedrock/bedrock.go
• Streaming and provider HTTP utilities: core/providers/utils/stream.go, core/network/http.go
Plugin Pipeline, Routing, and Governance
Plugins are a first-class interception boundary around inference and MCP operations. The core pipeline loads configured plugins; plugins can implement request policy, routing, caching, logging, telemetry, mocking, or compatibility behavior without embedding those concerns in providers.
• Pipeline and plugin interfaces: core/mcp/pluginpipeline.go, core/schemas/plugin.go
• Dynamic/native plugin loading: framework/plugins/loader.go, framework/plugins/soloader.go
• Governance applies virtual-key, budget, rate-limit, complexity, and routing policies: plugins/governance/main.go, plugins/governance/routing.go
• Built-in semantic caching uses vector stores: plugins/semanticcache/main.go, plugins/semanticcache/search.go
MCP Gateway
Bifrost can manage MCP clients/servers, discover and filter tools, execute tools, maintain sessions, and support agent/code modes. MCP credentials are isolated behind credential-store implementations, including per-user headers and OAuth.
• MCP initialization and client orchestration: core/mcp/init.go, core/mcp/clientmanager.go
• Tool inventory/execution: core/mcp/toolmanager.go, core/mcp/exec.go
• Agent and code-mode behavior: core/mcp/agent.go, core/mcp/codemode.go
• HTTP MCP, sessions, and OAuth surfaces: transports/bifrost-http/handlers/mcp.go, transports/bifrost-http/handlers/mcpoauth2.go
Persistence, Catalogs, and Observability
The optional framework layer abstracts operational state from inference execution. It supports SQLite/Postgres configuration and logs, ClickHouse logs, object storage, model/pricing catalogs, and vector-backed semantic cache. This permits zero-config/in-memory-style operation or durable production deployment.
• Config state, migrations, and secret-related persistence: framework/configstore/store.go, framework/configstore/migrations.go
• Request/usage log storage: framework/logstore/store.go, framework/logstore/clickhouse.go
• Model catalog and pricing synchronization: framework/modelcatalog/main.go, framework/modelcatalog/datasheet/sync.go
• Storage integrations: framework/vectorstore/store.go, framework/objectstore/s3.go, framework/objectstore/gcs.go
Operations and Deployment
The primary runtime is Go; the admin UI is TypeScript/React-style frontend code. Deployment targets include Docker, Kubernetes/Helm, and Terraform for AWS, Azure, GCP, and generic Kubernetes. External dependencies commonly include LLM provider APIs, PostgreSQL/SQLite, ClickHouse, Redis, Qdrant, Weaviate, Pinecone, S3/GCS, Prometheus, and OpenTelemetry collectors.
• Container build/runtime: transports/Dockerfile, transports/docker-entrypoint.sh
• Helm deployment plus optional PostgreSQL/Redis/Qdrant/Weaviate: helm-charts/bifrost/templates/deployment.yaml, helm-charts/bifrost/values.yaml
• Cloud and Kubernetes provisioning modules: terraform/modules/bifrost/main.tf
Main entry points: bifrost.go Inference orchestration · server.go HTTP server · inference.go Inference handlers · pluginpipeline.go Plugin pipeline
Components 21
Clients
HTTP Gateway
- HTTP server · Owns lifecycle, routing, middleware, authentication, and API endpoints.
transports/bifrost-http/server/server.go - Inference handlers · Validates canonical requests and manages streaming or websocket responses.
transports/bifrost-http/handlers/inference.go - Compatibility APIs · Adapts OpenAI and other provider-compatible request formats.
transports/bifrost-http/integrations/router.go - OpenAPI contract
docs/openapi/openapi.yaml - MCP HTTP & OAuth
transports/bifrost-http/handlers/mcp.go
Core Runtime
- Go SDK façade · Embeddable programmatic entry point.
core/bifrost.go - Inference orchestration · Normalizes execution, selects keys, invokes providers, and returns normalized results.
core/bifrost.go - Provider contract
core/schemas/provider.go - Provider adapters · Implements OpenAI, Anthropic, Bedrock, and other provider boundaries.
core/providers/openai/openai.go
Plugins & MCP
- Plugin pipeline · Runs configured hooks around inference and MCP operations.
core/mcp/pluginpipeline.go - Plugin loader · Loads configured native and dynamically linked plugins.
framework/plugins/loader.go - Governance plugin · Applies virtual-key, budget, rate-limit, complexity, and routing policy.
plugins/governance/main.go - Semantic cache · Uses vector similarity to serve cached inference results.
plugins/semanticcache/main.go - MCP orchestration · Manages clients, tools, sessions, agent/code modes, and credential isolation.
core/mcp/clientmanager.go
Operational State
- Config store · Stores configuration, migrations, and secret-related state in SQLite or Postgres.
framework/configstore/store.go - Usage log store · Persists request and usage logs, including ClickHouse support.
framework/logstore/store.go - Model & pricing catalog
framework/modelcatalog/main.go - Vector store · Backs semantic cache search.
framework/vectorstore/store.go
Delivery
- Container & Helm deployment · Packages the gateway for Docker and Kubernetes.
helm-charts/bifrost/templates/deployment.yaml
Connections 22
- Admin UI → HTTP server: served by
- Local CLI → Go SDK façade: uses
- HTTP server → Inference handlers: routes requests
- HTTP server → Compatibility APIs: routes compatible APIs
- OpenAPI contract → Inference handlers: defines contract
- Compatibility APIs → Inference handlers: adapts to canonical API
- Inference handlers → Inference orchestration: executes inference
- Go SDK façade → Inference orchestration: invokes
- Inference orchestration → Plugin pipeline: intercepts through
- Plugin loader → Plugin pipeline: loads plugins into
- Plugin pipeline → Governance plugin: applies policy
- Plugin pipeline → Semantic cache: checks cache
- Semantic cache → Vector store: similarity search
- Inference orchestration → Provider contract: targets
- Provider contract → Provider adapters: implemented by
- Inference handlers → MCP orchestration: coordinates MCP
- MCP HTTP & OAuth → MCP orchestration: exposes MCP and OAuth
- HTTP server → MCP HTTP & OAuth: routes MCP APIs
- Inference orchestration → Config store: reads runtime configuration
- Inference orchestration → Usage log store: records usage
- Inference orchestration → Model & pricing catalog: uses model metadata
- Container & Helm deployment → HTTP server: deploys