How bricks-cloud/bricksllm works
Project Type and Purpose
• BricksLLM is a backend API gateway built in Go that integrates with several large language model (LLM) providers (e.g., OpenAI, Anthropic, Azure OpenAI, vLLM).
• Its purpose is to provide enterprise features around rate limiting, cost management, PII redaction, caching, and failover for LLM usage.
• Use cases include controlled API key distribution, failovers, analytics, logging with privacy control, and integration with external model providers.
File Structure to Identify Architectural Components
• The top-level structure shows a clear Go project layout with “cmd”, “internal”, and configuration files. The main application entry point is in cmd/bricksllm.
• The “internal” folder is broken down into submodules such as authentication, cache, config, event, logger, manager, provider, server, storage, telemetry, and more. This indicates a layered, modular design.
• Notice the “internal/server/web” directory, which is split into “admin” and “proxy” subdirectories:
• The Admin component handles management, reporting, and policy configuration.
• The Proxy component is the entry point for handling client API requests (that are eventually forwarded to various LLM providers).
• There are also directories for providers (OpenAI, Anthropic, Azure, vLLM, etc.) that encapsulate external API interactions and cost/token handling.
• The existence of Dockerfiles, docker-compose.yml, and Kubernetes Helm Charts reveals that containerization and orchestration are supported.
Key Components and Their Relationships
a. Main Components to Include:
• Clients/Users: External clients, SDKs, or applications consuming the API.
• API Gateway (Proxy Server): The entry point for client requests, handling routing, rate limiting, caching, and failover.
• Admin Server: Provides Swagger documentation endpoints, management, and reporting functionalities.
• Internal Business Logic: Modules/services that implement authentication, access control, cost/rate management, key management, encryption, event handling, and PII detection.
• Providers: Components interfacing with external LLM providers (OpenAI, Anthropic, Azure, vLLM, Deepinfra, and custom integrations).
• Storage Layer:
• Postgresql for persistent storage (configuration, rate limits, user key data, etc.)
• Redis for caching and quick-access storage needs.
• Auxiliary Services: Logging (using Zap), telemetry (Prometheus, StatsD/Datadog), and messaging (internal event bus).
b. Relationships and Interactions:
• Client requests enter through the Proxy API server, with the admin server separately available for management.
• The API gateway will validate requests using authenticator modules and then route them to appropriate provider modules based on the target (e.g., OpenAI or Anthropic).
• Before forwarding, modules for rate limiting, cost check, and PII detection intervene.
• All key configurations and events are stored in Postgresql, while caching (such as API key data and temporary responses) utilizes Redis.
• Internal logging and telemetry modules collect metrics and send events to external monitoring systems (like Datadog or Prometheus).
• The project uses Docker, Kubernetes (via Helm charts), and standard CI/CD workflows (as seen in .github/workflows) to manage deployments.
Important Architectural Patterns and Design Principles
• Modular design: Separation between internal business logic (managers, providers, and validators) and API server layers.
• Separation of Concerns: Distinct roles managed by the Admin Server versus the Proxy Server.
• Extensibility: Provider modules are isolated, enabling ease of adding or modifying support for new LLM providers.
• Resiliency: Built-in caching, retry, and failover mechanisms to enhance stability and performance.
• Infrastructure as Code: Docker-compose and Kubernetes Helm charts for deployment consistency.
Elements
• Components should have clear labels (e.g., “Proxy API Server”, “Admin Server”, “Rate Limiter”, “PII Detection Module”, “OpenAI Provider”, etc.).
• From Clients -> Proxy API Layer.
• From Proxy -> Pipeline modules (Authentication, Rate Limit, Failover/Caching, etc.).
• From internal components -> External LLM providers.
• Feedback loops from internal logging/telemetry back to monitoring systems.
• Rectangular blocks for servers and services.