How katanemo/archgw works
Main Components
a. Gateway Layer:
• LLM Gateway (crates/llm_gateway)
• Prompt Gateway (crates/prompt_gateway)
• Envoy Proxy (as the underlying infrastructure)
b. Model Server Layer:
• Model Server component (model_server/)
• Function Calling System
• Guardrails System
c. External Integrations:
• LLM Providers (OpenAI, Mistral, etc.)
• Backend APIs
• Observability Stack (Grafana, Jaeger, Prometheus)
Key Relationships to Highlight
a. Data Flow:
• User requests → Envoy Proxy → Prompt Gateway → LLM Gateway → Model Server
• Model Server → LLM Providers
• Gateway → Backend APIs for function calling
• All components → Observability Stack
b. Control Flow:
• Configuration management flow
• Authentication and authorization paths
• Error handling paths
Specific Design Elements
Use the following visual elements:
Component Details
a. Gateway Layer (Top Level):
┌─────────────────────────────────────────┐
• Envoy Proxy │
• ┬───────────────────────┤
• Prompt Gateway │ LLM Gateway │
• ┴───────────────────────┘
b. Model Server Layer (Middle Level):
┌─────────────────────────────────────────┐
• Model Server │
• ┬─────────────┬────────────┤
• Function │ Guardrails │ Routing │
• Calling │ System │ Logic │
• ┴─────────────┴────────────┘
c. External Services (Bottom Level):
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
• LLM │ │ Backend │ │ Observability│
• Providers │ │ APIs │ │ Stack │
• ┘ └─────────────┘ └─────────────┘
Important Features to Highlight
• Centralized configuration management
• Prompt guarding and security features
• Observability pipelines (metrics, traces, logs)
• Load balancing and routing capabilities
• API integration points
Implementation Notes
• Maintain consistent spacing and alignment
• Use clear, readable fonts
• Brief annotations for complex interactions
• Creating both high-level and detailed views