# katanemo/archgw architecture

> Architecture diagram of the GitHub repository katanemo/archgw, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/katanemo/archgw
- Repository: https://github.com/katanemo/archgw

## Overview

##### Main Components

a. Gateway Layer:
- LLM Gateway (crates/llm_gateway)
- Prompt Gateway (crates/prompt_gateway)
- Envoy Proxy (as the underlying infrastructure)

b. Model Server Layer:
- Model Server component (model_server/)
- Function Calling System
- Guardrails System

c. External Integrations:
- LLM Providers (OpenAI, Mistral, etc.)
- Backend APIs
- Observability Stack (Grafana, Jaeger, Prometheus)

##### Key Relationships to Highlight

a. Data Flow:
- User requests → Envoy Proxy → Prompt Gateway → LLM Gateway → Model Server
- Model Server → LLM Providers
- Gateway → Backend APIs for function calling
- All components → Observability Stack

b. Control Flow:
- Configuration management flow
- Authentication and authorization paths
- Error handling paths

##### Specific Design Elements

Use the following visual elements:

##### Component Details

a. Gateway Layer (Top Level):
**┌─────────────────────────────────────────┐**
- Envoy Proxy                │
- ┬───────────────────────┤
- Prompt Gateway │     LLM Gateway       │
- ┴───────────────────────┘

b. Model Server Layer (Middle Level):
**┌─────────────────────────────────────────┐**
- Model Server                 │
- ┬─────────────┬────────────┤
- Function   │ Guardrails  │  Routing   │
- Calling    │  System     │  Logic     │
- ┴─────────────┴────────────┘

c. External Services (Bottom Level):
**┌─────────────┐ ┌─────────────┐ ┌─────────────┐**
- LLM      │ │  Backend    │ │ Observability│
- Providers  │ │    APIs     │ │   Stack      │
- ┘ └─────────────┘ └─────────────┘

##### Important Features to Highlight

- Centralized configuration management
- Prompt guarding and security features
- Observability pipelines (metrics, traces, logs)
- Load balancing and routing capabilities
- API integration points

##### Implementation Notes

- Maintain consistent spacing and alignment
- Use clear, readable fonts
- Brief annotations for complex interactions
- Creating both high-level and detailed views

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef gateway fill:#3498db,stroke:#2980b9,color:white
    classDef external fill:#2ecc71,stroke:#27ae60,color:white
    classDef security fill:#9b59b6,stroke:#8e44ad,color:white
    classDef observability fill:#e67e22,stroke:#d35400,color:white
    classDef model fill:#3498db,stroke:#2980b9,color:white
    classDef config fill:#95a5a6,stroke:#7f8c8d,color:white

    %% Legend
    subgraph Legend
        L1[Gateway Components]:::gateway
        L2[External Services]:::external
        L3[Security Components]:::security
        L4[Observability]:::observability
    end

    %% Gateway Layer
    subgraph Gateway
        EnvoyProxy[Envoy Proxy]:::gateway
        PromptGateway[Prompt Gateway]:::gateway
        LLMGateway[LLM Gateway]:::gateway
        RateLimit[Rate Limiting]:::security
        PromptGuards[Prompt Guards]:::security
        Config[Configuration Management]:::config
    end

    %% Model Server Layer
    subgraph ModelServer
        MS[Model Server]:::model
        FunctionCalling[Function Calling System]:::model
        Guardrails[Guardrails System]:::security
        Routing[Routing Logic]:::model
    end

    %% External Services
    subgraph External
        LLMProviders[LLM Providers]:::external
        BackendAPIs[Backend APIs]:::external
    end

    %% Observability Stack
    subgraph Observability
        Grafana[Grafana]:::observability
        Jaeger[Jaeger]:::observability
        Prometheus[Prometheus]:::observability
    end

    %% Relationships
    EnvoyProxy --> PromptGateway
    PromptGateway --> LLMGateway
    LLMGateway --> MS
    MS --> FunctionCalling
    MS --> Guardrails
    MS --> Routing
    Routing --> LLMProviders
    FunctionCalling --> BackendAPIs
    
    %% Observability connections
    MS -.-> Grafana
    MS -.-> Jaeger
    MS -.-> Prometheus
    LLMGateway -.-> Prometheus
    PromptGateway -.-> Prometheus

    %% Security and Config flows
    Config --> EnvoyProxy
    Config --> PromptGateway
    Config --> LLMGateway
    PromptGuards --> PromptGateway
    RateLimit --> LLMGateway

    %% Click events
    click LLMGateway "https://github.com/katanemo/archgw/tree/main/crates/llm_gateway/"
    click PromptGateway "https://github.com/katanemo/archgw/tree/main/crates/prompt_gateway/"
    click EnvoyProxy "https://github.com/katanemo/archgw/blob/main/arch/envoy.template.yaml"
    click MS "https://github.com/katanemo/archgw/tree/main/model_server/src/"
    click FunctionCalling "https://github.com/katanemo/archgw/blob/main/model_server/src/core/function_calling.py"
    click Guardrails "https://github.com/katanemo/archgw/blob/main/model_server/src/core/guardrails.py"
    click LLMProviders "https://github.com/katanemo/archgw/blob/main/crates/common/src/llm_providers.rs"
    click Grafana "https://github.com/katanemo/archgw/tree/main/demos/shared/grafana/"
    click Jaeger "https://github.com/katanemo/archgw/tree/main/demos/shared/jaeger/"
    click Prometheus "https://github.com/katanemo/archgw/tree/main/demos/shared/prometheus/"
    click PromptGuards "https://github.com/katanemo/archgw/blob/main/crates/common/src/api/prompt_guard.rs"
    click Config "https://github.com/katanemo/archgw/blob/main/arch/arch_config_schema.yaml"
    click RateLimit "https://github.com/katanemo/archgw/blob/main/crates/common/src/ratelimit.rs"
    click Routing "https://github.com/katanemo/archgw/blob/main/crates/common/src/routing.rs"
```
