# meta-llama/llama-stack architecture

> Architecture diagram of the GitHub repository meta-llama/llama-stack, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/meta-llama/llama-stack
- Repository: https://github.com/meta-llama/llama-stack

## Overview

Key Components to Include:

##### Core API Layer
- Core APIs into logical clusters:
- Inference API
- Safety API
- Memory API
- Agents API
- Eval API
- Telemetry API

##### Resource Management Layer
- Models Registry
- Shields Registry
- Memory Banks
- Eval Tasks
- Datasets
- Scoring Functions

##### Distribution System
- Multiple deployment environments:
- Single Node Deployment
- Hosted Cloud Deployment
- On-Device Deployment (iOS/Android)
- Container orchestration layer (Docker-based)

##### Provider Integration Layer
- Meta Reference
- Cerebras
- Fireworks
- AWS Bedrock
- Together
- Others (vLLM, TGI, etc.)

##### Client SDK Layer
- Multiple language SDK implementations:
- Python
- Swift
- Node
- Kotlin

##### Use a top-down approach
- Client SDKs at the top
- API Gateway in the middle
- Providers and Resources at the bottom

##### Component Grouping
- Use rectangles with rounded corners for logical grouping
- Clear labels for each group
- Version numbers where applicable

##### Additional Elements
- Deployment environment boundaries
- Security perimeter
- Monitoring and telemetry flow

Key Architectural Patterns to Highlight:

##### Service-Oriented Architecture (SOA)
- Clear service boundaries
- REST API interfaces

##### Provider Pattern
- Demonstrate how different providers implement the same interfaces
- Plugin architecture

##### Distribution Pattern
- How the same APIs work across different deployment scenarios

##### Resource Management
- How resources are shared and accessed across services

- API-first design philosophy
- Composability of services
- Deployment flexibility
- Provider extensibility
- Standardized interfaces

Remember to maintain a balance between detail and clarity, focusing on the architectural relationships while avoiding implementation-specific details that might change frequently.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Style Definitions
    classDef api fill:#2374AB,color:white
    classDef resource fill:#417B5A,color:white
    classDef provider fill:#9C6615,color:white
    classDef distribution fill:#738290,color:white
    classDef sdk fill:#D81159,color:white
    classDef experimental fill:#FFB30F,color:white

    %% Client SDKs
    subgraph SDKs
        Python["Python SDK"]:::sdk
        Swift["Swift SDK"]:::sdk
        Node["Node SDK"]:::sdk
        Kotlin["Kotlin SDK"]:::sdk
    end

    %% API Gateway
    Gateway["API Gateway"]

    %% Core APIs
    subgraph CoreAPIs
        InferenceAPI["Inference API"]:::api
        SafetyAPI["Safety API"]:::api
        MemoryAPI["Memory API"]:::api
        AgentsAPI["Agents API"]:::api
        EvalAPI["Eval API"]:::api
        TelemetryAPI["Telemetry API"]:::experimental
    end

    %% Resource Management
    subgraph Resources
        ModelsRegistry["Models Registry"]:::resource
        ShieldsRegistry["Shields Registry"]:::resource
        MemoryBanks["Memory Banks"]:::resource
        EvalTasks["Eval Tasks"]:::resource
        Datasets["Datasets"]:::resource
        ScoringFunctions["Scoring Functions"]:::resource
    end

    %% Distribution System
    subgraph Distribution
        SelfHosted["Self-hosted"]:::distribution
        Cloud["Cloud Deployment"]:::distribution
        OnDevice["On-Device"]:::distribution
    end

    %% Providers
    subgraph Providers
        MetaRef["Meta Reference"]:::provider
        Cerebras["Cerebras"]:::provider
        Fireworks["Fireworks"]:::provider
        Bedrock["AWS Bedrock"]:::provider
        Together["Together"]:::provider
        VLLM["vLLM"]:::provider
        TGI["TGI"]:::provider
    end

    %% Connections
    Python & Swift & Node & Kotlin --> Gateway
    Gateway --> CoreAPIs
    InferenceAPI & SafetyAPI & MemoryAPI & AgentsAPI & EvalAPI --> Resources
    CoreAPIs --> Distribution
    Distribution --> Providers

    %% Click Events
    click InferenceAPI "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/inference/inference.py"
    click SafetyAPI "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/safety/safety.py"
    click MemoryAPI "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/memory/memory.py"
    click AgentsAPI "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/agents/agents.py"
    click EvalAPI "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/eval/eval.py"
    click TelemetryAPI "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/telemetry/telemetry.py"
    click ModelsRegistry "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/models/models.py"
    click ShieldsRegistry "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/shields/shields.py"
    click MemoryBanks "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/memory_banks/memory_banks.py"
    click EvalTasks "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/eval_tasks/eval_tasks.py"
    click Datasets "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/datasets/datasets.py"
    click ScoringFunctions "https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/scoring_functions/scoring_functions.py"
    click SelfHosted "https://github.com/meta-llama/llama-stack/tree/main/distributions/meta-reference-gpu/"
    click OnDevice "https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/inline/ios/"
    click MetaRef "https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/inline/inference/meta_reference/"
    click Cerebras "https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/remote/inference/cerebras/"
    click Fireworks "https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/remote/inference/fireworks/"
    click Bedrock "https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/remote/inference/bedrock/"
    click Together "https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/remote/inference/together/"
    click VLLM "https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/remote/inference/vllm/"
    click TGI "https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/remote/inference/tgi/"

    %% Legend
    subgraph Legend
        L1["API"]:::api
        L2["Resource"]:::resource
        L3["Provider"]:::provider
        L4["Distribution"]:::distribution
        L5["SDK"]:::sdk
        L6["Experimental"]:::experimental
    end
```
