# nod-ai/shark-ai architecture

> Architecture diagram of the GitHub repository nod-ai/shark-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nod-ai/shark-ai
- Repository: https://github.com/nod-ai/shark-ai

## Overview

##### Main Components Overview

a. Shortfin (Inference & Serving)
- High-performance inference library
- Serving engine components
- Device management (AMD GPU, CPU)
- Array operations and storage

b. Sharktank (Model Management)
- Model recipes and conversion tools
- Quantization support
- Model implementations (SDXL, LLaMA, etc.)
- Dataset management

c. Tuner (Performance Optimization)
- Parameter configuration search
- Dispatch constraints
- Candidate generation
- Performance testing

##### Key Relationships to Highlight

- Data flow from Sharktank's model recipes to Shortfin's serving components
- Tuner's optimization feedback loop with both Sharktank and Shortfin
- Integration points between components

##### Component Details

For Shortfin:
**[Shortfin Core]**
- Device Management
- AMD GPU System
- CPU System
- Array Operations
- Serving Components
- LLM Service
- Stable Diffusion Service
- Async Runtime

For Sharktank:
**[Sharktank Core]**
- Model Implementations
- LLaMA
- SDXL
- Other Models
- Quantization Tools
- Dataset Management
- Export Tools

For Tuner:
**[Tuner Core]**
- Parameter Search
- Dispatch Constraints
- Candidate Generation
- Performance Testing

##### Integration Points

Highlight:
- Python API interfaces
- Model serving endpoints
- Inter-component communication paths
- External system interfaces (ROCm, IREE)

##### Deployment Flow

Progression:
**Model Definition (Sharktank)**
- ↓
**Optimization (Tuner)**
- ↓
**Deployment (Shortfin)**
- ↓
**Serving (Applications)**

##### Additional Elements

Include:
- Version compatibility information
- System requirements
- Key dependencies (ROCm, Python, Linux)

## Diagram (Mermaid)

```mermaid
graph TB
    %% Main Components
    subgraph Shortfin["Shortfin - Inference & Serving"]
        SF_Core["Shortfin Core"]:::core
        DevMgmt["Device Management"]:::system
        ArrayOps["Array Operations"]:::system
        ServComp["Serving Components"]:::service
        AsyncRT["Async Runtime"]:::system
        
        SF_Core --> DevMgmt
        SF_Core --> ArrayOps
        SF_Core --> ServComp
        SF_Core --> AsyncRT
        
        subgraph DeviceSystems
            AMD["AMD GPU System"]:::hardware
            CPU["CPU System"]:::hardware
            DevMgmt --> AMD
            DevMgmt --> CPU
        end
        
        subgraph Services
            LLM["LLM Service"]:::service
            SD["Stable Diffusion Service"]:::service
            ServComp --> LLM
            ServComp --> SD
        end
    end
    
    subgraph Sharktank["Sharktank - Model Management"]
        ST_Core["Sharktank Core"]:::core
        ModelImpl["Model Implementations"]:::model
        QuantTools["Quantization Tools"]:::tool
        DatasetMgmt["Dataset Management"]:::data
        ExportTools["Export Tools"]:::tool
        
        ST_Core --> ModelImpl
        ST_Core --> QuantTools
        ST_Core --> DatasetMgmt
        ST_Core --> ExportTools
        
        subgraph Models
            LLAMA["LLaMA"]:::model
            SDXL["SDXL"]:::model
            Mixtral["Mixtral"]:::model
            ModelImpl --> LLAMA
            ModelImpl --> SDXL
            ModelImpl --> Mixtral
        end
    end
    
    subgraph Tuner["Tuner - Performance Optimization"]
        T_Core["Tuner Core"]:::core
        ParamSearch["Parameter Search"]:::tool
        DispConst["Dispatch Constraints"]:::tool
        CandGen["Candidate Generation"]:::tool
        PerfTest["Performance Testing"]:::tool
        
        T_Core --> ParamSearch
        T_Core --> DispConst
        T_Core --> CandGen
        T_Core --> PerfTest
    end
    
    %% Inter-component Relationships
    Sharktank --> |"Model Recipes"| Shortfin
    Tuner --> |"Optimization Feedback"| Shortfin
    Tuner --> |"Performance Metrics"| Sharktank
    
    %% Click Events
    click DevMgmt "https://github.com/nod-ai/shark-ai/tree/main/shortfin/src/shortfin/local/systems/"
    click ArrayOps "https://github.com/nod-ai/shark-ai/tree/main/shortfin/src/shortfin/array/"
    click ServComp "https://github.com/nod-ai/shark-ai/tree/main/shortfin/python/shortfin_apps/"
    click AsyncRT "https://github.com/nod-ai/shark-ai/blob/main/shortfin/src/shortfin/local/async.h"
    click ModelImpl "https://github.com/nod-ai/shark-ai/tree/main/sharktank/sharktank/models/"
    click QuantTools "https://github.com/nod-ai/shark-ai/blob/main/sharktank/sharktank/types/quantizers.py"
    click DatasetMgmt "https://github.com/nod-ai/shark-ai/tree/main/sharktank/sharktank/transforms/dataset/"
    click ExportTools "https://github.com/nod-ai/shark-ai/blob/main/sharktank/sharktank/export.py"
    click ParamSearch "https://github.com/nod-ai/shark-ai/blob/main/tuner/tuner/libtuner.py"
    click DispConst "https://github.com/nod-ai/shark-ai/blob/main/tuner/tuner/dispatch_constraints.py"
    click CandGen "https://github.com/nod-ai/shark-ai/blob/main/tuner/tuner/candidate_gen.py"
    click PerfTest "https://github.com/nod-ai/shark-ai/blob/main/tuner/examples/test/tuner_test.py"
    click LLM "https://github.com/nod-ai/shark-ai/blob/main/shortfin/python/shortfin_apps/llm/server.py"
    click LLAMA "https://github.com/nod-ai/shark-ai/tree/main/sharktank/sharktank/models/llama/"
    click SDXL "https://github.com/nod-ai/shark-ai/tree/main/sharktank/sharktank/models/vae/"
    click Mixtral "https://github.com/nod-ai/shark-ai/tree/main/sharktank/sharktank/models/mixtral/"
    
    %% Styling
    classDef core fill:#2374ab
    classDef system fill:#48a9a6
    classDef service fill:#4ad7d1
    classDef tool fill:#c17767
    classDef model fill:#8db580
    classDef data fill:#d4afb9
    classDef hardware fill:#d1495b
```
