How nod-ai/shark-ai works
Main Components Overview
a. Shortfin (Inference & Serving)
• High-performance inference library
• Serving engine components
• Device management (AMD GPU, CPU)
• Array operations and storage
b. Sharktank (Model Management)
• Model recipes and conversion tools
• Quantization support
• Model implementations (SDXL, LLaMA, etc.)
• Dataset management
c. Tuner (Performance Optimization)
• Parameter configuration search
• Dispatch constraints
• Candidate generation
• Performance testing
Key Relationships to Highlight
• Data flow from Sharktank's model recipes to Shortfin's serving components
• Tuner's optimization feedback loop with both Sharktank and Shortfin
• Integration points between components
Component Details
For Shortfin:
[Shortfin Core]
• Device Management
• AMD GPU System
• CPU System
• Array Operations
• Serving Components
• LLM Service
• Stable Diffusion Service
• Async Runtime
For Sharktank:
[Sharktank Core]
• Model Implementations
• LLaMA
• SDXL
• Other Models
• Quantization Tools
• Dataset Management
• Export Tools
For Tuner:
[Tuner Core]
• Parameter Search
• Dispatch Constraints
• Candidate Generation
• Performance Testing
Integration Points
Highlight:
• Python API interfaces
• Model serving endpoints
• Inter-component communication paths
• External system interfaces (ROCm, IREE)
Deployment Flow
Progression:
Model Definition (Sharktank)
• ↓
Optimization (Tuner)
• ↓
Deployment (Shortfin)
• ↓
Serving (Applications)
Additional Elements
Include:
• Version compatibility information
• System requirements
• Key dependencies (ROCm, Python, Linux)