How streamingfast/substreams works
Main Components
a) Client Interface Layer
• CLI Interface (under /cmd/substreams)
• Client Libraries (under /client)
• TUI Components (under /tui and /tui2)
b) Core Processing Layer
• Pipeline Processing (/pipeline)
• Orchestration Engine (/orchestrator)
• WASM Runtime (/wasm)
• Storage System (/storage)
c) Data Processing Components
• Block Processing
• Module Execution
• Store Management
• Index Management
Key Relationships to Highlight
Client Layer → Core Processing Layer
• ↓
Pipeline Processor → Orchestrator
• ↓
WASM Runtime ← → Storage System
• ↓
Block Processing → Module Execution → Store Management
Specific Component Details
• Request Planning
• Block Processing
• Module Execution
• Store Management
• Cache Management
b) Storage Components (use database symbols):
• Store Storage
• Index Storage
• Execution Output Storage
c) Runtime Components (use compute symbols):
• WASM Module Execution
• Module Registry
• State Management
Integration Points
External integration points with:
• Blockchain Data Sources
• The Graph Network
• External Consumers
• Sink Services (/sink-server)
Technical Details
• Protocol Buffer Definitions (/pb)
• Manifest Handling (/manifest)
• Module Execution Pipeline
• Store Operations
Implementation Notes
Technical annotations for:
• WASM Runtime Options (Wasmtime/Wazero)
• Storage Implementations
• Pipeline Configurations
• Module Types (Map/Store)
Storage and caching mechanisms
Remember to:
• Keep components grouped logically
• Clear boundaries between subsystems
• Version/date information
• Brief technical notes where relevant
This will create a comprehensive system design that accurately represents Substreams' architecture while maintaining clarity and usefulness for technical discussions.