How streamingfast/substreams-sink-sql works
Main Components Identification
a. Input Layer:
• Substreams Module (WASM)
• Authentication Token Handler
• Network Endpoint Handler
b. Core Processing:
• Command Layer (cmd/substreams-sink-sql)
• Bundler Component
• Sinker Component
• State Management
c. Database Layer:
• Database Dialect System
• PostgreSQL Support
• ClickHouse Support
d. CSV Processing Pipeline (for high throughput):
• CSV Generator
• CSV Injector
[Input Layer]
• ▼
[Command Layer]
• ┬─────────────────┐
• ▼ ▼ ▼
[Bundler] [State Manager] [CSV Pipeline]
• ┬───────┘ │
• ▼ │
• [Sink Layer] │
• ┬──────────┘
• ▼
• [Database Layer]
Component Details
For each component, include these specific elements:
a. Input Layer:
• Substreams WASM module input
• Authentication token flow
• Network endpoint configuration
• Protocol buffer definitions (from proto/ directory)
b. Command Layer:
• Main commands: setup, run, generate-csv, inject-csv
• Configuration flow
• DSN parsing and validation
c. Processing Components:
• Bundler: Show data bundling and encoding process
• State Manager: Display cursor management and state tracking
• Sinker: Show database change processing
• CSV Pipeline: Show the alternative high-throughput path
d. Database Layer:
• Dialect abstraction
• Supported databases (PostgreSQL, ClickHouse)
• Connection management
• Schema management
Additional Details
Important aspects:
• Version information
• Configuration file references
• Schema file locations
• Protocol buffer message types
• Database connection parameters
• Buffer size configurations for CSV processing
Specific Implementation Notes
• The modular nature of the database dialect system
• The dual-path processing (direct vs. CSV-based)
• The cursor-based state management
• The configuration flexibility through DSN
• The protocol buffer integration