How turbolytics/sql-flow works
Main Components
a. Input Sources Layer:
• Kafka Consumer (implemented)
• WebSocket Input (implemented, e.g., Bluesky firehose)
• HTTP Webhooks (planned)
b. Core Processing Layer:
• DuckDB Engine Integration
• SQL Processing Pipeline
• Window Manager (for tumbling window operations)
• Static Data Manager (for CSV joins)
c. Output Sinks Layer:
• Kafka Producer (implemented)
• Stdout (implemented)
• Future sinks: Postgres, Local Disk, S3 (planned)
Key Architectural Elements
a. Pipeline Architecture:
Source → Handler (SQL Transform) → Sink
b. Data Flow Management:
• Serialization/Deserialization (JSON, future Parquet)
• Data persistence strategies (Memory/Disk)
• Window-based processing
Important Relationships to Highlight
• Data flow paths from sources through processing to sinks
• Persistence options (memory vs. disk)
• Integration points with external systems (Kafka, Bluesky)
Technical Details
• Configuration management system
• Serialization/deserialization points
• Window management for aggregations
• Static data join capabilities
• Consumer group management for Kafka