# turbolytics/sql-flow architecture

> Architecture diagram of the GitHub repository turbolytics/sql-flow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/turbolytics/sql-flow
- Repository: https://github.com/turbolytics/sql-flow

## Overview

##### Main Components

a. Input Sources Layer:
- Kafka Consumer (implemented)
- WebSocket Input (implemented, e.g., Bluesky firehose)
- HTTP Webhooks (planned)

b. Core Processing Layer:
- DuckDB Engine Integration
- SQL Processing Pipeline
- Window Manager (for tumbling window operations)
- Static Data Manager (for CSV joins)

c. Output Sinks Layer:
- Kafka Producer (implemented)
- Stdout (implemented)
- Future sinks: Postgres, Local Disk, S3 (planned)

##### Key Architectural Elements

a. Pipeline Architecture:
**Source → Handler (SQL Transform) → Sink**

b. Data Flow Management:
- Serialization/Deserialization (JSON, future Parquet)
- Data persistence strategies (Memory/Disk)
- Window-based processing

##### Important Relationships to Highlight

- Data flow paths from sources through processing to sinks
- Persistence options (memory vs. disk)
- Integration points with external systems (Kafka, Bluesky)

##### Technical Details

- Configuration management system
- Serialization/deserialization points
- Window management for aggregations
- Static data join capabilities
- Consumer group management for Kafka

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles
    classDef source fill:#3498db,stroke:#2980b9,color:white
    classDef sink fill:#e74c3c,stroke:#c0392b,color:white
    classDef core fill:#2ecc71,stroke:#27ae60,color:white
    classDef config fill:#9b59b6,stroke:#8e44ad,color:white
    classDef manager fill:#f1c40f,stroke:#f39c12,color:black
    classDef planned fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef benchmark fill:#e67e22,stroke:#d35400,color:white

    %% Input Sources Layer
    subgraph Sources
        KafkaIn["Kafka Consumer"]:::source
        WebSocket["WebSocket Input"]:::source
        HTTPHooks["HTTP Webhooks"]:::planned
    end

    %% Core Processing Layer
    subgraph Core
        DuckDB["DuckDB Engine"]:::core
        SQLPipe["SQL Pipeline"]:::core
        WindowMgr["Window Manager"]:::manager
        StaticData["Static Data Manager"]:::manager
        SerDe["Serialization/Deserialization"]:::core
    end

    %% Configuration and Management
    subgraph Config
        ConfigMgr["Configuration Manager"]:::config
        Settings["System Settings"]:::config
        Lifecycle["Lifecycle Manager"]:::config
    end

    %% Output Sinks Layer
    subgraph Sinks
        KafkaOut["Kafka Producer"]:::sink
        Stdout["Stdout"]:::sink
        Postgres["Postgres"]:::planned
        LocalDisk["Local Disk"]:::planned
        S3["S3"]:::planned
    end

    %% Benchmark Components
    subgraph Benchmarks
        WindowBench["Window Operations"]:::benchmark
        CSVBench["CSV Join"]:::benchmark
        AggBench["Simple Aggregation"]:::benchmark
    end

    %% Relationships
    KafkaIn --> SerDe
    WebSocket --> SerDe
    HTTPHooks -.-> SerDe
    SerDe --> SQLPipe
    SQLPipe --> DuckDB
    DuckDB --> WindowMgr
    DuckDB --> StaticData
    WindowMgr --> SerDe
    StaticData --> SerDe
    SerDe --> KafkaOut
    SerDe --> Stdout
    SerDe -.-> Postgres
    SerDe -.-> LocalDisk
    SerDe -.-> S3
    ConfigMgr --> Sources
    ConfigMgr --> Core
    ConfigMgr --> Sinks
    Settings --> ConfigMgr
    Lifecycle --> Sources
    Lifecycle --> Sinks

    %% Click Events
    click KafkaIn "https://github.com/turbolytics/sql-flow/blob/main/sqlflow/sources.py"
    click WebSocket "https://github.com/turbolytics/sql-flow/blob/main/sqlflow/sources.py"
    click KafkaOut "https://github.com/turbolytics/sql-flow/blob/main/sqlflow/sinks.py"
    click Stdout "https://github.com/turbolytics/sql-flow/blob/main/sqlflow/sinks.py"
    click WindowMgr "https://github.com/turbolytics/sql-flow/blob/main/sqlflow/managers/window.py"
    click SQLPipe "https://github.com/turbolytics/sql-flow/blob/main/sqlflow/sql.py"
    click ConfigMgr "https://github.com/turbolytics/sql-flow/blob/main/sqlflow/config.py"
    click Settings "https://github.com/turbolytics/sql-flow/blob/main/sqlflow/settings.py"
    click SerDe "https://github.com/turbolytics/sql-flow/blob/main/sqlflow/serde.py"
    click Lifecycle "https://github.com/turbolytics/sql-flow/blob/main/sqlflow/lifecycle.py"
    click WindowBench "https://github.com/turbolytics/sql-flow/blob/main/benchmark/tumbling-window.sh"
    click CSVBench "https://github.com/turbolytics/sql-flow/blob/main/benchmark/csv-disk-join.sh"
    click AggBench "https://github.com/turbolytics/sql-flow/blob/main/benchmark/simple-agg-mem.sh"

    %% Legend
    subgraph Legend
        L1["Source Component"]:::source
        L2["Sink Component"]:::sink
        L3["Core Component"]:::core
        L4["Manager Component"]:::manager
        L5["Config Component"]:::config
        L6["Planned Feature"]:::planned
        L7["Benchmark"]:::benchmark
    end
```
