# cube2222/octosql architecture

> Architecture diagram of the GitHub repository cube2222/octosql, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cube2222/octosql
- Repository: https://github.com/cube2222/octosql

## Overview

##### Main System Components
- CLI & Command Handling: Represented by the “cmd” package, which contains the main entry points (root.go and various subcommands for describe, plugin management, etc.). This is the interface through which users interact.
- SQL Parsing & Analysis: The “parser” folder (which includes the SQL parser implementation and supporting analyzer packages) is responsible for interpreting the SQL queries. This is where user queries are tokenized and transformed into abstract syntax trees (ASTs).
- Logical & Physical Planning:
– Logical Plan: The “logical” directory shows how queries get transformed into a logical representation (nodes for join, group by, filter, etc.).
– Optimizer: The “optimizer” package contains modules that perform optimizations like predicate pushdown and removal of unused fields.
– Physical Plan: The “physical” package converts logical plans into executable operator trees.
- Execution Engine: The “execution” folder implements the actual dataflow operations. It handles streams, node operations (such as joins, group by, filtering) and maintains state, watermarks, and retractions for streaming data.
- Data Sources / Connectors:
– The “datasources” package and its subdirectories (e.g., csv, json, parquet, lines, memory) provide implementations for reading and interpreting various file formats.
– The “datasources/plugins” package and the “plugins” folder provide a plugin system for connecting to external databases (like PostgreSQL, MySQL via externally developed plugins).
- Aggregates & Functions: The “aggregates” and “functions” directories contain the implementations of built-in aggregates and functions available for SQL queries.
- Output Format Handling: The “outputs” package defines various output formats (batch, eager, stream, CSV, table, etc.), ensuring that query results are formatted per user specifications.
- Helpers & Utilities: Additional utilities (for logging, graph manipulations, etc.) are provided in directories like “helpers” and “logs”.
- Configuration & Plugin Management: With configuration mechanisms in “config” and files like “plugin_repository.json”, the system supports extensibility and runtime configuration.

##### Relationships and Interactions
- The CLI layer accepts a SQL query and passes it to the SQL Parser.
- The Parser generates an AST which is then fed into the logical planning stage.
- The Logical Plan (composed of nodes like join, filter, group by) is processed by the Optimizer, which applies transformations such as pushing down filters (see “optimizer/filter_merge.go”, “push_filter_into_lookup_join_branch.go”, etc.).
- Next, the optimized logical plan is converted into a Physical Plan. The Physical node tree (see the “physical” package) defines the concrete execution strategy.
- The Execution Engine then executes the Physical Plan. As part of execution, it interacts with various Data Sources to load data (via the “datasources” packages) and with Plugins to fetch external data if needed.
- During execution, built-in aggregates and functions (from “aggregates” and “functions”) are calculated.
- Finally, the results flow out into the Output subsystem, formatting according to selected output types.
- Throughout this flow, helper and telemetry modules (e.g., “logs”, “telemetry”) capture diagnostic and performance data.
- The plugin system provides external extensibility by dynamically integrating connectors for databases and other services.

##### Architectural Patterns & Important Technologies
- Modularity: The codebase is clearly split into logical modules: parsing, planning, execution, and output.
- Dataflow & Streaming: With support for watermarks and triggers (as described in the README and in the “execution” package) the architecture is built to handle both bounded (batch) and unbounded (stream) data flows.
- Plugin Architecture: The plugin directories and configuration allow for dynamic extension of the data sources.
- Optimizer Layers: Logical optimizations ensure efficient query executions similar to traditional query processors.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Main Nodes
    CLI["CLI / Command Manager"]:::cli
    SQLParser["SQL Parser & Analyzer"]:::parser
    LogicalPlan["Logical Planner"]:::logical
    QueryOpt["Query Optimizer"]:::optimizer
    PhysicalPlan["Physical Planner"]:::physical
    ExecEng["Execution Engine"]:::execution
    DataSources["Data Sources / Connectors"]:::datasource
    PluginSys["Plugin System"]:::plugin
    AggFunc["Aggregates & Functions Library"]:::aggregates
    Outputs["Output Format Handling"]:::outputs
    TVFs["Table Valued Functions"]:::tvfs
    Helpers["Helpers & Utilities"]:::helpers
    Config["Configuration & Plugin Management"]:::config
    Telemetry["Telemetry & Monitoring"]:::telemetry

    %% Subgraphs
    subgraph "Query Processing Layer"
        SQLParser
        LogicalPlan
        QueryOpt
        PhysicalPlan
        ExecEng
    end

    subgraph "Data & Extension Layer"
        DataSources
        PluginSys
        AggFunc
        Outputs
        TVFs
    end

    subgraph "Supporting & Monitoring"
        Helpers
        Config
        Telemetry
    end

    %% Connections and Data Flow
    CLI -->|"triggers"| SQLParser
    SQLParser -->|"generatesAST"| LogicalPlan
    LogicalPlan -->|"transforms"| QueryOpt
    QueryOpt -->|"optimizes"| PhysicalPlan
    PhysicalPlan -->|"createsPlan"| ExecEng
    ExecEng -->|"fetchesData"| DataSources
    ExecEng -->|"invokes"| PluginSys
    ExecEng -->|"computes"| AggFunc
    ExecEng -->|"formats"| Outputs
    ExecEng -->|"appliesTVF"| TVFs
    ExecEng -->|"logsAndMonitors"| Telemetry
    CLI -->|"readsSettings"| Config
    Config -->|"configures"| PluginSys
    Helpers -->|"supports"| QueryOpt
    Helpers -->|"supports"| ExecEng
    Telemetry -->|"feedsBack"| Helpers

    %% Click Events for Component Mapping
    click CLI "https://github.com/cube2222/octosql/tree/main/cmd/"
    click SQLParser "https://github.com/cube2222/octosql/tree/main/parser/"
    click LogicalPlan "https://github.com/cube2222/octosql/tree/main/logical/"
    click QueryOpt "https://github.com/cube2222/octosql/tree/main/optimizer/"
    click PhysicalPlan "https://github.com/cube2222/octosql/tree/main/physical/"
    click ExecEng "https://github.com/cube2222/octosql/tree/main/execution/"
    click DataSources "https://github.com/cube2222/octosql/tree/main/datasources/"
    click PluginSys "https://github.com/cube2222/octosql/tree/main/datasources/plugins/"
    click PluginSys "https://github.com/cube2222/octosql/tree/main/plugins/"
    click AggFunc "https://github.com/cube2222/octosql/tree/main/aggregates/"
    click AggFunc "https://github.com/cube2222/octosql/tree/main/functions/"
    click Outputs "https://github.com/cube2222/octosql/tree/main/outputs/"
    click Helpers "https://github.com/cube2222/octosql/tree/main/helpers/"
    click Helpers "https://github.com/cube2222/octosql/tree/main/logs/"
    click Config "https://github.com/cube2222/octosql/tree/main/config/"
    click Config "https://github.com/cube2222/octosql/blob/main/plugin_repository.json"
    click TVFs "https://github.com/cube2222/octosql/tree/main/table_valued_functions/"
    click Telemetry "https://github.com/cube2222/octosql/tree/main/telemetry/"

    %% Style Definitions
    classDef cli fill:#f9c,stroke:#333,stroke-width:2px;
    classDef parser fill:#ccf,stroke:#333,stroke-width:2px;
    classDef logical fill:#cfc,stroke:#333,stroke-width:2px;
    classDef optimizer fill:#fc9,stroke:#333,stroke-width:2px;
    classDef physical fill:#9cf,stroke:#333,stroke-width:2px;
    classDef execution fill:#cff,stroke:#333,stroke-width:2px;
    classDef datasource fill:#ffc,stroke:#333,stroke-width:2px;
    classDef plugin fill:#c9f,stroke:#333,stroke-width:2px;
    classDef aggregates fill:#fcf,stroke:#333,stroke-width:2px;
    classDef outputs fill:#cff,stroke:#333,stroke-width:2px;
    classDef tvfs fill:#efe,stroke:#333,stroke-width:2px;
    classDef helpers fill:#fdd,stroke:#333,stroke-width:2px;
    classDef config fill:#ffd,stroke:#333,stroke-width:2px;
    classDef telemetry fill:#dfd,stroke:#333,stroke-width:2px;
```
