How cube2222/octosql works
Main System Components
• CLI & Command Handling: Represented by the “cmd” package, which contains the main entry points (root.go and various subcommands for describe, plugin management, etc.). This is the interface through which users interact.
• SQL Parsing & Analysis: The “parser” folder (which includes the SQL parser implementation and supporting analyzer packages) is responsible for interpreting the SQL queries. This is where user queries are tokenized and transformed into abstract syntax trees (ASTs).
• Logical & Physical Planning:
– Logical Plan: The “logical” directory shows how queries get transformed into a logical representation (nodes for join, group by, filter, etc.).
– Optimizer: The “optimizer” package contains modules that perform optimizations like predicate pushdown and removal of unused fields.
– Physical Plan: The “physical” package converts logical plans into executable operator trees.
• Execution Engine: The “execution” folder implements the actual dataflow operations. It handles streams, node operations (such as joins, group by, filtering) and maintains state, watermarks, and retractions for streaming data.
• Data Sources / Connectors:
– The “datasources” package and its subdirectories (e.g., csv, json, parquet, lines, memory) provide implementations for reading and interpreting various file formats.
– The “datasources/plugins” package and the “plugins” folder provide a plugin system for connecting to external databases (like PostgreSQL, MySQL via externally developed plugins).
• Aggregates & Functions: The “aggregates” and “functions” directories contain the implementations of built-in aggregates and functions available for SQL queries.
• Output Format Handling: The “outputs” package defines various output formats (batch, eager, stream, CSV, table, etc.), ensuring that query results are formatted per user specifications.
• Helpers & Utilities: Additional utilities (for logging, graph manipulations, etc.) are provided in directories like “helpers” and “logs”.
• Configuration & Plugin Management: With configuration mechanisms in “config” and files like “plugin_repository.json”, the system supports extensibility and runtime configuration.
Relationships and Interactions
• The CLI layer accepts a SQL query and passes it to the SQL Parser.
• The Parser generates an AST which is then fed into the logical planning stage.
• The Logical Plan (composed of nodes like join, filter, group by) is processed by the Optimizer, which applies transformations such as pushing down filters (see “optimizer/filter_merge.go”, “push_filter_into_lookup_join_branch.go”, etc.).
• Next, the optimized logical plan is converted into a Physical Plan. The Physical node tree (see the “physical” package) defines the concrete execution strategy.
• The Execution Engine then executes the Physical Plan. As part of execution, it interacts with various Data Sources to load data (via the “datasources” packages) and with Plugins to fetch external data if needed.
• During execution, built-in aggregates and functions (from “aggregates” and “functions”) are calculated.
• Finally, the results flow out into the Output subsystem, formatting according to selected output types.
• Throughout this flow, helper and telemetry modules (e.g., “logs”, “telemetry”) capture diagnostic and performance data.
• The plugin system provides external extensibility by dynamically integrating connectors for databases and other services.
Architectural Patterns & Important Technologies
• Modularity: The codebase is clearly split into logical modules: parsing, planning, execution, and output.
• Dataflow & Streaming: With support for watermarks and triggers (as described in the README and in the “execution” package) the architecture is built to handle both bounded (batch) and unbounded (stream) data flows.
• Plugin Architecture: The plugin directories and configuration allow for dynamic extension of the data sources.
• Optimizer Layers: Logical optimizations ensure efficient query executions similar to traditional query processors.