How polydbms/sheetreader-duckdb works
Project Type and Purpose
• Project Type: DuckDB extension
• Purpose: To enable DuckDB to read XLSX files efficiently using the SheetReader parser. The extension wraps sheetreader-core functionalities into a DuckDB function, which users can install and load to query Excel files directly.
File Structure
• Key Top-Level Components:
• Directories such as src, docs, benchmarks, test, and scripts indicate areas for source code, documentation, performance evaluation, testing, and deployment utilities.
• The “src” folder contains the core C++ code (both header files in src/include and source file(s) in src) that implement the extension interface.
• The “duckdb” directory and files like CMakeLists.txt, Makefile, and vcpkg.json suggest that this extension is built along with DuckDB using CMake and supports external dependency management.
• The .devcontainer, .github, and .vscode directories help ensure consistency across development and CI environments, while benchmarks provide performance insight.
Key System Components
a. DuckDB Core:
• DuckDB engine that would load and execute SQL queries.
• Binary shell (./build/release/duckdb) which auto-loads the extension.
b. SheetReader DuckDB Extension:
• Extension library (sheetreader.duckdb_extension) that hooks into DuckDB.
• Integration point where the extension registers the sheetreader() SQL function.
c. SheetReader Core:
• Although not directly embedded in this repo, indicate the dependency on the sheetreader-core (from the external GitHub repository) that handles the actual XLSX parsing.
d. Build and CI Components:
• Brief areas for build scripts (Makefile, CMake instructions) and CI pipelines from the .github/workflows. This is useful to understand automated build and testing flows.
e. Testing & Benchmarking:
• How the tests (in the test directory and benchmark notebooks/data) validate and measure performance of the extension.
Relationships and Interactions
• That user SQL commands in DuckDB trigger a call to the sheetreader() function, which is implemented in the extension.
• Optionally, include interactions with build tools (CMake, Make, ninja) that result in the final binaries.
• Present CI/CD components (GitHub workflows) that automate testing and build for the extension.
Architectural Patterns and Design Principles
• Modular Separation: Clearly separate concerns by differentiating between the database engine (DuckDB), the extension layer (the C++ components exposed in src), and the XLSX parsing logic (SheetReader core).
• Plugin Architecture: Emphasize how the extension plugs into DuckDB as an add-on—this modular design allows easy replacement or extension of core functionality.
• External Integration: Indicate the integration with external technologies (sheetreader-core library, containerized development via .devcontainer, dependency management via vcpkg).
Final Assembly
• At the top, place “User SQL Query” entering the system.
• Branch off from the extension to the SheetReader core, and show the result (data table) returning through the extension back to DuckDB.
• In a side panel or a secondary layer, include the build configuration and CI/CD pipelines, connecting them to the development and build artifacts.
• That all components, relationships, and technology references (e.g., CMake, GitHub workflows, Docker development environment) are clearly annotated.