How goccy/bigquery-emulator works
Project Type and Purpose
• This is an open-source emulator for Google’s BigQuery implemented in Go.
• Its purpose is to mimic BigQuery’s REST and gRPC endpoints so that both Go and non-Go BigQuery clients (like the bq command-line tool and Python clients) can test and develop against a local instance.
• The README confirms support for key BigQuery APIs, including SQL processing using ZetaSQL, storage via SQLite (including special treatment for types), and gRPC-based BigQuery Storage API implementations.
Analysis of the File Structure
• Top-level directories and files (e.g., Dockerfile, Makefile, README.md) indicate that this project is designed for both local and containerized deployments.
• The “cmd” directory contains the CLI for launching the BigQuery emulator (i.e., the “bigquery-emulator” binary).
• The “internal” directory includes various sub-packages:
• “connection” (managing client connections),
• “contentdata” (handling content/data repositories),
• “metadata” (managing projects, datasets, jobs, tables, etc.),
• “cmd/generator” (likely used for generating code or configuration based on API definitions),
• “logger” (for centralized logging).
• The “server” directory contains the core server logic – handlers, middleware, test servers, and configuration for both REST and gRPC endpoints.
Key Components and Their Roles
• CLI (cmd/bigquery-emulator): The entry point that starts the emulator.
• REST Server & gRPC Server (server/*):
• REST server listens on port 9050 for standard BigQuery API calls.
• gRPC server listens on port 9060 for BigQuery Storage API calls.
• Query Handling Flow:
• Incoming queries (via REST or gRPC) are handled by server handlers.
• These handlers pass query requests to the internal components that integrate with go-zetasqlite. The process converts ZetaSQL queries to SQLite queries.
• SQLite (accessed through go-sqlite3 and custom decoding functions) serves as the data storage, supporting both in-memory and file-based modes.
• Metadata/Repository Management (internal/metadata and internal/contentdata): These manage the BigQuery-specific metadata (projects, datasets, tables, jobs) and help emulate the stateful aspects of BigQuery.
• Logger and Utilities (internal/logger and server/util.go): Provide logging and various utility functions across the emulator system.
• Examples & Testing: The “_examples” and “server/testdata” directories offer usage demonstrations and seed data for startup.
Relationships and Interactions
• External Clients (bq CLI, Python, Go SDK): Connect to the REST or gRPC endpoints.
• The server acts as an intermediary between the client and the underlying SQLite database.
• Internal packages translate BigQuery-specific constructs to SQLite queries, taking care of type encoding/decoding (especially for non-native types like ARRAY or STRUCT).
• The CLI orchestrates launching the server with proper configuration (project, dataset, ports, etc.) and may leverage the “internal/cmd/generator” for code or config generation.
Architectural Patterns and Principles
• Separation of concerns is evident in the modular structure:
• Command-line interface (cmd), core business logic (internal), server APIs (server), and shared types (types).
• The design supports both in-process integration (using httptest) and standalone server deployment.
• The emulator leverages dependency injection (e.g., passing configuration, logger, and storage mechanisms), allowing for configurability between in-memory and file-based storage.