# goccy/bigquery-emulator architecture

> Architecture diagram of the GitHub repository goccy/bigquery-emulator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/goccy/bigquery-emulator
- Repository: https://github.com/goccy/bigquery-emulator

## Overview

##### Project Type and Purpose
- This is an open-source emulator for Google’s BigQuery implemented in Go.
- Its purpose is to mimic BigQuery’s REST and gRPC endpoints so that both Go and non-Go BigQuery clients (like the bq command-line tool and Python clients) can test and develop against a local instance.
- The README confirms support for key BigQuery APIs, including SQL processing using ZetaSQL, storage via SQLite (including special treatment for types), and gRPC-based BigQuery Storage API implementations.

##### Analysis of the File Structure
- Top-level directories and files (e.g., Dockerfile, Makefile, README.md) indicate that this project is designed for both local and containerized deployments.
- The “cmd” directory contains the CLI for launching the BigQuery emulator (i.e., the “bigquery-emulator” binary).
- The “internal” directory includes various sub-packages:
- “connection” (managing client connections),
- “contentdata” (handling content/data repositories),
- “metadata” (managing projects, datasets, jobs, tables, etc.),
- “cmd/generator” (likely used for generating code or configuration based on API definitions),
- “logger” (for centralized logging).
- The “server” directory contains the core server logic – handlers, middleware, test servers, and configuration for both REST and gRPC endpoints.

##### Key Components and Their Roles
- CLI (cmd/bigquery-emulator): The entry point that starts the emulator.
- REST Server & gRPC Server (server/*):
- REST server listens on port 9050 for standard BigQuery API calls.
- gRPC server listens on port 9060 for BigQuery Storage API calls.
- Query Handling Flow:
- Incoming queries (via REST or gRPC) are handled by server handlers.
- These handlers pass query requests to the internal components that integrate with go-zetasqlite. The process converts ZetaSQL queries to SQLite queries.
- SQLite (accessed through go-sqlite3 and custom decoding functions) serves as the data storage, supporting both in-memory and file-based modes.
- Metadata/Repository Management (internal/metadata and internal/contentdata): These manage the BigQuery-specific metadata (projects, datasets, tables, jobs) and help emulate the stateful aspects of BigQuery.
- Logger and Utilities (internal/logger and server/util.go): Provide logging and various utility functions across the emulator system.
- Examples & Testing: The “_examples” and “server/testdata” directories offer usage demonstrations and seed data for startup.

##### Relationships and Interactions
- External Clients (bq CLI, Python, Go SDK): Connect to the REST or gRPC endpoints.
- The server acts as an intermediary between the client and the underlying SQLite database.
- Internal packages translate BigQuery-specific constructs to SQLite queries, taking care of type encoding/decoding (especially for non-native types like ARRAY or STRUCT).
- The CLI orchestrates launching the server with proper configuration (project, dataset, ports, etc.) and may leverage the “internal/cmd/generator” for code or config generation.

##### Architectural Patterns and Principles
- Separation of concerns is evident in the modular structure:
- Command-line interface (cmd), core business logic (internal), server APIs (server), and shared types (types).
- The design supports both in-process integration (using httptest) and standalone server deployment.
- The emulator leverages dependency injection (e.g., passing configuration, logger, and storage mechanisms), allowing for configurability between in-memory and file-based storage.

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Layer
    subgraph "External Layer"
        A["External Clients"]:::external
    end

    %% Entry & Server Layer
    subgraph "Entry & Server Layer"
        B["CLI Entry Point"]:::entry
        subgraph "API Endpoints"
            C["REST Server (HTTP:9050)"]:::server
            D["gRPC Server (gRPC:9060)"]:::server
        end
        I["Connection Manager"]:::support
        J["Logger & Utilities"]:::support
        K["Code Generator"]:::support
    end

    %% Processing Layer
    subgraph "Processing Layer"
        E["Query Engine"]:::processing
    end

    %% Data & Metadata Layer
    subgraph "Data & Metadata Layer"
        F["SQLite Storage"]:::data
        G["Metadata Manager"]:::data
        H["Content Data Repository"]:::data
        L["Shared Types"]:::data
    end

    %% Testing & Examples
    subgraph "Testing & Examples"
        M["Examples & Testing"]:::test
    end

    %% Connections
    A -->|"connects"| C
    A -->|"connects"| D

    B -->|"starts"| C
    B -->|"starts"| D
    B -->|"invokes"| K

    C -->|"handles"| E
    D -->|"handles"| E

    C -->|"uses"| I
    D -->|"uses"| I

    E -->|"queries"| F
    E -->|"logs to"| J
    C -->|"logs from"| J
    D -->|"logs from"| J

    E -->|"integrates"| G
    G -->|"syncs with"| E

    E -->|"accesses"| H

    E -->|"utilizes"| L
    G -->|"defines"| L
    H -->|"defines"| L

    %% Examples influence
    M -->|"tests"| B
    M -->|"validates"| E

    %% Click Events
    click B "https://github.com/goccy/bigquery-emulator/blob/main/cmd/bigquery-emulator/main.go"
    click C "https://github.com/goccy/bigquery-emulator/blob/main/server/handler.go"
    click C "https://github.com/goccy/bigquery-emulator/blob/main/server/server.go"
    click C "https://github.com/goccy/bigquery-emulator/blob/main/server/middleware.go"
    click D "https://github.com/goccy/bigquery-emulator/blob/main/server/storage_handler.go"
    click D "https://github.com/goccy/bigquery-emulator/blob/main/server/storage.go"
    click E "https://github.com/goccy/bigquery-emulator/blob/main/server/handler_gen.go"
    click E "https://github.com/goccy/bigquery-emulator/blob/main/server/source.go"
    click G "https://github.com/goccy/bigquery-emulator/blob/main/internal/metadata/repository.go"
    click H "https://github.com/goccy/bigquery-emulator/blob/main/internal/contentdata/repository.go"
    click I "https://github.com/goccy/bigquery-emulator/blob/main/internal/connection/manager.go"
    click J "https://github.com/goccy/bigquery-emulator/blob/main/internal/logger/logger.go"
    click J "https://github.com/goccy/bigquery-emulator/blob/main/server/util.go"
    click K "https://github.com/goccy/bigquery-emulator/blob/main/internal/cmd/generator/main.go"
    click K "https://github.com/goccy/bigquery-emulator/blob/main/internal/cmd/generator/resources/bigquery-api.json"
    click L "https://github.com/goccy/bigquery-emulator/blob/main/types/types.go"
    click L "https://github.com/goccy/bigquery-emulator/blob/main/internal/types/types.go"
    click L "https://github.com/goccy/bigquery-emulator/blob/main/types/arrow.go"
    click L "https://github.com/goccy/bigquery-emulator/blob/main/types/avro.go"
    click L "https://github.com/goccy/bigquery-emulator/blob/main/types/storagepb.go"
    click L "https://github.com/goccy/bigquery-emulator/blob/main/types/validation.go"
    click M "https://github.com/goccy/bigquery-emulator/tree/main/_examples/python"
    click M "https://github.com/goccy/bigquery-emulator/blob/main/server/testdata/data.yaml"

    %% Styles
    classDef external fill:#a2d9ff,stroke:#333,stroke-width:2px;
    classDef entry fill:#f9e79f,stroke:#333,stroke-width:2px;
    classDef server fill:#aed6f1,stroke:#333,stroke-width:2px;
    classDef processing fill:#f5b7b1,stroke:#333,stroke-width:2px;
    classDef data fill:#abebc6,stroke:#333,stroke-width:2px;
    classDef support fill:#d7bde2,stroke:#333,stroke-width:2px;
    classDef test fill:#a3e4d7,stroke:#333,stroke-width:2px;
```
