# googleapis/python-bigquery-storage architecture

> Architecture diagram of the GitHub repository googleapis/python-bigquery-storage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/googleapis/python-bigquery-storage
- Repository: https://github.com/googleapis/python-bigquery-storage

## Overview

##### Main Components
- Client API Layer – This is what developers will interact with directly (e.g., classes in google/cloud/bigquery_storage_v1, v1alpha, v1beta2). Represent the public interface and client objects (e.g., BigQueryStorageClient, Reader, Writer).
- Logging and Configuration – A component to handle logging configuration, as described in the README, and the configuration routines used for secure authentication and API settings.
- Auxiliary and Support Components – Samples, tests (both unit and system) and continuous integration (Kokoro, GitHub Actions) that support the development process; these also validate the core functionality but are not directly part of the runtime client flow. Represent these separately.

##### Relationships and Interactions
- The client API layer interacts directly with the service implementations.
- The service implementations use the transport layer to send gRPC (or async gRPC) calls to the actual BigQuery Storage API endpoints on Google Cloud.
- Data flows from the client (user code) through the client library API, then via the services and transport layers to the BigQuery Storage backend.
- Configuration and logging modules operate in parallel, ensuring that all services and transport layers are properly instrumented.
- The samples and tests simulate client calls, validate service behavior, and help demonstrate proper usage and edge cases.

##### Architectural Patterns and Principles
- Versioning & Modularization: Notice that the project handles multiple API versions (v1, v1alpha, v1beta2) using a clear separation in the directory structure so that each version’s client, services, transports, and types are isolated.
- Abstraction and Encapsulation: The client library cleanly abstracts away the details of the underlying transport and message types—from the public API to the gRPC transports.
- Configuration & Logging: Incorporate how the library manages logging configuration (via both environment variables and code-based configuration) as a cross-cutting concern.

##### Technologies, Frameworks, and Tools
- Python (interpreting “venv” setups, Python 3.7+ support)
- gRPC (for transport layer implementations)
- Async IO (in transports for async client features)
- Standard Python logging

##### Core Elements
- Clear labels for each major component:
- “Client API (Public Interface)” for the front-facing methods.
- “Service Layer” for big_query_read & big_query_write functionalities.
- “Transport Layer (gRPC/Async)” showing the paths used for network communications.
- “Data Types & Schemas” for the types modules.
- “Configuration & Logging” for the cross-cutting concerns.
- Data flow from Client API → Service Layer → Transport Layer → Google Cloud (BigQuery Storage API).
- Feedback flow for error reporting and logging.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Developer Code
    Dev["Developer Code\n(Client Applications)"]:::external

    %% Client API Layer
    subgraph "Client API Layer\n(Public Interface)"
        client_v1["BigQueryStorageClient (v1)"]:::client
        client_v1beta2["BigQueryStorageClient (v1beta2)"]:::client
    end

    %% Service Implementation Layer
    subgraph "Service Implementation Layer"
        service_v1["big_query_read/write (v1)"]:::service
        service_v1alpha["metastore_partition_service (v1alpha)"]:::service
        service_v1beta2["big_query_read/write (v1beta2)"]:::service
    end

    %% Transport Layer
    subgraph "Transport Layer\n(gRPC / Async)"
        transport_v1_read["Transports for big_query_read (v1)"]:::transport
        transport_v1_write["Transports for big_query_write (v1)"]:::transport
        transport_v1alpha["Transports for metastore_partition_service (v1alpha)"]:::transport
        transport_v1beta2_read["Transports for big_query_read (v1beta2)"]:::transport
        transport_v1beta2_write["Transports for big_query_write (v1beta2)"]:::transport
    end

    %% Data Types & Schemas
    subgraph "Data Types & Schemas"
        types_v1["Data Types (v1)"]:::data
        types_v1alpha["Data Types (v1alpha)"]:::data
        types_v1beta2["Data Types (v1beta2)"]:::data
    end

    %% Configuration & Logging
    config_log["Configuration & Logging"]:::config

    %% Auxiliary & Support Components
    subgraph "Auxiliary & Support"
        aux_tests["Tests"]:::aux
        aux_samples["Samples"]:::aux
        aux_github["GitHub Actions CI/CD"]:::aux
        aux_kokoro["Kokoro CI/CD"]:::aux
    end

    %% External BigQuery API
    external_api["Google Cloud BigQuery Storage API"]:::external

    %% Main Flow
    Dev -->|"calls"| client_v1
    Dev -->|"calls"| client_v1beta2

    client_v1 -->|"invokes"| service_v1
    client_v1beta2 -->|"invokes"| service_v1beta2
    %% Optional: if v1alpha is used indirectly, it can be connected via configuration or advanced calls
    client_v1 -.->|"optional"| service_v1alpha

    service_v1 -->|"uses"| transport_v1_read
    service_v1 -->|"uses"| transport_v1_write
    service_v1alpha -->|"uses"| transport_v1alpha
    service_v1beta2 -->|"uses"| transport_v1beta2_read
    service_v1beta2 -->|"uses"| transport_v1beta2_write

    transport_v1_read -->|"communicates with"| external_api
    transport_v1_write -->|"communicates with"| external_api
    transport_v1alpha -->|"communicates with"| external_api
    transport_v1beta2_read -->|"communicates with"| external_api
    transport_v1beta2_write -->|"communicates with"| external_api

    %% Data Types used by Service Layers
    service_v1 ---|"defines"| types_v1
    service_v1alpha ---|"defines"| types_v1alpha
    service_v1beta2 ---|"defines"| types_v1beta2

    %% Configuration & Logging crosscutting
    config_log ---|"instruments"| client_v1
    config_log ---|"instruments"| service_v1
    config_log ---|"instruments"| transport_v1_read
    config_log ---|"instruments"| transport_v1_write
    config_log ---|"instruments"| service_v1beta2
    config_log ---|"instruments"| transport_v1beta2_read
    config_log ---|"instruments"| transport_v1beta2_write

    %% Auxiliary flows (used in testing and demonstrations)
    aux_tests ---|"validates"| service_v1
    aux_tests ---|"validates"| service_v1beta2
    aux_samples ---|"demonstrates"| client_v1
    aux_samples ---|"demonstrates"| client_v1beta2
    aux_github ---|"runs CI/CD for"| service_v1
    aux_kokoro ---|"runs CI/CD for"| service_v1alpha

    %% Legend (as a subgraph, not connected)
    subgraph Legend [Legend]
      LEG1[":::client - Client API Layer"]
      LEG2[":::service - Service Implementation"]
      LEG3[":::transport - Transport Layer"]
      LEG4[":::data - Data Types & Schemas"]
      LEG5[":::config - Configuration & Logging"]
      LEG6[":::aux - Auxiliary & Support"]
      LEG7[":::external - External/Developer/BigQuery API"]
    end

    %% Click Events for Component Mapping
    click client_v1 "https://github.com/googleapis/python-bigquery-storage/blob/main/google/cloud/bigquery_storage_v1/client.py"
    click client_v1beta2 "https://github.com/googleapis/python-bigquery-storage/blob/main/google/cloud/bigquery_storage_v1beta2/client.py"
    click service_v1 "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1/services/"
    click service_v1alpha "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1alpha/services/metastore_partition_service"
    click service_v1beta2 "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1beta2/services/"
    click transport_v1_read "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1/services/big_query_read/transports/"
    click transport_v1_write "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1/services/big_query_write/transports/"
    click transport_v1beta2_read "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1beta2/services/big_query_read/transports/"
    click transport_v1beta2_write "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1beta2/services/big_query_write/transports/"
    click transport_v1alpha "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1alpha/services/metastore_partition_service/transports/"
    click types_v1 "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1/types/"
    click types_v1alpha "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1alpha/types/"
    click types_v1beta2 "https://github.com/googleapis/python-bigquery-storage/tree/main/google/cloud/bigquery_storage_v1beta2/types/"
    click config_log "https://github.com/googleapis/python-bigquery-storage/blob/main/setup.cfg"
    click aux_tests "https://github.com/googleapis/python-bigquery-storage/tree/main/tests/"
    click aux_samples "https://github.com/googleapis/python-bigquery-storage/tree/main/samples/"
    click aux_github "https://github.com/googleapis/python-bigquery-storage/tree/main/.github/workflows/"
    click aux_kokoro "https://github.com/googleapis/python-bigquery-storage/tree/main/.kokoro/"

    %% Class Styles
    classDef client fill:#fef3bd,stroke:#d4a017,stroke-width:2px;
    classDef service fill:#d0e6a5,stroke:#70a800,stroke-width:2px;
    classDef transport fill:#f9c0c0,stroke:#d94e4e,stroke-width:2px;
    classDef data fill:#c0e0f9,stroke:#4078c0,stroke-width:2px;
    classDef config fill:#e3c0f9,stroke:#a040c0,stroke-width:2px;
    classDef aux fill:#c9f9c0,stroke:#70c045,stroke-width:2px;
    classDef external fill:#e0e0e0,stroke:#8d8d8d,stroke-width:2px;
```
