# hsbc/bqtools architecture

> Architecture diagram of the GitHub repository hsbc/bqtools, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hsbc/bqtools
- Repository: https://github.com/hsbc/bqtools

## Overview

bqtools is a Python utility package for converting JSON structures into BigQuery schemas and supporting BigQuery operations; its bqsync command also copies datasets between environments. The main runtime is concentrated in one large module, so several capabilities share its path. The README documents additional JSON-cleaning, view-management, and schema-patching features, but their internal calls are not shown in the excerpts; those links are therefore omitted. The sync entry point’s construction of the coordinator is visible, but most execution details are not.

Main entry points: [`src/bqtools/__init__.py`](https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py) (Sync Driver), [`src/bqtools/bqsync.py`](https://github.com/hsbc/bqtools/blob/main/src/bqtools/bqsync.py) (bqsync Command)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_json_bq["JSON and BigQuery"]
  node_schema_tools["Schema Generation<br/>[__init__.py]"]
  node_json_cleaning["JSON Cleaning<br/>[__init__.py]"]
  node_bq_helpers["BigQuery Helpers<br/>[__init__.py]"]
end

subgraph group_view_ops["View Management"]
  node_view_templates["View Templates<br/>[bqschema.in]"]
  node_view_tools["View Operations<br/>[__init__.py]"]
end

subgraph group_sync_ops["Dataset Sync"]
  node_bqsync_cli["bqsync Command<br/>[bqsync.py]"]
  node_sync_coordinator["Sync Coordinator<br/>[__init__.py]"]
  node_sync_driver["Sync Driver<br/>[__init__.py]"]
end

node_developer(("Application Developer"))
node_json_data["JSON Data"]
node_bigquery[("BigQuery")]
node_cloud_storage[("Cloud Storage")]

node_developer -->|"uses"| node_schema_tools
node_json_data -->|"provides structure"| node_schema_tools
node_schema_tools -->|"produces schema for"| node_bigquery
node_json_data -->|"provides input"| node_json_cleaning
node_json_cleaning -->|"prepares load data"| node_bigquery
node_developer -->|"uses"| node_bq_helpers
node_developer -->|"configures views"| node_view_tools
node_developer -->|"runs"| node_bqsync_cli
node_bqsync_cli -->|"constructs"| node_sync_coordinator
node_sync_coordinator -.->|"coordinates"| node_sync_driver
node_sync_driver -.->|"copies datasets"| node_bigquery
node_sync_driver -.->|"stages cross-region data"| node_cloud_storage

click node_schema_tools "https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py"
click node_json_cleaning "https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py"
click node_bq_helpers "https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py"
click node_view_templates "https://github.com/hsbc/bqtools/blob/main/src/bqtools/templates/bqschema.in"
click node_view_tools "https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py"
click node_bqsync_cli "https://github.com/hsbc/bqtools/blob/main/src/bqtools/bqsync.py"
click node_sync_coordinator "https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py"
click node_sync_driver "https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_schema_tools,node_json_cleaning,node_bq_helpers toneBlue
class node_view_templates,node_view_tools,node_bigquery,node_cloud_storage toneAmber
class node_bqsync_cli,node_sync_coordinator,node_sync_driver toneMint
class node_developer,node_json_data toneIndigo
```

## Components

### JSON and BigQuery

- [Schema Generation](https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py): `src/bqtools/__init__.py`
- [JSON Cleaning](https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py): `src/bqtools/__init__.py`
- [BigQuery Helpers](https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py): `src/bqtools/__init__.py`

### View Management

- [View Templates](https://github.com/hsbc/bqtools/blob/main/src/bqtools/templates/bqschema.in): `src/bqtools/templates/bqschema.in`
- [View Operations](https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py): `src/bqtools/__init__.py`

### Dataset Sync

- [bqsync Command](https://github.com/hsbc/bqtools/blob/main/src/bqtools/bqsync.py): `src/bqtools/bqsync.py`
- [Sync Coordinator](https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py): `src/bqtools/__init__.py`
- [Sync Driver](https://github.com/hsbc/bqtools/blob/main/src/bqtools/__init__.py): `src/bqtools/__init__.py`

### Other components

- **Application Developer**
- **JSON Data**
- **BigQuery**
- **Cloud Storage**

## Connections

- Application Developer → Schema Generation: uses (evidence: `README.md`)
- JSON Data → Schema Generation: provides structure (evidence: `README.md`)
- Schema Generation → BigQuery: produces schema for (evidence: `README.md`)
- JSON Data → JSON Cleaning: provides input (evidence: `README.md`)
- JSON Cleaning → BigQuery: prepares load data (evidence: `README.md`)
- Application Developer → BigQuery Helpers: uses (evidence: `README.md`)
- Application Developer → View Operations: configures views (evidence: `README.md`)
- Application Developer → bqsync Command: runs (evidence: `README.md`)
- bqsync Command → Sync Coordinator: constructs (evidence: `src/bqtools/bqsync.py`)
- Sync Coordinator → Sync Driver: coordinates (evidence: `src/bqtools/__init__.py`)
- Sync Driver → BigQuery: copies datasets
- Sync Driver → Cloud Storage: stages cross-region data