How hsbc/bqtools works
bqtools is a Python utility package for converting JSON structures into BigQuery schemas and supporting BigQuery operations; its bqsync command also copies datasets between environments. The main runtime is concentrated in one large module, so several capabilities share its path. The README documents additional JSON-cleaning, view-management, and schema-patching features, but their internal calls are not shown in the excerpts; those links are therefore omitted. The sync entry point’s construction of the coordinator is visible, but most execution details are not.
Main entry points: __init__.py Sync Driver · bqsync.py bqsync Command
Components 12
JSON and BigQuery
- Schema Generation
src/bqtools/__init__.py - JSON Cleaning
src/bqtools/__init__.py - BigQuery Helpers
src/bqtools/__init__.py
View Management
- View Templates
src/bqtools/templates/bqschema.in - View Operations
src/bqtools/__init__.py
Dataset Sync
- bqsync Command
src/bqtools/bqsync.py - Sync Coordinator
src/bqtools/__init__.py - Sync Driver
src/bqtools/__init__.py
Other components
- Application Developer
- JSON Data
- BigQuery
- Cloud Storage
Connections 12
- Application Developer → Schema Generation: uses
README.md - JSON Data → Schema Generation: provides structure
README.md - Schema Generation → BigQuery: produces schema for
README.md - JSON Data → JSON Cleaning: provides input
README.md - JSON Cleaning → BigQuery: prepares load data
README.md - Application Developer → BigQuery Helpers: uses
README.md - Application Developer → View Operations: configures views
README.md - Application Developer → bqsync Command: runs
README.md - bqsync Command → Sync Coordinator: constructs
src/bqtools/bqsync.py - Sync Coordinator → Sync Driver: coordinates
src/bqtools/__init__.py - Sync Driver → BigQuery: copies datasets
- Sync Driver → Cloud Storage: stages cross-region data