How crate/cratedb-toolkit works
1. Overall shape
This repository is a Python toolkit, not a single app. The top-level package cratedb_toolkit is a collection of domain modules for moving data into/out of CrateDB, inspecting and managing CrateDB/CrateDB Cloud, and providing helper CLIs, HTTP services, and library APIs. The root ctk CLI is the main user entrypoint.
2. Primary architecture style
The code is organized as a layered toolkit:
- thin command/CLI wrappers in */cli.py
- domain orchestration modules in feature-specific packages
- shared utility modules under util/
- model and configuration objects near each subsystem
This makes most features reusable both as library calls and as standalone commands.
3. Core I/O subsystem
The largest area is cratedb_toolkit/io, which acts as a routing hub for ingestion/export workflows. It contains connectors and adapters for sources and targets such as MongoDB, DynamoDB, Kinesis, InfluxDB, CrateDB, Delta Lake, Iceberg, and an “ingestr” integration. io/router.py likely dispatches to the right backend based on source/target type, while module-specific api.py, adapter.py, copy.py, relay.py, and extract.py files implement the actual data movement steps.
4. CrateDB-specific data paths
Within io/cratedb and io/cratedb/cloud, the toolkit handles direct CrateDB bulk loading and Cloud-specific metadata or connection handling. These modules are the sink side of many flows, receiving transformed records from source adapters and pushing them into CrateDB through bulk or cloud-aware APIs.
5. MongoDB migration and CDC pipeline
io/mongodb is a full subsystem with extract, transform, translate, export, copy, and cdc modules. This suggests two main flows: batch migration from MongoDB to CrateDB, and change-data-capture replication. The presence of tests with BSON/NDJSON fixtures and a dedicated CLI indicates this is one of the most mature pipelines.
6. Stream and event ingestion
io/kinesis and io/awslambda/kinesis.py model a streaming path from AWS Kinesis into CrateDB, including checkpointing and relay logic. io/dynamodb covers DynamoDB integration, likely both bulk copy and CDC-style ingestion. These modules are built around AWS event sources and durable state for resuming ingestion.
7. Other source adapters
io/influxdb and io/deltalake / io/iceberg support time-series and open-table data sources. io/ingestr appears to provide a bridge to the external ingestr ecosystem for loading from common databases into CrateDB. This makes the toolkit a connector hub rather than a single-purpose loader.
8. Compatibility/adaptation layers
adapter/pymongo and adapter/rockset provide protocol or API compatibility layers. The PyMongo adapter exposes CrateDB-backed collection/cursor behavior for Mongo-style clients, while the Rockset adapter includes a FastAPI-style server under adapter/rockset/server with endpoints for collection, document, and query operations. These adapters translate external client expectations into CrateDB-backed behavior.
9. Operational and management tools
Several subsystems are for operational use rather than data movement:
- cluster/ for CrateDB cluster management and guidance
- info/ for runtime/environment/library introspection
- cmd/tail/ for tailing or streaming outputs
- shell/ for shell-oriented utilities
- settings/, docs/, and query/ for configuration, documentation generation, and query conversion / MCP support
These are supporting surfaces around the core ingestion and cluster workflows.
10. Retention and lifecycle management
retention/ is a separate domain with setup schema, store, and strategies such as delete, reallocate, and snapshot. This is a lifecycle subsystem that likely operates on CrateDB data or metadata to age off or reorganize historical records.
11. Dataset and example ecosystem
datasets/ and the extensive examples/ tree show that the repo is designed to bootstrap real workflows. Examples cover CDC with PostgreSQL, AWS-based pipelines, Python and shell usage, and Rockset client samples. This is a strong indication the package is meant to be both executable tooling and documentation-by-example.
12. Testing and local infrastructure
The tests/ tree mirrors nearly every subsystem, with focused integration fixtures for MongoDB, DynamoDB, Kinesis, InfluxDB, and CrateDB. testing/testcontainers/ provides containerized dependencies such as MongoDB, CrateDB, MinIO, LocalStack, Azurite, and InfluxDB, so local and CI tests can exercise real external systems. Architecture-wise, this repo depends heavily on service emulation and integration testing.
13. Build, release, and distribution
The repo is packaged as a distributable Python project (pyproject.toml, setup.py) with CI workflows for subsystem-specific test matrices and release pipelines. There are container release artifacts under release/oci-full and release/oci-ingest, plus PyPI publishing. This reinforces that the toolkit is intended for command-line, container, and library consumption.
14. Important boundary
The main architectural boundary is between source-specific ingestion logic and the shared CrateDB target/runtime utilities. Everything in the repo either:
- adapts an external system into a common internal model,
- transforms that model into CrateDB-compatible operations,
- or provides operational tooling around those flows.
That boundary is the key to drawing an accurate diagram of the repository.
Main entry points: router.py IO router · cli.py ctk CLI · bulk.py CrateDB bulk · app.py Shared util
Components 20
Entry points
Top-level CLI and command surfaces
- ctk CLI · Main repository command-line entrypoint
cratedb_toolkit/cli.py
Data I/O
Routing hub and connector subsystems
- IO router · Routes ingestion and export workflows to the right backend
cratedb_toolkit/io/router.py - MongoDB flow · MongoDB migration and CDC orchestration
cratedb_toolkit/io/mongodb/core.py - Extract/transform · MongoDB extraction, normalization, and translation steps
cratedb_toolkit/io/mongodb/extract.py - CDC pipeline · MongoDB change-data-capture replication path
cratedb_toolkit/io/mongodb/cdc.py - Kinesis flow · Kinesis relay and ingestion logic
cratedb_toolkit/io/kinesis/api.py - Checkpointing · Durable offset and resume handling for streams
cratedb_toolkit/io/kinesis/checkpointer.py - DynamoDB flow · DynamoDB copy and adapter layer
cratedb_toolkit/io/dynamodb/api.py - ingestr bridge · Integration with the external ingestr ecosystem
cratedb_toolkit/io/ingestr/api.py - CrateDB bulk · Bulk load path into CrateDB
cratedb_toolkit/io/cratedb/bulk.py - Cloud model · CrateDB Cloud connection and metadata model
cratedb_toolkit/io/cratedb/cloud/model.py - Lambda relay · AWS Lambda wrapper for Kinesis event ingestion
cratedb_toolkit/io/awslambda/kinesis.py
Adapters
Compatibility layers for external client protocols
- PyMongo adapter · CrateDB-backed Mongo-style collection and cursor behavior
cratedb_toolkit/adapter/pymongo/api.py - Rockset server · FastAPI-style Rockset-compatible API server
cratedb_toolkit/adapter/rockset/server/main.py
Operations
Cluster, retention, inspection, and query tooling
- Cluster tools · Cluster management and guidance workflows
cratedb_toolkit/cluster/core.py - Retention · Data retention strategy and store orchestration
cratedb_toolkit/retention/core.py - Info tools · Runtime and environment inspection utilities
cratedb_toolkit/info/core.py - Query convert · SQL/query conversion utilities
cratedb_toolkit/query/convert/basic.py - MCP query · MCP inquiry and registry support
cratedb_toolkit/query/mcp/pg_mcp.py
Support
Shared utilities, config, and release helpers
- Shared util · Common client, config, runtime, and service helpers
cratedb_toolkit/util/app.py
Connections 23
- ctk CLI → IO router: dispatches · Main CLI exposes I/O workflows
- ctk CLI → Cluster tools: manages · Cluster operations are surfaced as commands
- ctk CLI → Retention: runs · Retention lifecycle is exposed through CLI
- ctk CLI → Info tools: inspects · Runtime introspection is surfaced as commands
- ctk CLI → Query convert: converts · Query conversion is exposed through CLI
- IO router → MongoDB flow: routes · MongoDB is one of the major ingestion paths
- IO router → Kinesis flow: routes · Streaming flows are dispatched through the router
- IO router → DynamoDB flow: routes · AWS table ingestion is dispatched through the router
- IO router → ingestr bridge: routes · External loader integration is routed here
- IO router → CrateDB bulk: targets · Common sink-side bulk loading lands in CrateDB
- MongoDB flow → Extract/transform: steps · Batch migration is built from extraction and translation stages
- MongoDB flow → CDC pipeline: extends · CDC is the streaming branch of the MongoDB subsystem
- MongoDB flow → CrateDB bulk: loads · MongoDB flows terminate in CrateDB bulk ingestion
- Kinesis flow → Checkpointing: persists · Stream relay uses checkpoint state for resume
- Kinesis flow → Lambda relay: wraps · Lambda entrypoint hosts the stream relay
- DynamoDB flow → CrateDB bulk: loads · DynamoDB data is written through CrateDB sink paths
- PyMongo adapter → CrateDB bulk: backed by · Mongo-style compatibility is backed by CrateDB operations
- Rockset server → CrateDB bulk: queries · Rockset-compatible APIs are translated onto CrateDB-backed behavior
- Cluster tools → Shared util: uses · Cluster tools share common runtime and client helpers
- Retention → Shared util: uses · Retention logic relies on shared helpers and settings
- Info tools → Shared util: uses · Introspection uses shared runtime and formatting utilities
- MCP query → Shared util: uses · MCP tooling reuses shared helper layers
- Cloud model → Shared util: uses · Cloud-aware connectors depend on shared client/config helpers