# crate/cratedb-toolkit architecture

> Architecture diagram of the GitHub repository crate/cratedb-toolkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/crate/cratedb-toolkit
- Repository: https://github.com/crate/cratedb-toolkit

## Overview

1. **Overall shape**
   This repository is a Python toolkit, not a single app. The top-level package `cratedb_toolkit` is a collection of domain modules for moving data into/out of CrateDB, inspecting and managing CrateDB/CrateDB Cloud, and providing helper CLIs, HTTP services, and library APIs. The root `ctk` CLI is the main user entrypoint.

2. **Primary architecture style**
   The code is organized as a layered toolkit:
   - thin command/CLI wrappers in `*/cli.py`
   - domain orchestration modules in feature-specific packages
   - shared utility modules under `util/`
   - model and configuration objects near each subsystem
   This makes most features reusable both as library calls and as standalone commands.

3. **Core I/O subsystem**
   The largest area is `cratedb_toolkit/io`, which acts as a routing hub for ingestion/export workflows. It contains connectors and adapters for sources and targets such as MongoDB, DynamoDB, Kinesis, InfluxDB, CrateDB, Delta Lake, Iceberg, and an “ingestr” integration. `io/router.py` likely dispatches to the right backend based on source/target type, while module-specific `api.py`, `adapter.py`, `copy.py`, `relay.py`, and `extract.py` files implement the actual data movement steps.

4. **CrateDB-specific data paths**
   Within `io/cratedb` and `io/cratedb/cloud`, the toolkit handles direct CrateDB bulk loading and Cloud-specific metadata or connection handling. These modules are the sink side of many flows, receiving transformed records from source adapters and pushing them into CrateDB through bulk or cloud-aware APIs.

5. **MongoDB migration and CDC pipeline**
   `io/mongodb` is a full subsystem with `extract`, `transform`, `translate`, `export`, `copy`, and `cdc` modules. This suggests two main flows: batch migration from MongoDB to CrateDB, and change-data-capture replication. The presence of tests with BSON/NDJSON fixtures and a dedicated CLI indicates this is one of the most mature pipelines.

6. **Stream and event ingestion**
   `io/kinesis` and `io/awslambda/kinesis.py` model a streaming path from AWS Kinesis into CrateDB, including checkpointing and relay logic. `io/dynamodb` covers DynamoDB integration, likely both bulk copy and CDC-style ingestion. These modules are built around AWS event sources and durable state for resuming ingestion.

7. **Other source adapters**
   `io/influxdb` and `io/deltalake` / `io/iceberg` support time-series and open-table data sources. `io/ingestr` appears to provide a bridge to the external `ingestr` ecosystem for loading from common databases into CrateDB. This makes the toolkit a connector hub rather than a single-purpose loader.

8. **Compatibility/adaptation layers**
   `adapter/pymongo` and `adapter/rockset` provide protocol or API compatibility layers. The PyMongo adapter exposes CrateDB-backed collection/cursor behavior for Mongo-style clients, while the Rockset adapter includes a FastAPI-style server under `adapter/rockset/server` with endpoints for collection, document, and query operations. These adapters translate external client expectations into CrateDB-backed behavior.

9. **Operational and management tools**
   Several subsystems are for operational use rather than data movement:
   - `cluster/` for CrateDB cluster management and guidance
   - `info/` for runtime/environment/library introspection
   - `cmd/tail/` for tailing or streaming outputs
   - `shell/` for shell-oriented utilities
   - `settings/`, `docs/`, and `query/` for configuration, documentation generation, and query conversion / MCP support
   These are supporting surfaces around the core ingestion and cluster workflows.

10. **Retention and lifecycle management**
   `retention/` is a separate domain with setup schema, store, and strategies such as delete, reallocate, and snapshot. This is a lifecycle subsystem that likely operates on CrateDB data or metadata to age off or reorganize historical records.

11. **Dataset and example ecosystem**
   `datasets/` and the extensive `examples/` tree show that the repo is designed to bootstrap real workflows. Examples cover CDC with PostgreSQL, AWS-based pipelines, Python and shell usage, and Rockset client samples. This is a strong indication the package is meant to be both executable tooling and documentation-by-example.

12. **Testing and local infrastructure**
   The `tests/` tree mirrors nearly every subsystem, with focused integration fixtures for MongoDB, DynamoDB, Kinesis, InfluxDB, and CrateDB. `testing/testcontainers/` provides containerized dependencies such as MongoDB, CrateDB, MinIO, LocalStack, Azurite, and InfluxDB, so local and CI tests can exercise real external systems. Architecture-wise, this repo depends heavily on service emulation and integration testing.

13. **Build, release, and distribution**
   The repo is packaged as a distributable Python project (`pyproject.toml`, `setup.py`) with CI workflows for subsystem-specific test matrices and release pipelines. There are container release artifacts under `release/oci-full` and `release/oci-ingest`, plus PyPI publishing. This reinforces that the toolkit is intended for command-line, container, and library consumption.

14. **Important boundary**
   The main architectural boundary is between source-specific ingestion logic and the shared CrateDB target/runtime utilities. Everything in the repo either:
   - adapts an external system into a common internal model,
   - transforms that model into CrateDB-compatible operations,
   - or provides operational tooling around those flows.
   That boundary is the key to drawing an accurate diagram of the repository.

Main entry points: [`cratedb_toolkit/io/router.py`](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/router.py) (IO router), [`cratedb_toolkit/cli.py`](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/cli.py) (ctk CLI), [`cratedb_toolkit/io/cratedb/bulk.py`](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/cratedb/bulk.py) (CrateDB bulk), [`cratedb_toolkit/util/app.py`](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/util/app.py) (Shared util)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entrypoints["Entry points"]
  node_ctk_cli["ctk CLI<br/>entrypoint<br/>[cli.py]"]
end

subgraph group_io["Data I/O"]
  node_io_router(("IO router<br/>dispatcher<br/>[router.py]"))
  node_mongo_core["MongoDB flow<br/>pipeline<br/>[core.py]"]
  node_mongo_extract["Extract/transform<br/>stages<br/>[extract.py]"]
  node_mongo_cdc["CDC pipeline<br/>streaming<br/>[cdc.py]"]
  node_kinesis_api["Kinesis flow<br/>streaming<br/>[api.py]"]
  node_kinesis_checkpoint["Checkpointing<br/>state<br/>[checkpointer.py]"]
  node_dynamodb_api["DynamoDB flow<br/>connector<br/>[api.py]"]
  node_ingestr_api["ingestr bridge<br/>[api.py]"]
  node_cratedb_bulk[("CrateDB bulk<br/>sink<br/>[bulk.py]")]
  node_cratedb_cloud["Cloud model<br/>[model.py]"]
  node_awslambda_kinesis["Lambda relay<br/>runtime<br/>[kinesis.py]"]
end

subgraph group_adapters["Adapters"]
  node_pymongo_adapter["PyMongo adapter<br/>compatibility<br/>[api.py]"]
  node_rockset_server["Rockset server<br/>compatibility<br/>[main.py]"]
end

subgraph group_ops["Operations"]
  node_cluster_core["Cluster tools<br/>ops<br/>[core.py]"]
  node_retention_core["Retention<br/>lifecycle<br/>[core.py]"]
  node_info_core["Info tools<br/>introspection<br/>[core.py]"]
  node_query_convert["Query convert<br/>translation<br/>[basic.py]"]
  node_query_mcp["MCP query<br/>[pg_mcp.py]"]
end

subgraph group_support["Support"]
  node_shared_util["Shared util<br/>helpers<br/>[app.py]"]
end

node_ctk_cli -->|"dispatches"| node_io_router
node_ctk_cli -->|"manages"| node_cluster_core
node_ctk_cli -->|"runs"| node_retention_core
node_ctk_cli -->|"inspects"| node_info_core
node_ctk_cli -->|"converts"| node_query_convert
node_io_router -->|"routes"| node_mongo_core
node_io_router -->|"routes"| node_kinesis_api
node_io_router -->|"routes"| node_dynamodb_api
node_io_router -->|"routes"| node_ingestr_api
node_io_router -->|"targets"| node_cratedb_bulk
node_mongo_core -->|"steps"| node_mongo_extract
node_mongo_core -->|"extends"| node_mongo_cdc
node_mongo_core -->|"loads"| node_cratedb_bulk
node_kinesis_api -->|"persists"| node_kinesis_checkpoint
node_kinesis_api -->|"wraps"| node_awslambda_kinesis
node_dynamodb_api -->|"loads"| node_cratedb_bulk
node_pymongo_adapter -->|"backed by"| node_cratedb_bulk
node_rockset_server -->|"queries"| node_cratedb_bulk
node_cluster_core -->|"uses"| node_shared_util
node_retention_core -->|"uses"| node_shared_util
node_info_core -->|"uses"| node_shared_util
node_query_mcp -->|"uses"| node_shared_util
node_cratedb_cloud -->|"uses"| node_shared_util

click node_ctk_cli "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/cli.py"
click node_io_router "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/router.py"
click node_mongo_core "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/mongodb/core.py"
click node_mongo_extract "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/mongodb/extract.py"
click node_mongo_cdc "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/mongodb/cdc.py"
click node_kinesis_api "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/kinesis/api.py"
click node_kinesis_checkpoint "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/kinesis/checkpointer.py"
click node_dynamodb_api "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/dynamodb/api.py"
click node_ingestr_api "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/ingestr/api.py"
click node_cratedb_bulk "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/cratedb/bulk.py"
click node_cratedb_cloud "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/cratedb/cloud/model.py"
click node_awslambda_kinesis "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/awslambda/kinesis.py"
click node_pymongo_adapter "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/adapter/pymongo/api.py"
click node_rockset_server "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/adapter/rockset/server/main.py"
click node_cluster_core "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/cluster/core.py"
click node_retention_core "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/retention/core.py"
click node_info_core "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/info/core.py"
click node_query_convert "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/query/convert/basic.py"
click node_query_mcp "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/query/mcp/pg_mcp.py"
click node_shared_util "https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/util/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ctk_cli toneBlue
class node_io_router,node_mongo_core,node_mongo_extract,node_mongo_cdc,node_kinesis_api,node_kinesis_checkpoint,node_dynamodb_api,node_ingestr_api,node_cratedb_bulk,node_cratedb_cloud,node_awslambda_kinesis toneAmber
class node_pymongo_adapter,node_rockset_server toneMint
class node_cluster_core,node_retention_core,node_info_core,node_query_convert,node_query_mcp toneRose
class node_shared_util toneIndigo
```

## Components

### Entry points

Top-level CLI and command surfaces

- [ctk CLI](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/cli.py): entrypoint · Main repository command-line entrypoint · `cratedb_toolkit/cli.py`

### Data I/O

Routing hub and connector subsystems

- [IO router](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/router.py): dispatcher · Routes ingestion and export workflows to the right backend · `cratedb_toolkit/io/router.py`
- [MongoDB flow](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/mongodb/core.py): pipeline · MongoDB migration and CDC orchestration · `cratedb_toolkit/io/mongodb/core.py`
- [Extract/transform](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/mongodb/extract.py): stages · MongoDB extraction, normalization, and translation steps · `cratedb_toolkit/io/mongodb/extract.py`
- [CDC pipeline](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/mongodb/cdc.py): streaming · MongoDB change-data-capture replication path · `cratedb_toolkit/io/mongodb/cdc.py`
- [Kinesis flow](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/kinesis/api.py): streaming · Kinesis relay and ingestion logic · `cratedb_toolkit/io/kinesis/api.py`
- [Checkpointing](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/kinesis/checkpointer.py): state · Durable offset and resume handling for streams · `cratedb_toolkit/io/kinesis/checkpointer.py`
- [DynamoDB flow](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/dynamodb/api.py): connector · DynamoDB copy and adapter layer · `cratedb_toolkit/io/dynamodb/api.py`
- [ingestr bridge](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/ingestr/api.py): bridge · Integration with the external ingestr ecosystem · `cratedb_toolkit/io/ingestr/api.py`
- [CrateDB bulk](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/cratedb/bulk.py): sink · Bulk load path into CrateDB · `cratedb_toolkit/io/cratedb/bulk.py`
- [Cloud model](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/cratedb/cloud/model.py): cloud · CrateDB Cloud connection and metadata model · `cratedb_toolkit/io/cratedb/cloud/model.py`
- [Lambda relay](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/io/awslambda/kinesis.py): runtime · AWS Lambda wrapper for Kinesis event ingestion · `cratedb_toolkit/io/awslambda/kinesis.py`

### Adapters

Compatibility layers for external client protocols

- [PyMongo adapter](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/adapter/pymongo/api.py): compatibility · CrateDB-backed Mongo-style collection and cursor behavior · `cratedb_toolkit/adapter/pymongo/api.py`
- [Rockset server](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/adapter/rockset/server/main.py): compatibility · FastAPI-style Rockset-compatible API server · `cratedb_toolkit/adapter/rockset/server/main.py`

### Operations

Cluster, retention, inspection, and query tooling

- [Cluster tools](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/cluster/core.py): ops · Cluster management and guidance workflows · `cratedb_toolkit/cluster/core.py`
- [Retention](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/retention/core.py): lifecycle · Data retention strategy and store orchestration · `cratedb_toolkit/retention/core.py`
- [Info tools](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/info/core.py): introspection · Runtime and environment inspection utilities · `cratedb_toolkit/info/core.py`
- [Query convert](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/query/convert/basic.py): translation · SQL/query conversion utilities · `cratedb_toolkit/query/convert/basic.py`
- [MCP query](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/query/mcp/pg_mcp.py): service · MCP inquiry and registry support · `cratedb_toolkit/query/mcp/pg_mcp.py`

### Support

Shared utilities, config, and release helpers

- [Shared util](https://github.com/crate/cratedb-toolkit/blob/main/cratedb_toolkit/util/app.py): helpers · Common client, config, runtime, and service helpers · `cratedb_toolkit/util/app.py`

## Connections

- ctk CLI → IO router: dispatches · Main CLI exposes I/O workflows
- ctk CLI → Cluster tools: manages · Cluster operations are surfaced as commands
- ctk CLI → Retention: runs · Retention lifecycle is exposed through CLI
- ctk CLI → Info tools: inspects · Runtime introspection is surfaced as commands
- ctk CLI → Query convert: converts · Query conversion is exposed through CLI
- IO router → MongoDB flow: routes · MongoDB is one of the major ingestion paths
- IO router → Kinesis flow: routes · Streaming flows are dispatched through the router
- IO router → DynamoDB flow: routes · AWS table ingestion is dispatched through the router
- IO router → ingestr bridge: routes · External loader integration is routed here
- IO router → CrateDB bulk: targets · Common sink-side bulk loading lands in CrateDB
- MongoDB flow → Extract/transform: steps · Batch migration is built from extraction and translation stages
- MongoDB flow → CDC pipeline: extends · CDC is the streaming branch of the MongoDB subsystem
- MongoDB flow → CrateDB bulk: loads · MongoDB flows terminate in CrateDB bulk ingestion
- Kinesis flow → Checkpointing: persists · Stream relay uses checkpoint state for resume
- Kinesis flow → Lambda relay: wraps · Lambda entrypoint hosts the stream relay
- DynamoDB flow → CrateDB bulk: loads · DynamoDB data is written through CrateDB sink paths
- PyMongo adapter → CrateDB bulk: backed by · Mongo-style compatibility is backed by CrateDB operations
- Rockset server → CrateDB bulk: queries · Rockset-compatible APIs are translated onto CrateDB-backed behavior
- Cluster tools → Shared util: uses · Cluster tools share common runtime and client helpers
- Retention → Shared util: uses · Retention logic relies on shared helpers and settings
- Info tools → Shared util: uses · Introspection uses shared runtime and formatting utilities
- MCP query → Shared util: uses · MCP tooling reuses shared helper layers
- Cloud model → Shared util: uses · Cloud-aware connectors depend on shared client/config helpers
