How crate/dlt-cratedb works
1. This repository is a Python package, dlt-cratedb, that provides a CrateDB destination adapter for dlt. It is explicitly described as a temporary packaging layer until the implementation can be upstreamed into dlt itself.
2. The main runtime boundary is the adapter layer under src/dlt_cratedb/impl/cratedb. That package contains the CrateDB-specific implementation pieces: configuration, the destination class, a factory, a SQL client, and utility helpers. This is the core path an engineer would diagram for “dlt → CrateDB”.
3. The adapter is heavily based on dlt’s postgres adapter. The README says the CrateDbSqlClient diverges from the original psycopg2-based SQL client only where CrateDB needs special handling. Architecturally, this means the repo is an adaptation/override layer rather than a full standalone destination stack.
4. configuration.py represents the external configuration boundary. This is where destination-specific settings are defined and validated before being consumed by the rest of the adapter, so it sits between user-provided dlt settings and the SQL execution layer.
5. cratedb.py and factory.py are the adapter construction and integration points. These files likely assemble the destination object, wire in CrateDB-specific components, and expose the entry points dlt expects when selecting a destination implementation.
6. sql_client.py is the operational heart of the repo. It is the component that turns dlt’s abstract loading and schema operations into concrete SQL behavior against CrateDB, with utils.py supporting SQL or dialect-specific transformations.
7. The src/dlt_cratedb/patch package indicates a second architectural boundary: compatibility patches applied to dlt or the inherited postgres behavior. This is a common pattern when a destination needs to override behavior without forking the entire upstream adapter.
8. The package includes py.typed, so the public code is intended to be type-checked and consumed as typed Python. That matters for architecture because it signals the project is designed as a reusable library, not just a private integration script.
9. The test architecture is split between unit/common test helpers and load/integration tests. Under tests/load/cratedb, the repository uses a compose.yml, which strongly suggests containerized CrateDB integration tests rather than pure mocks for the end-to-end path.
10. The test suite also contains shared cases and normalizers under tests/common, implying reusable assertion logic for schema or SQL normalization across multiple destination behaviors. That usually means the adapter is being verified against a broader dlt destination contract, not just isolated functions.
11. CI/CD is handled through GitHub Actions: tests.yml for validation and release-pypi.yml for publishing. The repository also includes codecov.yml and Dependabot config, so the operational boundary includes automated testing, coverage reporting, dependency maintenance, and package release.
12. In diagram terms, the cleanest architecture is: dlt application/configuration → adapter factory → CrateDB destination object → CrateDB SQL client/patch layer → CrateDB server, with tests exercising both internal helpers and a Docker-based live database path.
Main entry points: __init__.py cratedb impl · sql_client.py sql client · test_cratedb_client.py live tests · tests.yml tests workflow
Components 24
Adapter
Core CrateDB destination implementation
- dlt-cratedb · Typed Python package that packages the CrateDB destination adapter
src/dlt_cratedb/__init__.py - impl · Internal implementation namespace for destination-specific code
src/dlt_cratedb/impl/__init__.py - cratedb impl · CrateDB adapter implementation boundary
src/dlt_cratedb/impl/cratedb/__init__.py - config · Destination configuration definitions and validation
src/dlt_cratedb/impl/cratedb/configuration.py - factory · Constructs the destination object and wires CrateDB components
src/dlt_cratedb/impl/cratedb/factory.py - destination · CrateDB destination entry point exposed to dlt
src/dlt_cratedb/impl/cratedb/cratedb.py - sql client · Executes dlt load and schema operations as CrateDB SQL
src/dlt_cratedb/impl/cratedb/sql_client.py - utils · CrateDB-specific SQL and transformation helpers
src/dlt_cratedb/impl/cratedb/utils.py
Compatibility
Overrides and compatibility helpers layered on upstream dlt behavior
- patch · Compatibility boundary for patched upstream behavior
src/dlt_cratedb/patch/__init__.py - patch utils · Overrides or adapts inherited postgres/dlt behavior
src/dlt_cratedb/patch/utils.py
Tests
Shared cases plus live CrateDB integration coverage
- shared cases · Reusable assertions and normalization cases for destination behavior
tests/common/cases/__init__.py - sql upper · Shared SQL normalization helper for tests
tests/common/cases/normalizers/sql_upper.py - title case · Shared title-case normalization helper for tests
tests/common/cases/normalizers/title_case.py - cfg utils · Reusable configuration utilities for test setup
tests/common/configuration/utils.py - live tests · End-to-end adapter tests against a containerized CrateDB
tests/load/cratedb/test_cratedb_client.py - compose · Docker Compose service definition for live CrateDB testing
tests/load/cratedb/compose.yml - table builder · Load-path verification for table construction behavior
tests/load/cratedb/test_cratedb_table_builder.py
Automation
Validation, coverage, dependency maintenance, and release workflows
- tests workflow · GitHub Actions validation pipeline
.github/workflows/tests.yml - release workflow · GitHub Actions publishing pipeline for PyPI releases
.github/workflows/release-pypi.yml - dependabot · Automated dependency update configuration
.github/dependabot.yml - codecov · Coverage reporting configuration
codecov.yml
Other components
- dlt core · Upstream dlt application and destination abstractions
- postgres adapter · Upstream postgres-based destination behavior reused by this package
- CrateDB · Target database server for the adapter
Connections 28
- dlt-cratedb → impl: contains · Top-level package exposes internal implementation namespace
- impl → cratedb impl: contains · CrateDB destination implementation lives here
- cratedb impl → config: defines · Destination configuration boundary
- cratedb impl → factory: assembles · Factory wires the destination components
- cratedb impl → destination: exposes · Destination entry point for dlt selection
- cratedb impl → sql client: operates · SQL client performs the runtime database work
- cratedb impl → utils: supports · Dialect-specific helpers shared by adapter code
- factory → destination: builds · Factory constructs the destination object
- destination → sql client: uses · Destination delegates execution to the SQL client
- sql client → utils: calls · Client relies on CrateDB-specific transformations
- config → dlt core: adapts · Configuration aligns with dlt destination settings
- destination → dlt core: integrates · Adapter plugs into dlt's destination model
- sql client → postgres adapter: inherits from · Implementation is based on the postgres adapter
- patch → patch utils: contains · Compatibility helpers are isolated under patch namespace
- patch utils → postgres adapter: patches · Overrides inherited postgres behavior where CrateDB differs
- sql client → CrateDB: executes against · SQL client talks to the CrateDB backend
- shared cases → sql upper: includes · Shared SQL normalization case
- shared cases → title case: includes · Shared title-case normalization case
- live tests → compose: uses · Integration test runtime is containerized
- live tests → CrateDB: verifies · Exercises the live database path
- live tests → sql client: covers · Primary runtime path under test
- table builder → shared cases: reuses · Table-building tests share common destination assertions
- live tests → cfg utils: reuses · Tests share configuration helpers
- tests workflow → live tests: runs · CI validates the live adapter tests
- tests workflow → table builder: runs · CI validates table-building behavior
- tests workflow → codecov: reports · Validation feeds coverage reporting
- release workflow → dlt-cratedb: publishes · Release workflow ships the typed package
- dependabot → tests workflow: supports · Dependency automation keeps validation current