# crate/commons-codec architecture

> Architecture diagram of the GitHub repository crate/commons-codec, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/crate/commons-codec
- Repository: https://github.com/crate/commons-codec

## Overview

1. `commons-codec` is a Python library package, not an application service. Its architecture is centered around reusable codecs and transformers for turning heterogeneous payloads into normalized data structures or SQL-oriented change events.

2. The codebase is split into two main functional subsystems under `src/commons_codec/`: `decode` and `transform`. `decode` handles source-specific payload decoding; `transform` handles converging CDC and event records into downstream-friendly representations.

3. The decoding layer is source-adapter driven. It contains dedicated modules for Sensor Community, Tasmota, and TTN/TTS payloads. These modules likely parse incoming JSON or device payloads into a common internal shape before later processing.

4. The transformation layer focuses on data integration and CDC. Modules for MongoDB, DynamoDB, and AWS DMS indicate that the library normalizes change events and emits SQL-like output or platform-specific transformations. The README explicitly calls out CDC transformers that converge event messages to SQL statements.

5. A shared model layer in `model.py` and a transformation-specific model in `transform/dynamodb_model.py` provide the common object structure used across decoders and transformers. This is the main contract boundary in the package: source formats enter on one side, normalized records exit on the other.

6. Utility code under `util/` supports the core pipeline with general-purpose data and I/O helpers. These are supporting pieces rather than primary architecture nodes, but they reduce duplication across the decode/transform modules.

7. The `vendor/` directory indicates bundled third-party code. Architecturally, this matters because it keeps the package self-contained and minimizes external runtime dependencies for core decoding/transformation behavior.

8. The package has optional capability boundaries. The README highlights extra installation support for MongoDB and Tikray, which suggests that some transformations are optional or dependency-gated rather than always included in the minimal install.

9. Example usage is centered in `examples/mongodb_cdc_cratedb.py`, which shows the intended flow for one important use case: taking MongoDB CDC input and converting it toward a CrateDB-friendly form. That example is a good reference for the library’s end-to-end data path.

10. The tests mirror the architecture closely. There are separate test areas for decoding, generic models, utilities, and each transformation family, including CDC, DynamoDB, AWS DMS, and MongoDB. This strongly suggests the public API is organized around these subsystem boundaries.

11. Documentation is a first-class part of the repo, with Sphinx docs under `doc/` covering CDC, MongoDB, telemetry, and transformation topics. The docs likely describe the supported input/output formats and the intended data flow across the library.

12. In diagram terms, the key flow is: external payloads or CDC events in, source-specific decoder/transformer modules in the middle, shared model and utility layer underneath, and SQL/normalized output or downstream integration artifacts out. The main boundaries are source adapters, shared canonical models, and destination-oriented transformation logic.

Main entry points: [`src/commons_codec/transform`](https://github.com/crate/commons-codec/tree/main/src/commons_codec/transform) (Transform), [`src/commons_codec/decode`](https://github.com/crate/commons-codec/tree/main/src/commons_codec/decode) (Decode), [`src/commons_codec`](https://github.com/crate/commons-codec/tree/main/src/commons_codec) (commons-codec), [`src/commons_codec/model.py`](https://github.com/crate/commons-codec/blob/main/src/commons_codec/model.py) (Model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_decode["Decoding"]
  node_decode_pkg["Decode"]
  node_sensor_community["Sensor Comm.<br/>source decoder"]
  node_tasmota["Tasmota<br/>source decoder<br/>[tasmota.py]"]
  node_tts_ttn["TTN/TTS<br/>source decoder<br/>[tts_ttn.py]"]
end

subgraph group_transform["Transforming"]
  node_transform_pkg["Transform"]
  node_mongodb["MongoDB<br/>cdc transformer<br/>[mongodb.py]"]
  node_dynamodb["DynamoDB<br/>cdc transformer<br/>[dynamodb.py]"]
  node_aws_dms["AWS DMS<br/>cdc transformer<br/>[aws_dms.py]"]
  node_dynamodb_model["DynamoDB model<br/>transform model<br/>[dynamodb_model.py]"]
end

subgraph group_shared["Shared Core"]
  node_pkg["commons-codec<br/>Python package"]
  node_model["Model<br/>[model.py]"]
  node_exception["Exception<br/>error types<br/>[exception.py]"]
end

subgraph group_support["Support"]
  node_util["Util<br/>helper package"]
  node_data_util["Data helpers<br/>utility module<br/>[data.py]"]
  node_io_util["I/O helpers<br/>utility module<br/>[io.py]"]
  node_vendor["Vendor<br/>bundled third-party"]
  node_example["MongoDB example<br/>usage example"]
end

subgraph group_docs["Docs"]
  node_docs_root["Docs<br/>documentation"]
  node_cdc_docs["CDC docs<br/>topic docs"]
  node_transform_docs["Transform docs<br/>topic docs<br/>[transform.md]"]
  node_telemetry_docs["Telemetry docs<br/>topic docs<br/>[telemetry.md]"]
end

node_pkg -->|"contains"| node_decode_pkg
node_pkg -->|"contains"| node_transform_pkg
node_pkg -->|"shares"| node_model
node_pkg -->|"defines"| node_exception
node_pkg -->|"supports"| node_util
node_decode_pkg -->|"adapts"| node_sensor_community
node_decode_pkg -->|"adapts"| node_tasmota
node_decode_pkg -->|"adapts"| node_tts_ttn
node_decode_pkg -->|"normalizes to"| node_model
node_transform_pkg -->|"transforms"| node_mongodb
node_transform_pkg -->|"transforms"| node_dynamodb
node_transform_pkg -->|"transforms"| node_aws_dms
node_transform_pkg -->|"specializes"| node_dynamodb_model
node_transform_pkg -->|"builds on"| node_model
node_dynamodb -->|"uses"| node_dynamodb_model
node_util -->|"contains"| node_data_util
node_util -->|"contains"| node_io_util
node_mongodb -->|"emits"| node_model
node_dynamodb -->|"emits"| node_model
node_aws_dms -->|"emits"| node_model
node_example -->|"demonstrates"| node_mongodb
node_docs_root -->|"covers"| node_cdc_docs
node_docs_root -->|"covers"| node_transform_docs
node_docs_root -->|"covers"| node_telemetry_docs
node_vendor -->|"supports"| node_decode_pkg
node_vendor -->|"supports"| node_transform_pkg
node_util -->|"supports"| node_decode_pkg
node_util -->|"supports"| node_transform_pkg

click node_pkg "https://github.com/crate/commons-codec/tree/main/src/commons_codec"
click node_decode_pkg "https://github.com/crate/commons-codec/tree/main/src/commons_codec/decode"
click node_sensor_community "https://github.com/crate/commons-codec/blob/main/src/commons_codec/decode/sensor_community.py"
click node_tasmota "https://github.com/crate/commons-codec/blob/main/src/commons_codec/decode/tasmota.py"
click node_tts_ttn "https://github.com/crate/commons-codec/blob/main/src/commons_codec/decode/tts_ttn.py"
click node_transform_pkg "https://github.com/crate/commons-codec/tree/main/src/commons_codec/transform"
click node_mongodb "https://github.com/crate/commons-codec/blob/main/src/commons_codec/transform/mongodb.py"
click node_dynamodb "https://github.com/crate/commons-codec/blob/main/src/commons_codec/transform/dynamodb.py"
click node_aws_dms "https://github.com/crate/commons-codec/blob/main/src/commons_codec/transform/aws_dms.py"
click node_dynamodb_model "https://github.com/crate/commons-codec/blob/main/src/commons_codec/transform/dynamodb_model.py"
click node_model "https://github.com/crate/commons-codec/blob/main/src/commons_codec/model.py"
click node_exception "https://github.com/crate/commons-codec/blob/main/src/commons_codec/exception.py"
click node_util "https://github.com/crate/commons-codec/tree/main/src/commons_codec/util"
click node_data_util "https://github.com/crate/commons-codec/blob/main/src/commons_codec/util/data.py"
click node_io_util "https://github.com/crate/commons-codec/blob/main/src/commons_codec/util/io.py"
click node_vendor "https://github.com/crate/commons-codec/tree/main/src/commons_codec/vendor"
click node_example "https://github.com/crate/commons-codec/blob/main/examples/mongodb_cdc_cratedb.py"
click node_docs_root "https://github.com/crate/commons-codec/tree/main/doc"
click node_cdc_docs "https://github.com/crate/commons-codec/tree/main/doc/cdc"
click node_transform_docs "https://github.com/crate/commons-codec/blob/main/doc/transform.md"
click node_telemetry_docs "https://github.com/crate/commons-codec/blob/main/doc/telemetry.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_decode_pkg,node_sensor_community,node_tasmota,node_tts_ttn toneBlue
class node_transform_pkg,node_mongodb,node_dynamodb,node_aws_dms,node_dynamodb_model toneAmber
class node_pkg,node_model,node_exception toneMint
class node_util,node_data_util,node_io_util,node_vendor,node_example toneRose
class node_docs_root,node_cdc_docs,node_transform_docs,node_telemetry_docs toneIndigo
```

## Components

### Decoding

Source-adapter layer for turning incoming payloads into normalized records.

- [Decode](https://github.com/crate/commons-codec/tree/main/src/commons_codec/decode): decoder package · Source-specific payload decoders. · `src/commons_codec/decode`
- [Sensor Comm.](https://github.com/crate/commons-codec/blob/main/src/commons_codec/decode/sensor_community.py): source decoder · Parses Sensor Community payloads. · `src/commons_codec/decode/sensor_community.py`
- [Tasmota](https://github.com/crate/commons-codec/blob/main/src/commons_codec/decode/tasmota.py): source decoder · Parses Tasmota payloads. · `src/commons_codec/decode/tasmota.py`
- [TTN/TTS](https://github.com/crate/commons-codec/blob/main/src/commons_codec/decode/tts_ttn.py): source decoder · Parses TTN/TTS payloads. · `src/commons_codec/decode/tts_ttn.py`

### Transforming

CDC and event transformation layer for downstream-friendly output.

- [Transform](https://github.com/crate/commons-codec/tree/main/src/commons_codec/transform): transform package · CDC and event transformation modules. · `src/commons_codec/transform`
- [MongoDB](https://github.com/crate/commons-codec/blob/main/src/commons_codec/transform/mongodb.py): cdc transformer · Converts MongoDB change streams into normalized downstream data. · `src/commons_codec/transform/mongodb.py`
- [DynamoDB](https://github.com/crate/commons-codec/blob/main/src/commons_codec/transform/dynamodb.py): cdc transformer · Transforms DynamoDB change events and table data. · `src/commons_codec/transform/dynamodb.py`
- [AWS DMS](https://github.com/crate/commons-codec/blob/main/src/commons_codec/transform/aws_dms.py): cdc transformer · Handles AWS DMS event normalization. · `src/commons_codec/transform/aws_dms.py`
- [DynamoDB model](https://github.com/crate/commons-codec/blob/main/src/commons_codec/transform/dynamodb_model.py): transform model · Specialized record model for DynamoDB transformations. · `src/commons_codec/transform/dynamodb_model.py`

### Shared Core

Canonical models and package-level exceptions used across the library.

- [commons-codec](https://github.com/crate/commons-codec/tree/main/src/commons_codec): Python package · Reusable codec and transformer library. · `src/commons_codec`
- [Model](https://github.com/crate/commons-codec/blob/main/src/commons_codec/model.py): shared model · Common internal data model shared by decoders and transformers. · `src/commons_codec/model.py`
- [Exception](https://github.com/crate/commons-codec/blob/main/src/commons_codec/exception.py): error types · Package-specific exceptions. · `src/commons_codec/exception.py`

### Support

Utility, bundled vendor code, and runnable examples that support the core pipeline.

- [Util](https://github.com/crate/commons-codec/tree/main/src/commons_codec/util): helper package · General-purpose data and I/O helpers. · `src/commons_codec/util`
- [Data helpers](https://github.com/crate/commons-codec/blob/main/src/commons_codec/util/data.py): utility module · Reusable data-manipulation helpers. · `src/commons_codec/util/data.py`
- [I/O helpers](https://github.com/crate/commons-codec/blob/main/src/commons_codec/util/io.py): utility module · Reusable input/output helpers. · `src/commons_codec/util/io.py`
- [Vendor](https://github.com/crate/commons-codec/tree/main/src/commons_codec/vendor): bundled third-party · Bundled external code kept inside the package. · `src/commons_codec/vendor`
- [MongoDB example](https://github.com/crate/commons-codec/blob/main/examples/mongodb_cdc_cratedb.py): usage example · End-to-end CDC example for CrateDB-oriented conversion. · `examples/mongodb_cdc_cratedb.py`

### Docs

User-facing documentation and architectural guidance.

- [Docs](https://github.com/crate/commons-codec/tree/main/doc): documentation · Sphinx documentation for CDC, telemetry, and transformation topics. · `doc`
- [CDC docs](https://github.com/crate/commons-codec/tree/main/doc/cdc): topic docs · CDC and MongoDB-oriented documentation. · `doc/cdc`
- [Transform docs](https://github.com/crate/commons-codec/blob/main/doc/transform.md): topic docs · Transformation-focused documentation. · `doc/transform.md`
- [Telemetry docs](https://github.com/crate/commons-codec/blob/main/doc/telemetry.md): topic docs · Telemetry-related documentation. · `doc/telemetry.md`

## Connections

- commons-codec → Decode: contains · Top-level decoder subsystem.
- commons-codec → Transform: contains · Top-level transformation subsystem.
- commons-codec → Model: shares · Common model used across the package.
- commons-codec → Exception: defines · Package-level exceptions.
- commons-codec → Util: supports · General helpers used by core modules.
- Decode → Sensor Comm.: adapts · Sensor Community source format.
- Decode → Tasmota: adapts · Tasmota source format.
- Decode → TTN/TTS: adapts · TTN/TTS source format.
- Decode → Model: normalizes to · Decoded payloads converge on the shared model.
- Transform → MongoDB: transforms · MongoDB CDC pipeline.
- Transform → DynamoDB: transforms · DynamoDB CDC pipeline.
- Transform → AWS DMS: transforms · AWS DMS CDC pipeline.
- Transform → DynamoDB model: specializes · DynamoDB-specific record structure.
- Transform → Model: builds on · Transforms operate on the shared data model.
- DynamoDB → DynamoDB model: uses · DynamoDB transformation-specific structure.
- Util → Data helpers: contains · Data helper functions.
- Util → I/O helpers: contains · I/O helper functions.
- MongoDB → Model: emits · Produces normalized records or SQL-oriented output.
- DynamoDB → Model: emits · Produces normalized records or SQL-oriented output.
- AWS DMS → Model: emits · Produces normalized records or SQL-oriented output.
- MongoDB example → MongoDB: demonstrates · Shows MongoDB CDC conversion flow.
- Docs → CDC docs: covers · CDC documentation area.
- Docs → Transform docs: covers · Transformation documentation area.
- Docs → Telemetry docs: covers · Telemetry documentation area.
- Vendor → Decode: supports · Bundled dependencies used by decoders.
- Vendor → Transform: supports · Bundled dependencies used by transformers.
- Util → Decode: supports · Shared helpers reduce duplication in decoding code.
- Util → Transform: supports · Shared helpers reduce duplication in transformation code.
