How crate/commons-codec works
1. commons-codec is a Python library package, not an application service. Its architecture is centered around reusable codecs and transformers for turning heterogeneous payloads into normalized data structures or SQL-oriented change events.
2. The codebase is split into two main functional subsystems under src/commons_codec/: decode and transform. decode handles source-specific payload decoding; transform handles converging CDC and event records into downstream-friendly representations.
3. The decoding layer is source-adapter driven. It contains dedicated modules for Sensor Community, Tasmota, and TTN/TTS payloads. These modules likely parse incoming JSON or device payloads into a common internal shape before later processing.
4. The transformation layer focuses on data integration and CDC. Modules for MongoDB, DynamoDB, and AWS DMS indicate that the library normalizes change events and emits SQL-like output or platform-specific transformations. The README explicitly calls out CDC transformers that converge event messages to SQL statements.
5. A shared model layer in model.py and a transformation-specific model in transform/dynamodb_model.py provide the common object structure used across decoders and transformers. This is the main contract boundary in the package: source formats enter on one side, normalized records exit on the other.
6. Utility code under util/ supports the core pipeline with general-purpose data and I/O helpers. These are supporting pieces rather than primary architecture nodes, but they reduce duplication across the decode/transform modules.
7. The vendor/ directory indicates bundled third-party code. Architecturally, this matters because it keeps the package self-contained and minimizes external runtime dependencies for core decoding/transformation behavior.
8. The package has optional capability boundaries. The README highlights extra installation support for MongoDB and Tikray, which suggests that some transformations are optional or dependency-gated rather than always included in the minimal install.
9. Example usage is centered in examples/mongodb_cdc_cratedb.py, which shows the intended flow for one important use case: taking MongoDB CDC input and converting it toward a CrateDB-friendly form. That example is a good reference for the library’s end-to-end data path.
10. The tests mirror the architecture closely. There are separate test areas for decoding, generic models, utilities, and each transformation family, including CDC, DynamoDB, AWS DMS, and MongoDB. This strongly suggests the public API is organized around these subsystem boundaries.
11. Documentation is a first-class part of the repo, with Sphinx docs under doc/ covering CDC, MongoDB, telemetry, and transformation topics. The docs likely describe the supported input/output formats and the intended data flow across the library.
12. In diagram terms, the key flow is: external payloads or CDC events in, source-specific decoder/transformer modules in the middle, shared model and utility layer underneath, and SQL/normalized output or downstream integration artifacts out. The main boundaries are source adapters, shared canonical models, and destination-oriented transformation logic.
Main entry points: transform Transform · decode Decode · commons_codec commons-codec · model.py Model
Components 21
Decoding
Source-adapter layer for turning incoming payloads into normalized records.
- Decode · Source-specific payload decoders.
src/commons_codec/decode - Sensor Comm. · Parses Sensor Community payloads.
src/commons_codec/decode/sensor_community.py - Tasmota · Parses Tasmota payloads.
src/commons_codec/decode/tasmota.py - TTN/TTS · Parses TTN/TTS payloads.
src/commons_codec/decode/tts_ttn.py
Transforming
CDC and event transformation layer for downstream-friendly output.
- Transform · CDC and event transformation modules.
src/commons_codec/transform - MongoDB · Converts MongoDB change streams into normalized downstream data.
src/commons_codec/transform/mongodb.py - DynamoDB · Transforms DynamoDB change events and table data.
src/commons_codec/transform/dynamodb.py - AWS DMS · Handles AWS DMS event normalization.
src/commons_codec/transform/aws_dms.py - DynamoDB model · Specialized record model for DynamoDB transformations.
src/commons_codec/transform/dynamodb_model.py
Shared Core
Canonical models and package-level exceptions used across the library.
- commons-codec · Reusable codec and transformer library.
src/commons_codec - Model · Common internal data model shared by decoders and transformers.
src/commons_codec/model.py - Exception · Package-specific exceptions.
src/commons_codec/exception.py
Support
Utility, bundled vendor code, and runnable examples that support the core pipeline.
- Util · General-purpose data and I/O helpers.
src/commons_codec/util - Data helpers · Reusable data-manipulation helpers.
src/commons_codec/util/data.py - I/O helpers · Reusable input/output helpers.
src/commons_codec/util/io.py - Vendor · Bundled external code kept inside the package.
src/commons_codec/vendor - MongoDB example · End-to-end CDC example for CrateDB-oriented conversion.
examples/mongodb_cdc_cratedb.py
Docs
User-facing documentation and architectural guidance.
- Docs · Sphinx documentation for CDC, telemetry, and transformation topics.
doc - CDC docs · CDC and MongoDB-oriented documentation.
doc/cdc - Transform docs · Transformation-focused documentation.
doc/transform.md - Telemetry docs · Telemetry-related documentation.
doc/telemetry.md
Connections 28
- commons-codec → Decode: contains · Top-level decoder subsystem.
- commons-codec → Transform: contains · Top-level transformation subsystem.
- commons-codec → Model: shares · Common model used across the package.
- commons-codec → Exception: defines · Package-level exceptions.
- commons-codec → Util: supports · General helpers used by core modules.
- Decode → Sensor Comm.: adapts · Sensor Community source format.
- Decode → Tasmota: adapts · Tasmota source format.
- Decode → TTN/TTS: adapts · TTN/TTS source format.
- Decode → Model: normalizes to · Decoded payloads converge on the shared model.
- Transform → MongoDB: transforms · MongoDB CDC pipeline.
- Transform → DynamoDB: transforms · DynamoDB CDC pipeline.
- Transform → AWS DMS: transforms · AWS DMS CDC pipeline.
- Transform → DynamoDB model: specializes · DynamoDB-specific record structure.
- Transform → Model: builds on · Transforms operate on the shared data model.
- DynamoDB → DynamoDB model: uses · DynamoDB transformation-specific structure.
- Util → Data helpers: contains · Data helper functions.
- Util → I/O helpers: contains · I/O helper functions.
- MongoDB → Model: emits · Produces normalized records or SQL-oriented output.
- DynamoDB → Model: emits · Produces normalized records or SQL-oriented output.
- AWS DMS → Model: emits · Produces normalized records or SQL-oriented output.
- MongoDB example → MongoDB: demonstrates · Shows MongoDB CDC conversion flow.
- Docs → CDC docs: covers · CDC documentation area.
- Docs → Transform docs: covers · Transformation documentation area.
- Docs → Telemetry docs: covers · Telemetry documentation area.
- Vendor → Decode: supports · Bundled dependencies used by decoders.
- Vendor → Transform: supports · Bundled dependencies used by transformers.
- Util → Decode: supports · Shared helpers reduce duplication in decoding code.
- Util → Transform: supports · Shared helpers reduce duplication in transformation code.