# ruslano69/tdtp-framework architecture

> Architecture diagram of the GitHub repository ruslano69/tdtp-framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ruslano69/tdtp-framework
- Repository: https://github.com/ruslano69/tdtp-framework

## Overview

TDTP is a data-integration framework built around self-describing packets. Its main user flow is database discovery, extraction and transformation through a CLI pipeline, then delivery as files or broker messages; packet integrity, encryption and licensing protect that flow. Separate services provide scheduled orchestration, read-only data access, visual pipeline authoring and key management. The graph preserves README-documented capabilities where source was not sampled; those links cite the README rather than implying unsampled implementation details were inspected.

Main entry points: [`pkg/core/packet`](https://github.com/ruslano69/tdtp-framework/tree/main/pkg/core/packet) (Packet engine), [`pkg/cli/commands/pipeline.go`](https://github.com/ruslano69/tdtp-framework/blob/main/pkg/cli/commands/pipeline.go) (ETL pipeline), [`cmd/tdtpcli/main.go`](https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtpcli/main.go) (TDTP CLI), [`cmd/tdtpserve/server.go`](https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtpserve/server.go) (Read-only data server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_transfer["Transfer and core"]
  node_cli["TDTP CLI<br/>[main.go]"]
  node_discovery["Database discovery"]
  node_pipeline["ETL pipeline<br/>[pipeline.go]"]
  node_packet["Packet engine"]
  node_mapping["Mapping and TDTQL"]
  node_adapters["Database adapters"]
  node_brokers["Message brokers"]
  node_storage["File and S3 storage"]
end

subgraph group_operations["Operations and access"]
  node_orchestrator["Job orchestrator"]
  node_serve["Read-only data server<br/>[server.go]"]
  node_workspace[("In-memory workspace<br/>[workspace.go]")]
end

subgraph group_authoring["Authoring and formats"]
  node_xray["Visual ETL authoring<br/>[app.go]"]
  node_preview["Source and packet preview<br/>[preview_service.go]"]
  node_tdtpservice["Packet transformations<br/>[tdtp_service.go]"]
  node_svg["SVG conversion"]
  node_sdk["Python SDK"]
end

subgraph group_protection["Protection and governance"]
  node_license["License issuance<br/>[main.go]"]
  node_licenseuse["License enforcement"]
  node_crypto["Encryption and integrity<br/>[encryption.go]"]
  node_mercuryclient["Mercury client<br/>[client.go]"]
end

subgraph group_mercury["xZMercury services"]
  node_certify["CA administration<br/>[main.go]"]
  node_mercury["Key and notary service<br/>[main.go]"]
  node_mercuryapi["Mercury API<br/>[router.go]"]
  node_ca_client["CA client<br/>[client.go]"]
end

node_engineer(("Data engineer"))
node_dbusers[("Database systems")]
node_consumers(("Packet consumers"))
node_httpclients(("HTTP data clients"))

node_engineer -->|"runs commands"| node_cli
node_cli -->|"discovers schemas"| node_discovery
node_discovery -->|"inspects sources"| node_adapters
node_adapters -->|"reads and writes"| node_dbusers
node_cli -->|"runs pipeline"| node_pipeline
node_pipeline -.->|"transforms data"| node_mapping
node_pipeline -->|"produces packets"| node_packet
node_pipeline -.->|"extracts and loads"| node_adapters
node_packet -->|"writes packet files"| node_storage
node_packet -->|"publishes messages"| node_brokers
node_packet -->|"transfers data"| node_consumers
node_orchestrator -->|"runs pipelines"| node_cli
node_httpclients -->|"requests data"| node_serve
node_serve -->|"loads datasets"| node_workspace
node_serve -->|"filters packet rows"| node_packet
node_engineer -->|"authors pipelines"| node_xray
node_xray -->|"initializes service"| node_preview
node_xray -->|"initializes service"| node_tdtpservice
node_preview -->|"parses packet data"| node_packet
node_tdtpservice -->|"processes packets"| node_packet
node_engineer -->|"converts SVG"| node_svg
node_svg -->|"creates packets"| node_packet
node_packet -->|"exposes APIs"| node_sdk
node_license -->|"issues capabilities"| node_licenseuse
node_cli -.->|"checks capabilities"| node_licenseuse
node_pipeline -->|"requests key binding"| node_mercuryclient
node_pipeline -.->|"encrypts output"| node_crypto
node_mercuryclient -.->|"requests keys"| node_mercury
node_certify -.->|"manages CA"| node_mercury
node_mercury -->|"builds router"| node_mercuryapi
node_mercuryapi -.->|"uses CA client"| node_ca_client

click node_cli "https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtpcli/main.go"
click node_discovery "https://github.com/ruslano69/tdtp-framework/tree/main/pkg/adapters"
click node_pipeline "https://github.com/ruslano69/tdtp-framework/blob/main/pkg/cli/commands/pipeline.go"
click node_packet "https://github.com/ruslano69/tdtp-framework/tree/main/pkg/core/packet"
click node_mapping "https://github.com/ruslano69/tdtp-framework/tree/main/pkg/core/mapping"
click node_adapters "https://github.com/ruslano69/tdtp-framework/tree/main/pkg/adapters"
click node_brokers "https://github.com/ruslano69/tdtp-framework/tree/main/pkg/brokers"
click node_storage "https://github.com/ruslano69/tdtp-framework/tree/main/pkg/storage"
click node_orchestrator "https://github.com/ruslano69/tdtp-framework/tree/main/cmd/orchestrator"
click node_serve "https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtpserve/server.go"
click node_workspace "https://github.com/ruslano69/tdtp-framework/blob/main/pkg/etl/workspace.go"
click node_xray "https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtp-xray/app.go"
click node_preview "https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtp-xray/services/preview_service.go"
click node_tdtpservice "https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtp-xray/services/tdtp_service.go"
click node_svg "https://github.com/ruslano69/tdtp-framework/tree/main/pkg/svg"
click node_sdk "https://github.com/ruslano69/tdtp-framework/tree/main/bindings/python/tdtp"
click node_license "https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtp-license/main.go"
click node_licenseuse "https://github.com/ruslano69/tdtp-framework/tree/main/pkg/license"
click node_crypto "https://github.com/ruslano69/tdtp-framework/blob/main/pkg/crypto/encryption.go"
click node_mercuryclient "https://github.com/ruslano69/tdtp-framework/blob/main/pkg/mercury/client.go"
click node_certify "https://github.com/ruslano69/tdtp-framework/blob/main/xzmercury/cmd/tdtp-certify/main.go"
click node_mercury "https://github.com/ruslano69/tdtp-framework/blob/main/xzmercury/cmd/xzmercury/main.go"
click node_mercuryapi "https://github.com/ruslano69/tdtp-framework/blob/main/xzmercury/internal/api/router.go"
click node_ca_client "https://github.com/ruslano69/tdtp-framework/blob/main/xzmercury/internal/caClient/client.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_discovery,node_pipeline,node_packet,node_mapping,node_adapters,node_brokers,node_storage,node_httpclients toneBlue
class node_orchestrator,node_serve,node_workspace,node_dbusers toneAmber
class node_xray,node_preview,node_tdtpservice,node_svg,node_sdk toneMint
class node_license,node_licenseuse,node_crypto,node_mercuryclient toneRose
class node_certify,node_mercury,node_mercuryapi,node_ca_client,node_engineer,node_consumers toneIndigo
```

## Components

### Transfer and core

- [TDTP CLI](https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtpcli/main.go): `cmd/tdtpcli/main.go`
- [Database discovery](https://github.com/ruslano69/tdtp-framework/tree/main/pkg/adapters): `pkg/adapters`
- [ETL pipeline](https://github.com/ruslano69/tdtp-framework/blob/main/pkg/cli/commands/pipeline.go): `pkg/cli/commands/pipeline.go`
- [Packet engine](https://github.com/ruslano69/tdtp-framework/tree/main/pkg/core/packet): `pkg/core/packet`
- [Mapping and TDTQL](https://github.com/ruslano69/tdtp-framework/tree/main/pkg/core/mapping): `pkg/core/mapping`
- [Database adapters](https://github.com/ruslano69/tdtp-framework/tree/main/pkg/adapters): `pkg/adapters`
- [Message brokers](https://github.com/ruslano69/tdtp-framework/tree/main/pkg/brokers): `pkg/brokers`
- [File and S3 storage](https://github.com/ruslano69/tdtp-framework/tree/main/pkg/storage): `pkg/storage`

### Operations and access

- [Job orchestrator](https://github.com/ruslano69/tdtp-framework/tree/main/cmd/orchestrator): `cmd/orchestrator`
- [Read-only data server](https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtpserve/server.go): `cmd/tdtpserve/server.go`
- [In-memory workspace](https://github.com/ruslano69/tdtp-framework/blob/main/pkg/etl/workspace.go): `pkg/etl/workspace.go`

### Authoring and formats

- [Visual ETL authoring](https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtp-xray/app.go): `cmd/tdtp-xray/app.go`
- [Source and packet preview](https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtp-xray/services/preview_service.go): `cmd/tdtp-xray/services/preview_service.go`
- [Packet transformations](https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtp-xray/services/tdtp_service.go): `cmd/tdtp-xray/services/tdtp_service.go`
- [SVG conversion](https://github.com/ruslano69/tdtp-framework/tree/main/pkg/svg): `pkg/svg`
- [Python SDK](https://github.com/ruslano69/tdtp-framework/tree/main/bindings/python/tdtp): `bindings/python/tdtp`

### Protection and governance

- [License issuance](https://github.com/ruslano69/tdtp-framework/blob/main/cmd/tdtp-license/main.go): `cmd/tdtp-license/main.go`
- [License enforcement](https://github.com/ruslano69/tdtp-framework/tree/main/pkg/license): `pkg/license`
- [Encryption and integrity](https://github.com/ruslano69/tdtp-framework/blob/main/pkg/crypto/encryption.go): `pkg/crypto/encryption.go`
- [Mercury client](https://github.com/ruslano69/tdtp-framework/blob/main/pkg/mercury/client.go): `pkg/mercury/client.go`

### xZMercury services

- [CA administration](https://github.com/ruslano69/tdtp-framework/blob/main/xzmercury/cmd/tdtp-certify/main.go): `xzmercury/cmd/tdtp-certify/main.go`
- [Key and notary service](https://github.com/ruslano69/tdtp-framework/blob/main/xzmercury/cmd/xzmercury/main.go): `xzmercury/cmd/xzmercury/main.go`
- [Mercury API](https://github.com/ruslano69/tdtp-framework/blob/main/xzmercury/internal/api/router.go): `xzmercury/internal/api/router.go`
- [CA client](https://github.com/ruslano69/tdtp-framework/blob/main/xzmercury/internal/caClient/client.go): `xzmercury/internal/caClient/client.go`

### Other components

- **Data engineer**
- **Database systems**
- **Packet consumers**
- **HTTP data clients**

## Connections

- Data engineer → TDTP CLI: runs commands (evidence: `README.md`)
- TDTP CLI → Database discovery: discovers schemas (evidence: `README.md`)
- Database discovery → Database adapters: inspects sources (evidence: `README.md`)
- Database adapters → Database systems: reads and writes (evidence: `README.md`)
- TDTP CLI → ETL pipeline: runs pipeline (evidence: `cmd/tdtpcli/main.go`)
- ETL pipeline → Mapping and TDTQL: transforms data
- ETL pipeline → Packet engine: produces packets (evidence: `pkg/cli/commands/pipeline.go`)
- ETL pipeline → Database adapters: extracts and loads
- Packet engine → File and S3 storage: writes packet files (evidence: `README.md`)
- Packet engine → Message brokers: publishes messages (evidence: `README.md`)
- Packet engine → Packet consumers: transfers data (evidence: `README.md`)
- Job orchestrator → TDTP CLI: runs pipelines (evidence: `README.md`)
- HTTP data clients → Read-only data server: requests data (evidence: `README.md`)
- Read-only data server → In-memory workspace: loads datasets (evidence: `cmd/tdtpserve/server.go`)
- Read-only data server → Packet engine: filters packet rows (evidence: `cmd/tdtpserve/server.go`)
- Data engineer → Visual ETL authoring: authors pipelines (evidence: `README.md`)
- Visual ETL authoring → Source and packet preview: initializes service (evidence: `cmd/tdtp-xray/app.go`)
- Visual ETL authoring → Packet transformations: initializes service (evidence: `cmd/tdtp-xray/app.go`)
- Source and packet preview → Packet engine: parses packet data (evidence: `cmd/tdtp-xray/services/preview_service.go`)
- Packet transformations → Packet engine: processes packets (evidence: `cmd/tdtp-xray/services/tdtp_service.go`)
- Data engineer → SVG conversion: converts SVG
- SVG conversion → Packet engine: creates packets (evidence: `cmd/tdtp-svg/main.go`)
- Packet engine → Python SDK: exposes APIs (evidence: `README.md`)
- License issuance → License enforcement: issues capabilities (evidence: `README.md`)
- TDTP CLI → License enforcement: checks capabilities
- ETL pipeline → Mercury client: requests key binding (evidence: `pkg/cli/commands/pipeline.go`)
- ETL pipeline → Encryption and integrity: encrypts output
- Mercury client → Key and notary service: requests keys (evidence: `README.md`)
- CA administration → Key and notary service: manages CA
- Key and notary service → Mercury API: builds router (evidence: `xzmercury/cmd/xzmercury/main.go`)
- Mercury API → CA client: uses CA client
