How ruslano69/tdtp-framework works
TDTP is a data-integration framework built around self-describing packets. Its main user flow is database discovery, extraction and transformation through a CLI pipeline, then delivery as files or broker messages; packet integrity, encryption and licensing protect that flow. Separate services provide scheduled orchestration, read-only data access, visual pipeline authoring and key management. The graph preserves README-documented capabilities where source was not sampled; those links cite the README rather than implying unsampled implementation details were inspected.
Main entry points: packet Packet engine · pipeline.go ETL pipeline · main.go TDTP CLI · server.go Read-only data server
Components 28
Transfer and core
- TDTP CLI
cmd/tdtpcli/main.go - Database discovery
pkg/adapters - ETL pipeline
pkg/cli/commands/pipeline.go - Packet engine
pkg/core/packet - Mapping and TDTQL
pkg/core/mapping - Database adapters
pkg/adapters - Message brokers
pkg/brokers - File and S3 storage
pkg/storage
Operations and access
- Job orchestrator
cmd/orchestrator - Read-only data server
cmd/tdtpserve/server.go - In-memory workspace
pkg/etl/workspace.go
Authoring and formats
- Visual ETL authoring
cmd/tdtp-xray/app.go - Source and packet preview
cmd/tdtp-xray/services/preview_service.go - Packet transformations
cmd/tdtp-xray/services/tdtp_service.go - SVG conversion
pkg/svg - Python SDK
bindings/python/tdtp
Protection and governance
- License issuance
cmd/tdtp-license/main.go - License enforcement
pkg/license - Encryption and integrity
pkg/crypto/encryption.go - Mercury client
pkg/mercury/client.go
xZMercury services
- CA administration
xzmercury/cmd/tdtp-certify/main.go - Key and notary service
xzmercury/cmd/xzmercury/main.go - Mercury API
xzmercury/internal/api/router.go - CA client
xzmercury/internal/caClient/client.go
Other components
- Data engineer
- Database systems
- Packet consumers
- HTTP data clients
Connections 31
- Data engineer → TDTP CLI: runs commands
README.md - TDTP CLI → Database discovery: discovers schemas
README.md - Database discovery → Database adapters: inspects sources
README.md - Database adapters → Database systems: reads and writes
README.md - TDTP CLI → ETL pipeline: runs pipeline
cmd/tdtpcli/main.go - ETL pipeline → Mapping and TDTQL: transforms data
- ETL pipeline → Packet engine: produces packets
pkg/cli/commands/pipeline.go - ETL pipeline → Database adapters: extracts and loads
- Packet engine → File and S3 storage: writes packet files
README.md - Packet engine → Message brokers: publishes messages
README.md - Packet engine → Packet consumers: transfers data
README.md - Job orchestrator → TDTP CLI: runs pipelines
README.md - HTTP data clients → Read-only data server: requests data
README.md - Read-only data server → In-memory workspace: loads datasets
cmd/tdtpserve/server.go - Read-only data server → Packet engine: filters packet rows
cmd/tdtpserve/server.go - Data engineer → Visual ETL authoring: authors pipelines
README.md - Visual ETL authoring → Source and packet preview: initializes service
cmd/tdtp-xray/app.go - Visual ETL authoring → Packet transformations: initializes service
cmd/tdtp-xray/app.go - Source and packet preview → Packet engine: parses packet data
cmd/tdtp-xray/services/preview_service.go - Packet transformations → Packet engine: processes packets
cmd/tdtp-xray/services/tdtp_service.go - Data engineer → SVG conversion: converts SVG
- SVG conversion → Packet engine: creates packets
cmd/tdtp-svg/main.go - Packet engine → Python SDK: exposes APIs
README.md - License issuance → License enforcement: issues capabilities
README.md - TDTP CLI → License enforcement: checks capabilities
- ETL pipeline → Mercury client: requests key binding
pkg/cli/commands/pipeline.go - ETL pipeline → Encryption and integrity: encrypts output
- Mercury client → Key and notary service: requests keys
README.md - CA administration → Key and notary service: manages CA
- Key and notary service → Mercury API: builds router
xzmercury/cmd/xzmercury/main.go - Mercury API → CA client: uses CA client