How datayoga-io/datayoga works
Main Components Identification
The system should be represented with these key components:
a) Core Runtime Engine:
• Job Executor
• Block Manager
• Connection Manager
• Expression Engine
• Processing Strategies Handler
b) Block Types (Group by category):
• Input Blocks (read operations)
• Transformation Blocks
• Output Blocks (write operations)
• Utility Blocks
c) External Systems Integration:
• Databases (Relational & NoSQL)
• Message Queues
• File Systems
• Cloud Services
• HTTP Services
┌─ Application Layer ─────────────────────┐
• CLI Interface │
• Configuration Management │
• ┬─┘
┌─ Core Processing Layer ───────────────┴─┐
• ┌─ Job Execution Engine ─┐ │
• Job Validator │ │
• Step Orchestrator │ │
• Block Manager │ │
• ┘ │
• ┌─ Processing Strategies ┐ │
• Async Processing │ │
• Multi-threading │ │
• Multi-processing │ │
• ┘ │
• ┬─┘
┌─ Block Layer ────────────────────────┴─┐
• ┌─ Input ─┐ ┌─ Transform ┐ ┌─ Output ┐ │
• CSV │ │ Filter │ │ Redis │ │
• Redis │ │ Map │ │ DB │ │
• DB │ │ Template │ │ HTTP │ │
• ┘ └───────────┘ └─────────┘ │
• ┬─┘
┌─ Integration Layer ──────────────────┴─┐
• Database Connectors │
• Message Queue Adapters │
• File System Handlers │
• External Service Clients │
• ┘
Key Relationships
• Data flow between steps in a job
• Block interaction with external systems
• Configuration management flow
• Error handling and monitoring paths
• Processing strategy implementation
Additional Details
Supporting elements:
• Metrics/Monitoring integration
• Configuration validation flow
• Error handling paths
• Resource management
• Security boundaries